Accelerating Github Project Cloning by Configuring Git Proxy
Introduction⌗
Perhaps because I’ve always used environment variables to set proxies when installing packages with NPM, Composer, or Homebrew to improve download speeds, I subconsciously thought that the ALL_PROXY environment variable might apply to all protocols.
However, when I used git clone [email protected]:u-boot/u-boot.git to clone a project, I noticed there was no response for a long time. Looking at Surge’s traffic monitor, I saw no traffic passing through, which meant it wasn’t using the ALL_PROXY I had set.
After some search-engine-oriented learning, I discovered that Git supports three protocols:
- HTTP
- SSH
- Git Protocol
Common Misconceptions⌗
Many online tutorials suggest configuring http and https proxies in the ~/.gitconfig file:
[http "https://github.com/"]
proxy = http://<host>:<port>/
[https "https://github.com/"]
proxy = https://<host>:<port>/
Or setting proxies through commands like:
git config --global http.proxy http://username:[email protected]:8080
git config --global https.proxy http://username:[email protected]:8080
Or using temporary environment variables as I did before:
export https_proxy=http://127.0.0.1:8234;export http_proxy=http://127.0.0.1:8234;export all_proxy=socks5://127.0.0.1:8235
Note: This method only works for HTTP protocol! If you’re using SSH protocol for cloning, these settings won’t have any effect.
Configuring SSH Proxy⌗
As in my previous article “Using ClashX to Accelerate SSH”, we just need to add the following configuration to the ~/.ssh/config file:
Host github.com
User git
ProxyCommand nc -x 127.0.0.1:8235 -X 5 %h %p
After setting this up, you can test cloning projects from Github using SSH.
For developers using gh 1, the same applies if you’re using the SSH protocol.
gh repo clone u-boot/u-boot
Git Protocol⌗
I initially didn’t plan to write about the Git Protocol since it’s rarely used, but when learning about operating systems, I needed to install riscv-tools. Following the steps from 6.S081: Operating System Engineering, I found that cloning the glibc source code was stuck due to network issues. Looking at the logs, I saw that the repository address was git://sourceware.org/git/glibc.git.
At first, I didn’t recognize this as the Git Protocol. I set up a proxy for sourceware.org in the configuration file using the SSH method, but it didn’t work. After pondering for a while, I realized this wasn’t the SSH protocol.
Now that I had identified the problem, the solution was simple. First, I needed to create a git-proxy script:
#! /bin/bash
# connect to the Git repository through a SOCKS proxy
# default setting is to use port 1080 on the local host
proxy="localhost:1080"
from="default"
# check if there is a value in the git configuration
if git config --get socks.proxy >& /dev/null; then
proxy=`git config --get socks.proxy`
from="git's socks.proxy"
fi
# check if a generic proxy has been defined in the environment
if [ -n "$ALL_PROXY" ]; then
proxy="$ALL_PROXY"
from="\$ALL_PROXY"
fi
if [ -n "$all_proxy" ]; then
proxy="$all_proxy"
from="\$all_proxy"
fi
# check if a SOCKS proxy has been defined in the environment
if [ -n "$SOCKS_PROXY" ]; then
proxy="$SOCKS_PROXY"
from="\$SOCKS_PROXY"
fi
if [ -n "$socks_proxy" ]; then
proxy="$socks_proxy"
from="\$socks_proxy"
fi
if [ -n "$SOCKS5_PROXY" ]; then
proxy="$SOCKS5_PROXY"
from="\$SOCKS5_PROXY"
fi
if [ -n "$socks5_proxy" ]; then
proxy="$socks5_proxy"
from="\$socks5_proxy"
fi
# check if a git specific SOCKS proxy has been defined in the environment
if [ -n "$GIT_SOCKS_PROXY" ]; then
proxy="$GIT_SOCKS_PROXY"
from="\$GIT_SOCKS_PROXY"
fi
function usage() {
cat << @EOF
Usage:
`basename $0` HOST PORT
Helper script to connect to a Git repository over the git:// protocol at host HOST and port PORT through a SOCKS proxy at $proxy ($from).
To use the proxy for all git:// traffic, set the core.gitproxy option to "git-proxy":
git config core.gitproxy "git-proxy"
To use the proxy only for some reporitories, use the syntax explained in git-config(1).
To configure which proxy to use, set an appropriate environment variable (see below) or socks.proxy option to the proxy address, for example "localhost:1080":
git config socks.proxy "localhost:1080"
The address of the proxy is read from (in order of priority):
- the GIT_SOCKS_PROXY environment variable;
- the SOCKS_PROXY or SOCKS5_PROXY environment variable;
- the ALL_PROXY environment variable (see curl(1));
- the socks.proxy git option;
- the default value: localhost:1080 .
@EOF
}
if [ -z "$1" ] || [ -z "$2" ] || [ -n "$3" ]; then
usage
exit 1
fi
# connect through the specifid proxy
nc -x "$proxy" "$1" "$2"
Save the above code as a script in any directory in your PATH, such as /usr/local/bin/, and name it git-proxy. Then run the following command to verify it works:
git-proxy --help
Usage:
git-proxy HOST PORT
Helper script to connect to a Git repository over the git:// protocol at host HOST and port PORT through a SOCKS proxy at 127.0.0.1:8235 (git's socks.proxy).
To use the proxy for all git:// traffic, set the core.gitproxy option to "git-proxy":
git config core.gitproxy "git-proxy"
To use the proxy only for some reporitories, use the syntax explained in git-config(1).
To configure which proxy to use, set an appropriate environment variable (see below) or socks.proxy option to the proxy address, for example "localhost:1080":
git config socks.proxy "localhost:1080"
The address of the proxy is read from (in order of priority):
- the GIT_SOCKS_PROXY environment variable;
- the SOCKS_PROXY or SOCKS5_PROXY environment variable;
- the ALL_PROXY environment variable (see curl(1));
- the socks.proxy git option;
- the default value: localhost:1080 .
Then follow the instructions and run:
# It's recommended to use the absolute path for the script, otherwise you might get: error: cannot run git-proxy: No such file or directory.
git config core.gitproxy "/usr/local/bin/git-proxy"
git config socks.proxy "127.0.0.1:8235"
Also, since Surge’s Copy Shell Export Command generates commands that include export all_proxy=socks5://127.0.0.1:8235, according to the git-proxy script logic, if the all_proxy environment variable is set, it will override Git’s global configuration!!! This is what led me to discover the following issue.
Note: The proxy address here doesn’t need the protocol type; just use the HOST:PORT format. If you include the protocol, like socks5://127.0.0.1:8235, it will cause “nc: getaddrinfo: nodename nor servname provided, or not known” and fail to proxy the Git Protocol.
Now test if traffic through the Git Protocol is routed through our Surge proxy as expected:
brew install riscv-tools
==> Downloading https://ghcr.io/v2/homebrew/core/mpfr/manifests/4.1.0
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/e8b18e060649c00d9e1e351e3e05515c365e52dbd7bd392e2b99808d1f355b5b--mpfr-4.1.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/mpfr/blobs/sha256:81ced499f237acfc2773711a3f8aa985572eaab2344a70485c06f72405e4a5e7
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/1816ae3e6ff09ea7ad42b8f5ff16142fd0a973c41c5633dfa3b1cf4fa2e1d51b--mpfr--4.1.0.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/gawk/manifests/5.1.1
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/13f45768bf43ccad0a4fabeca4923e0a6836a412b1255585fba6260956e53b70--gawk-5.1.1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/gawk/blobs/sha256:093465f34b94ec8ddeb4ff8dab2a02dafbccf8ec05f6ef0391673b7c4fd0a91f
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/0e765b98a270520e3539fb021f555e78705e08b6f24da7e20a56d1f228be9ff1--gawk--5.1.1.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/gnu-sed/manifests/4.8
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/71fd4a7118e6fb954728ce6a8f92871d721711578de17baa01337b3570581179--gnu-sed-4.8.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/gnu-sed/blobs/sha256:78481cc3509f617328d3c361c21beef829f24f4b130cabfc08ed6e4ce83f2286
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/c0d5ea51ee74ae84aee90d77161075b8567d8c949e99a85582e2fa1570eb824b--gnu-sed--4.8.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/isl/manifests/0.24
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/cf108f3cbdbad48a5ecbe0a148e97b47590045a58151ad768bcf7219ec23305a--isl-0.24.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/isl/blobs/sha256:be08c3e9765655ad5bfd227f9b97acb0ef88ad2307dc214ea4064cc1f51db641
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/65e4670b48d6dda0e181978e35cf506256f350f33cf7e1371b9451856ad21d41--isl--0.24.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/libmpc/manifests/1.2.1
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/3f193de9dd7f9e68742cf207cfadd3a63b695a8026d343b668e10a9b7bcf6e52--libmpc-1.2.1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/libmpc/blobs/sha256:658a1d260b6f77c431451a554ef8fa36ea2b6db19b38adc05c52821598ce2647
Already downloaded: /Users/George/Library/Caches/Homebrew/downloads/d18d439506c54ddbad35cc4dc0c77e5fc3fd442c9ec926ff212d6fed1a11c8b6--libmpc--1.2.1.arm64_monterey.bottle.tar.gz
==> Cloning https://github.com/riscv/riscv-gnu-toolchain.git
Updating /Users/George/Library/Caches/Homebrew/riscv-gnu-toolchain--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 1342cd7 Merge pull request #1073 from palmer-dabbelt/qemu-system
Entering 'qemu'
Entering 'riscv-binutils'
Entering 'riscv-dejagnu'
Entering 'riscv-gcc'
Entering 'riscv-gdb'
Cloning into '/Users/George/Library/Caches/Homebrew/riscv-gnu-toolchain--git/glibc'...
Cloning into '/Users/George/Library/Caches/Homebrew/riscv-gnu-toolchain--git/musl'...
Cloning into '/Users/George/Library/Caches/Homebrew/riscv-gnu-toolchain--git/newlib'..
ps -ef | grep "nc -x 127.0.0.1:8235" | grep -v grep
501 53932 53462 0 11:58PM ttys002 0:00.00 nc -x 127.0.0.1:8235 sourceware.org 9418
As you can see, when executing the Homebrew install command, a netcat process was started, establishing a connection to port 9418 of sourceware.org through the proxy. 9418 is the port used by the Git Protocol!
Using HTTP Port for Project Cloning with SSH Protocol⌗
Recently, due to the shutdown of Suwa Cloud, I had to switch to another proxy service. After switching, I found I couldn’t clone Github projects normally using the SSH protocol, getting the following error:
gh repo clone golang-module/carbon
Cloning into 'carbon'...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
exit status 128
After inquiring, I learned that the proxy service administrators had blocked port 22 traffic for security reasons. They also provided a solution, which is documented in the Github official documentation: Using SSH over the HTTPS port.
Following the documentation, I modified my ~/.ssh/config configuration:
Host github.com
User git
HostName ssh.github.com
Port 443
ProxyCommand nc -x 127.0.0.1:8235 -X 5 %h %p
Now I can normally accelerate Git operations using the SSH protocol!
I hope this is helpful, Happy hacking…