Load Go modules from private repos

Edit on GitHub
Jun 28, 2024
Notes
3 minutes

Private repos and non-public modules

By default go get uses HTTPS. In a continuous deployment scenario, specially if you have 2FA enabled with your Git provider, you will need to generate a PAT (Personal Access Token) in order to use HTTPS without a prompt, or use SSH instead of HTTPS.

If you’re deploying with Netlify and have continuous deployment, you will likely use PAT instead of SSH.

PAT (Personal Access Token)

Generate an access token from Github settings. It should have read access to your code

 1add_github_token() {
 2  # Add a PAT so that Go can fetch modules from private repos
 3
 4  GITHUB_USERNAME=""
 5  GITHUB_PAT=""
 6  FILE=${HOME}/.netrc
 7
 8  if [ ! -f ${FILE} ]; then
 9    log "Creating file: ${FILE}"
10    touch ${FILE} 
11  fi
12
13  echo -e >> "machine github.com login ${GITHUB_USERNAME} password ${GITHUB_PAT}"
14}
15
16add_github_token

PAT on Netlify

Save the PAT as an environment variable in Netlify

The value for $HOME in Netlify build context is /opt/buildhome/. If you create a file at $HOME/.netrc it will end up at the location /opt/buildhome/.netrc, and not ~/.netrc or /home/FOO/.netrc as you would normally expect

You can set git config --global inside a Netlify build script. To access a private repo, you can pass your credentials as part of the repo URL. Git repo URLs with credentials look like this:

https://<TOKEN>@github.com/<REPO_OWNER>/<REPO_NAME>
https://<USERNAME>:<TOKEN>@github.com/<REPO_PATH>
https://<USERNAME>:<TOKEN>@<SELF_HOSTED_GITLAB_SERVER>/<PATH_TO_REPO>.git

You’ll replace the values for TOKEN, USERNAME, REPO_NAME, REPO_PATH etc. and the use the final URL to set git config with a insteadOf directive. The config command will look like this:

1git config --global --add url.https://${USERNAME}:${TOKEN}@github.com/.insteadof https://github.com/
2
3# or a more specific
4git config --global --add url."https://${USERNAME}:${TOKEN}@github.com/${REPO_PATH}".insteadOf "https://github.com/${REPO_PATH}"

Note: Both insteadOf and insteadof (with lowercase ‘o’) work. Quoting the strings is optional but recommended.

1# netlify.toml
2[build]
3  command = "./netlify_build.sh"

 1# netlify_build.sh
 2settings_for_private_go_modules() {
 3  REPO_PATH="blah_username/blah_repo"
 4
 5  # REPO_PATH is a Go module inside a private repo
 6  # GITHUB_PAT is a Personal Access Token, set as Netlify env var, that has read access for that repo
 7  # token will expire after the validity period
 8  git config --global --add url."https://${GITHUB_USERNAME}:${GITHUB_PAT}@github.com/${REPO_PATH}".insteadOf "https://github.com/${REPO_PATH}"
 9}
10
11settings_for_private_go_modules

Enforce SSH

Whether you clone a repo over HTTPS or SSH is determined by the repo URL.

1# HTTPS
2https://github.com/USER/BLAH.git

1# SSH
2git@github.com:USER/BLAH.git

In order to force Git to always get the repos over SSH, we can use the insteadOf directive in our (global) git config.

1# Force all URL to be SSH 
2git config --global --add url."git@github.com:".insteadOf "https://github.com/"

It will add code that looks like this to your ~/.gitconfig

1[url "git@github.com:"]
2 insteadOf = https://github.com/

GOPRIVATE

You should also set the GOPRIVATE variable (go 1.13 onwards)

The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database. ref

go env -w GOPRIVATE=github.com/{ORG_NAME/USER_NAME}/*

OR Add this to your ~/.bashrc

1export GOPRIVATE="github.com/{ORG_NAME/USER_NAME}/*"

Multiple links can be provided as a comma separated list

1export GOPRIVATE="gitlab.com/FOO,bitbucket.org/FOO,github.com/FOO"

You can verify your changes with go env

Notes