Introduction⌗

During the development of enterprise internal projects, we often find code that spans across multiple projects. To facilitate maintenance, we abstract this part of the code and create a separate extension package. This approach greatly improves maintenance efficiency across multiple projects.

However, there’s also a challenge: how to manage these extension packages and make them quickly discoverable by Go modules!

Setting GOPRIVATE⌗

go env -w GOPRIVATE=gitlab.example.com

GOPRIVATE supports setting multiple private repositories and also supports using the wildcard *.

Setting GOPROXY⌗

This requires Gitlab to enable the Go Proxy feature. Currently, the Go Package Registry is not available in production, so this approach is not recommended!!!

go env -w GOPROXY='https://gitlab.example.com/api/v4/projects/{id}/packages/go,https://proxy.golang.org,direct'

It’s best to place the private extension package URL at the beginning of the proxy list. Additionally, if you have multiple packages and the network allows, you can simply set the GOPROXY value to direct.

Configuring Authentication⌗

This step is mainly to allow Go modules to authenticate when downloading Release packages from Gitlab, as private projects require login to access.

Create a .netrc file in the current system user’s home directory to store the Personal Access Token for the internal Gitlab.

echo "machine <url> login <username> password <token>" >> ~/.netrc
  • <url>: Replace with your internal Gitlab domain, e.g. gitlab.example.com;
  • <username>: Gitlab username, e.g. [email protected]. In CI/CD, you can use gitlab-ci-token instead of the actual user;
  • <token>: Your Gitlab Personal Access Token. In CI/CD, you can use the $CI_JOB_TOKEN environment variable.

Disabling Extension Package Verification⌗

When downloading dependencies using Go 1.13 and higher versions, the sources are verified against the checksum database sum.golang.org. Since our extension packages are private, sum.golang.org cannot verify them.

go env -w GONOSUMDB="gitlab.example.com/services/modules"

This can be specific to a certain project or just at the domain level. If you’ve set GOPRIVATE, then all packages under that domain will not be verified!

Using Extension Packages⌗

Now you can use the go get command to install private extension packages.

go get gitlab.example.com/services/modules/inquiry@latest

Extension Package Versions⌗

If we need to use the latest commit from a specific branch in the test environment, we can use the following command:

go get gitlab.example.com/services/modules/inquiry@<branch>

# For example, if I want to use the latest develop branch, after execution, the package will be upgraded from v1.1.0 to v1.1.1-0.20231123032709-e22204ae03de
go get gitlab.test/modules/mail@develop
go: downloading gitlab.test/modules/mail v1.1.1-0.20231123032709-e22204ae03de
go: upgraded gitlab.test/modules/mail v1.1.0 => v1.1.1-0.20231123032709-e22204ae03de

The version v1.1.1-0.20231123032709-e22204ae03de above is an automatically generated pseudo-version number for a development stage, generated by Go Module with the following syntax:

baseVersionPrefix-timestamp-revisionIdentifier

  • baseVersionPrefix (v1.1.1-0): Derived from the latest release version of the current package. For example, in my example above, it’s automatically derived from v1.1.0 to v1.1.1-0. If the package hasn’t released any tags yet, it will be v0.0.0
  • timestamp (20231123032709): This is the UTC timestamp of the commit
  • revisionIdentifier (e22204ae03de): The first 12 characters of the Git commit hash, or a zero-padded revision number for Subversion.

For major version changes, Go’s official best practice is to change the package’s import path simultaneously. For example, for the package gitlab.test/modules/mail mentioned above, to upgrade to v2, you need to modify the go.mod declaration in the package:

module "gitlab.test/modules/mail/v2"

If it doesn’t follow this version management best practice, the package will have +incompatible added after the version in the require declaration in go.mod.

While this design allows importing multiple incompatible versions of the same package in a single project, it’s quite troublesome for source code management. For example, maintaining code for both v1 and subsequent major versions simultaneously, switching between branches causes all sorts of conflicts in go.mod!

Personally, I don’t think this approach solves the versioning problem well. On the contrary, it increases the workload for developers maintaining PATCH versions of older versions! And for package users, they not only have to adapt to the new version syntax but also have to globally replace the package’s import PATH.

The above is my personal understanding. For more about Go module versioning, refer to the Go official documentation: Module version numbering

Conclusion⌗

After implementing the installation of private extension packages, a new problem arose: every time we need to update dependencies in the project, we have to PUSH changes to the Gitlab Repository and then manually execute go get in the project.

This workflow is too cumbersome, so I was wondering if there’s a solution similar to PHP Composer that differentiates dependency sources by environment. For example, in the development environment, a certain dependency extension package could point to a local directory, while in CI/CD builds, it would get the extension package from Gitlab.

Finally found a solution! In Go 1.18, a new Workspace feature was added that can solve this problem well. For specific details, refer to “Using Go Workspace to Implement Local Extension Packages”

I hope this is helpful, Happy hacking…