Introduction⌗

There are many ways to build Go projects. How can you find one that suits your team?

Today I’ll share some experiences I’ve discovered through exploration. By leveraging Gitlab’s powerful ecosystem, we can make Go projects more efficient from development to deployment!

Environment⌗

For this experiment, I ran the following services locally using Docker:

Traefik: For service auto-discovery and reverse proxy
Gitlab: For testing CI/CD workflows (with Container Registry enabled)
Gitlab Runner: For executing CI/CD tasks

For Traefik and Gitlab deployment, you can refer to my previous two articles with detailed deployment solutions!

Domains⌗

gitlab.test
registry.test

Since this is an internal test environment, all service HTTPS certificates are self-signed using mkcert. This created a significant challenge for the upcoming infrastructure setup, but through troubleshooting, I gained a deeper understanding of the underlying principles!

Registering a Runner⌗

In the latest version of Gitlab, you can access the Runner management page at https://gitlab.test/admin/runners:

Once there, click the New instance runner button to create one
Fill in Tags and description information, then click Create runner
Save the Gitlab-Runner registration Token

Next, write the Gitlab Runner configuration file and docker-compose.yaml file:

tree
.
├── README.md
├── config
│   ├── certs
│   │   └── ca.crt          # CA certificate for self-signed certificates
│   └── config.toml         # Gitlab Runner configuration file 
└── docker-compose.yml

3 directories, 4 files

config/config.toml file:

user = "gitlab-runner"
log_level = "error"
log_format = "text"
concurrent = 5
check_interval = 10
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  id = 1
  url = "https://gitlab.test"
  name = "docker-runner"
  token = "glrt-fgsyprdA3oxDJzxQJxbB"
  executor = "docker"
  cache_dir = "/cache"
  builds_dir = "/builds"
  token_obtained_at = 2023-11-22T06:12:41Z
  token_expires_at = 0001-01-01T00:00:00Z
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    image = "docker:latest"
    volumes = ["/cache", "/builds", "/var/run/docker.sock:/var/run/docker.sock:ro"]
    shm_size = 0
    privileged = true
    tls_verify = false
    extra_hosts = ["gitlab.test:10.8.10.254", "registry.test:10.8.10.254"]
    network_mtu = 0
    pull_policy = ["if-not-present"]
    network_mode = "traefik"
    tls_cert_path = "/certs"
    disable_cache = false
    oom_kill_disable = false
    allowed_services = ["docker:*"]
    services_privileged = true
    allowed_pull_policies = ["if-not-present"]
    disable_entrypoint_overwrite = false

On lines 13 and 15, add the URL and Token obtained earlier
On line 25, add /var/run/docker.sock:/var/run/docker.sock:ro to volumes
On line 29, set custom TLD resolution, where 10.8.10.254 is the IP I manually set for Traefik in the traefik network

docker-compose.yaml file:

services:
  runner:
    image: gitlab/gitlab-runner:latest
    restart: always
    hostname: runner
    container_name: runner
    extra_hosts:
      - gitlab.test:10.8.10.254
    volumes:
      - ./config:/etc/gitlab-runner
      - runner-cache:/cache
      - runner-builds:/builds
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      TZ: PRC
      CA_CERTIFICATES_PATH: /etc/gitlab-runner/certs/ca.crt
      CI_SERVER_TLS_CA_FILE: /etc/gitlab-runner/certs/ca.crt

volumes:
  runner-cache:
    name: runner-cache
  runner-builds:
    name: runner-builds

networks:
  traefik:
    external: true

Lines 7-8 set HOST resolution for Gitlab Runner
Lines 16-17 set trusted certificates for Gitlab Runner when requesting https://gitlab.test domain!

docker compose up -d

[+] Building 0.0s (0/0)                                                             docker:orbstack
[+] Running 1/1
 ✔ Container runner  Started

After successful startup, go to the Gitlab Admin Runner page to check if the Runner is working properly:

Gtilab Runner

Project Adaptation⌗

To simulate a real development environment, I created two project groups and two projects:

The main configuration is in the services/sendbox project, with the following directory structure:

tree -a -I '.git|.idea'
.
├── .gitignore
├── .gitlab-ci.yml
├── Dockerfile
├── LICENSE
├── README.md
├── cmd
│   ├── root.go
│   └── serve.go
├── go.mod
├── go.sum
├── go.work
├── go.work.sum
└── main.go

2 directories, 12 files

I used Go Workspace locally to import local packages:

cat go.work
go 1.21.4

use .

replace gitlab.test/modules/mail => ../modules/mail

Dockerfile⌗

FROM golang:latest AS compiler-source-code

ARG VERSION=latest
ARG CI_JOB_USER=gitlab-ci-token
ARG GITLAB_HOST=gitlab.test
ARG CI_JOB_TOKEN
ARG CA_CERTIFICATE

WORKDIR /go/src/sendbox
RUN echo "machine gitlab.test login $CI_JOB_USER password $CI_JOB_TOKEN" >> ~/.netrc && \
    echo $CA_CERTIFICATE >> /usr/local/share/ca-certificates/ca.crt && \
    update-ca-certificates
ADD . /go/src/sendbox
RUN go env -w GOPRIVATE=gitlab.test && go mod tidy && GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w -X gitlab.test/services/sendbox/cmd.version=$VERSION -X gitlab.test/services/sendbox/cmd.commit=$(git rev-parse HEAD)" -o sendbox main.go

FROM ubuntu:latest AS build-docker-image
COPY --from=compiler-source-code /go/src/sendbox/sendbox /usr/local/bin/sendbox
ENTRYPOINT ["/usr/local/bin/sendbox"]

Line 10: Write authentication information to ~/.netrc
Line 11: Write the self-signed CA certificate into the system (can be deleted if not using self-signed certificates or HTTPS)
Line 12: Update the system’s trusted CA certificates (can be deleted if not using self-signed certificates or HTTPS)
Line 14: Use -ldflags to write version information into project variables, which may be used later for log analysis and version stability analysis

During Docker image building, you can add settings to pull different private packages based on the environment. For example, in a test environment, you can execute go get gitlab.test/modules/mail@develop before building to pull the latest commit from the develop branch. This avoids issues in the test environment after a private package has been tagged, which would otherwise require multiple releases of the private package!

CI/CD Configuration⌗

Note that the latest version of Gitlab defaults to using the .yml suffix, so using .yaml will cause CI/CD to fail to retrieve it. If you must use the latter, you can set it in Gitlab:

Default CI/CD configuration file

You can modify the default CI configuration filename under Default CI/CD configuration file in the image above!

The .gitlab-ci.yml configuration is as follows:

default:
  image: docker:latest
  before_script:
    - export    # You can delete this if you don't need to print all environment variables
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

variables:
  GITLAB_IP: 10.0.6.3
  CI_JOB_USER: gitlab-ci-token
  GITLAB_HOST: gitlab.test
  CA_CERTIFICATE: $CA_CERTIFICATE
  CONTAINER_VER_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
  CONTAINER_LATEST_TAG: $CI_REGISTRY_IMAGE:latest

stages:
  - build
  - release

build:
  tags:
    - docker
  stage: build
  script:
    - docker build --add-host gitlab.test:"$GITLAB_IP" --pull --tag "$CONTAINER_VER_TAG" --build-arg VERSION="$CI_COMMIT_REF_SLUG" --build-arg CI_JOB_USER="$CI_JOB_USER" --build-arg GITLAB_HOST="$GITLAB_HOST" --build-arg CI_JOB_TOKEN="$CI_JOB_TOKEN" --build-arg CA_CERTIFICATE="$CA_CERTIFICATE" .
    - docker push $CONTAINER_VER_TAG

release:
  only:
    refs:
      - tags
  tags:
    - docker
  stage: release
  script:
    - docker pull $CONTAINER_VER_TAG
    - docker tag $CONTAINER_VER_TAG $CONTAINER_LATEST_TAG
    - docker push $CONTAINER_LATEST_TAG

Gitlab CI preset environment variables:

CI_REGISTRY: Container Registry HOST, which corresponds to registry.test here
CI_REGISTRY_USER: gitlab-ci-token
CI_REGISTRY_PASSWORD: Same value as CI_JOB_TOKEN. If you need to operate on images from other projects, refer to the Permission Issues section below
CI_REGISTRY_IMAGE: Automatically generated based on the project’s namespace, e.g., registry.test/services/sendbox in my case
CI_COMMIT_REF_NAME: The name of the commit. If it’s a branch, it’s the branch name; if it’s a tag, it’s the corresponding version, e.g., master, develop, or v1.0.0

Using CI_COMMIT_REF_NAME directly for Image Tag might cause issues. For example, if the branch is feature/auth, the generated Image Tag would be registry.test/services/sendbox:feature/auth, which clearly doesn’t conform to Tag naming conventions. Instead, you can use another environment variable CI_COMMIT_REF_SLUG, which replaces non-alphanumeric characters with slugs, e.g., feature/auth becomes feature-auth!

For more preset environment variables, refer to the Gitlab official documentation GitLab CI/CD variables.

Line 4: Used to print all preset Gitlab CI environment variables. If you don’t want to read the documentation, you can execute export. Sensitive information will be marked as [MASKED]
Line 26: --add-host gitlab.test:"$GITLAB_IP" is not needed if gitlab is not running locally. In my case, I used dnsmasq to resolve the *.test TLD to 127.0.0.1, which causes buildx to request 127.0.0.1:443 when accessing gitlab.test!
Lines 36-37: Tag the latest version as latest

Permission Issues⌗

If you use a Personal Token for authentication in CI/CD, the owner of that Personal Token needs access permissions to the private package repository.

I’m using CI_JOB_TOKEN here, which is valid during the Pipeline lifecycle. After Gitlab 15.9, for security reasons, cross-project access is not allowed. This means that when using CI_JOB_TOKEN in the services/sendbox project’s CI/CD to pull the modules/mail private package, authentication will fail, with an error as shown on line 233 in the image below:

Gitlab pipeline failed

When this problem occurs, you can either disable the Limit Access to the project feature or add the project you want to access to the Allow CI job tokens from the following projects to access this project list below:

Gitlab CI/CD Settings - Token Access

After this, you’ll be able to successfully retrieve the private package code on the next execution!

Summary⌗

At this point, the project’s build and distribution issues have been well resolved. This combination of Overwrite DNS + Self-signed certificates can really drive you crazy!

However, after going through the most difficult process, I gained a better understanding of how services in the Gitlab ecosystem interact with each other, which might be a blessing in disguise.

I’ll share more about CD best practices in the future, so stay tuned…

I hope this is helpful, Happy hacking…

Building Docker Images for Go Projects with Gitlab Ecosystem