Background⌗

I chose Traefik mainly because of its strong ecosystem support, such as Kubernetes, Docker Swarm, etc., which can automatically discover containers and register routes. Later, I switched from Nginx to Traefik locally. Initially, I used mkcert to generate certificates for all my local services, but I found that every time I deployed a project, I had to manually generate a TLD certificate for the project and then manually add the certificate to Traefik’s configuration file. This is not a big workload, but it becomes extremely time-consuming when dealing with many projects:

Here are the certificate files I’m currently using for existing projects:

tree
.
├── certs
│   ├── acme.json
│   ├── adguard.test.cer
│   ├── adguard.test.key
│   ├── aptabase.test.cer
│   ├── aptabase.test.key
│   ├── authorizer.test.cer
│   ├── authorizer.test.key
│   ├── bytebase.test.cer
│   ├── bytebase.test.key
│   ├── calcom.test.cer
│   ├── calcom.test.key
│   ├── cockroachdb.test.cer
│   ├── cockroachdb.test.key
│   ├── consul.test.cer
│   ├── consul.test.key
│   ├── dify.test.cer
│   ├── dify.test.key
│   ├── directus.test.cer
│   ├── directus.test.key
│   ├── emqx.test.cer
│   ├── emqx.test.key
│   ├── ente.test.cer
│   ├── ente.test.key
│   ├── expose.test.cer
│   ├── expose.test.key
│   ├── fleet.test.cer
│   ├── fleet.test.key
│   ├── formbricks.test.cer
│   ├── formbricks.test.key
│   ├── gitlab.test.cer
│   ├── gitlab.test.key
│   ├── hatchet.test.cer
│   ├── hatchet.test.key
│   ├── huly.test.cer
│   ├── huly.test.key
│   ├── ingress.test.cer
│   ├── ingress.test.key
│   ├── livekit.test.cer
│   ├── livekit.test.key
│   ├── logto.test.cer
│   ├── logto.test.key
│   ├── mailpit.test.cer
│   ├── mailpit.test.key
│   ├── matrix.test.cer
│   ├── matrix.test.key
│   ├── mattermost.test.cer
│   ├── mattermost.test.key
│   ├── mercure.test.cer
│   ├── mercure.test.key
│   ├── minio.test.cer
│   ├── minio.test.key
│   ├── outline.test.cer
│   ├── outline.test.key
│   ├── pages.test.cer
│   ├── pages.test.key
│   ├── rallly.test.cer
│   ├── rallly.test.key
│   ├── redpanda.test.cer
│   ├── redpanda.test.key
│   ├── registry.test.cer
│   ├── registry.test.key
│   ├── river.test.cer
│   ├── river.test.key
│   ├── rocket.test.cer
│   ├── rocket.test.key
│   ├── snipe-it.test.cer
│   ├── snipe-it.test.key
│   ├── sqlchat.test.cer
│   ├── sqlchat.test.key
│   ├── sshx.test.cer
│   ├── sshx.test.key
│   ├── svc.dev.cer
│   ├── svc.dev.key
│   ├── typebot.test.cer
│   ├── typebot.test.key
│   ├── unit.test.cer
│   ├── unit.test.key
│   ├── wallos.test.cer
│   ├── wallos.test.key
│   ├── warrant.test.cer
│   ├── warrant.test.key
│   ├── zitadel.test.cer
│   └── zitadel.test.key
├── config
│   └── tls.yaml
└── docker-compose.yaml

3 directories, 85 files

config/tls.yaml configuration:

global:
  checkNewVersion: true
  sendAnonymousUsage: true
serversTransport:
  insecureSkipVerify: true
tls:
  options:
    default:
      sniStrict: true
      minVersion: VersionTLS12
      maxVersion: VersionTLS13
      cipherSuites:
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
  certificates:
    - certFile: /certs/svc.dev.cer
      keyFile: /certs/svc.dev.key
    - certFile: /certs/unit.test.cer
      keyFile: /certs/unit.test.key
    - certFile: /certs/ente.test.cer
      keyFile: /certs/ente.test.key
    - certFile: /certs/emqx.test.cer
      keyFile: /certs/emqx.test.key
    - certFile: /certs/dify.test.cer
      keyFile: /certs/dify.test.key
    - certFile: /certs/huly.test.cer
      keyFile: /certs/huly.test.key
    - certFile: /certs/river.test.cer
      keyFile: /certs/river.test.key
    - certFile: /certs/fleet.test.cer
      keyFile: /certs/fleet.test.key
    - certFile: /certs/wallos.test.cer
      keyFile: /certs/wallos.test.key
    - certFile: /certs/calcom.test.cer
      keyFile: /certs/calcom.test.key
    - certFile: /certs/expose.test.cer
      keyFile: /certs/expose.test.key
    - certFile: /certs/sshx.test.cer
      keyFile: /certs/sshx.test.key
    - certFile: /certs/minio.test.cer
      keyFile: /certs/minio.test.key
    - certFile: /certs/logto.test.cer
      keyFile: /certs/logto.test.key
    - certFile: /certs/pages.test.cer
      keyFile: /certs/pages.test.key
    - certFile: /certs/gitlab.test.cer
      keyFile: /certs/gitlab.test.key
    - certFile: /certs/consul.test.cer
      keyFile: /certs/consul.test.key
    - certFile: /certs/livekit.test.cer
      keyFile: /certs/livekit.test.key
    - certFile: /certs/mailpit.test.cer
      keyFile: /certs/mailpit.test.key
    - certFile: /certs/rallly.test.cer
      keyFile: /certs/rallly.test.key
    - certFile: /certs/ingress.test.cer
      keyFile: /certs/ingress.test.key
    - certFile: /certs/rocket.test.cer
      keyFile: /certs/rocket.test.key
    - certFile: /certs/warrant.test.cer
      keyFile: /certs/warrant.test.key
    - certFile: /certs/matrix.test.cer
      keyFile: /certs/matrix.test.key
    - certFile: /certs/adguard.test.cer
      keyFile: /certs/adguard.test.key
    - certFile: /certs/typebot.test.cer
      keyFile: /certs/typebot.test.key
    - certFile: /certs/hatchet.test.cer
      keyFile: /certs/hatchet.test.key
    - certFile: /certs/registry.test.cer
      keyFile: /certs/registry.test.key
    - certFile: /certs/zitadel.test.cer
      keyFile: /certs/zitadel.test.key
    - certFile: /certs/mercure.test.cer
      keyFile: /certs/mercure.test.key
    - certFile: /certs/outline.test.cer
      keyFile: /certs/outline.test.key
    - certFile: /certs/sqlchat.test.cer
      keyFile: /certs/sqlchat.test.key
    - certFile: /certs/aptabase.test.cer
      keyFile: /certs/aptabase.test.key
    - certFile: /certs/snipe-it.test.cer
      keyFile: /certs/snipe-it.test.key
    - certFile: /certs/directus.test.cer
      keyFile: /certs/directus.test.key
    - certFile: /certs/bytebase.test.cer
      keyFile: /certs/bytebase.test.key
    - certFile: /certs/redpanda.test.cer
      keyFile: /certs/redpanda.test.key
    - certFile: /certs/formbricks.test.cer
      keyFile: /certs/formbricks.test.key
    - certFile: /certs/authorizer.test.cer
      keyFile: /certs/authorizer.test.key
    - certFile: /certs/mattermost.test.cer
      keyFile: /certs/mattermost.test.key
    - certFile: /certs/cockroachdb.test.cer
      keyFile: /certs/cockroachdb.test.key

You might wonder why I don’t use a single FQDN? Using a single FQDN is possible, but the domain name would be longer, and some projects contain multiple containers with HTTP services. For example, the MinIO project requires multiple subdomains for different needs:

api.minio.test: MinIO’s API Endpoint
console.minio.test: MinIO’s management console
*.minio.test: Directly mapping Subdomain to Bucket

If we don’t use TLDs to distinguish different services but instead use different subdomains of the same FQDN, there are two approaches:

Use multi-level subdomains to distinguish, but this approach has the same issue - projects requiring fourth-level domains still need manually generated certificates. For example, if the FQDN is svc.dev, then console.minio.test would become console.minio.svc.dev, and the wildcard certificate for *.svc.dev would be invalid, requiring a manual generation of a wildcard certificate for *.minio.svc.dev.
Use hyphens to distinguish different services of the same project, for example: console.minio.test would correspond to minio-console.svc.dev. This approach has the lowest cost, but the domain names are too long, which I feel provides a poor user experience.

Considering all these factors, I chose the previous approach, using second-level domains with specific TLDs to distinguish projects, and then using third-level domains to distinguish different services within projects. This approach provides the best experience for visitors but requires more complex network configuration.

First, you need a TLD that doesn’t conflict with the public internet, such as *.test
You need to run a local DNS, such as dnsmasq
The system needs to support specifying a NAMESERVER for specific TLDs or FQDNs, which in macOS is defined in the /etc/resolver directory
You need to be familiar with underlying network principles, such as IP, DNS, etc.

For example, my computer has the following configuration:

tree /etc/resolver
/etc/resolver
├── infra
├── svc.dev
└── test

1 directory, 3 files

Since the TLD of svc.dev is a publicly available domain name, I only use second-level domains as local access domains to avoid overriding a large number of internet domains!

The configuration of /etc/resolver/svc.dev is as follows:

cat /etc/resolver/svc.dev
nameserver 127.0.0.1

When I need to resolve domains like *.svc.dev, the system will send the request to the specified DNS, which is dnsmasq listening on 127.0.0.1:53.

Then I need to configure A records for *.svc.dev in dnsmasq:

# ACME DNS
address=/dns.svc.dev/10.8.10.253

# Smallstep CA
address=/ca.svc.dev/10.8.10.254

# Traefik ingress
address=/.svc.dev/10.8.10.252

# Forward DNS request
server=/./223.5.5.5

# Listen Address
listen-address=0.0.0.0

All my container services run in the 10.8.10.0/24 network segment, and I’ve configured fixed IPs for Step-CA, ACME DNS, and Traefik.

You might wonder why I need to run an ACME DNS inside the container when the host machine already has dnsmasq. This DNS is mainly for handling the dnsChallenge business in the ACME protocol.

Due to space limitations, I will share ACME DNS-related content in another article!

Automatic HTTPS⌗

Actually, you only need the Traefik and Step-CA containers to implement certificates for FQDNs, mainly using httpChallenge or tlsChallenge. The configuration in Traefik’s docker-compose.yaml is as follows:

services:
  traefik:
    dns: 
      - 10.0.6.8
    image: traefik:latest
    ports:
      - 0.0.0.0:80:80/tcp
      - 0.0.0.0:443:443/tcp
    labels:
      - traefik.tls.stores.default.defaultgeneratedcert.resolver=step-ca
      - traefik.tls.stores.default.defaultgeneratedcert.domain.main=svc.dev
      - traefik.tls.stores.default.defaultgeneratedcert.domain.sans=*.svc.dev

      - traefik.enable=true
      - traefik.docker.network=traefik

      - traefik.http.routers.traefik-dashboard.tls=true
      - traefik.http.routers.traefik-dashboard.tls.certresolver=step-ca
      - traefik.http.routers.traefik-dashboard.entrypoints=http,https
      - traefik.http.routers.traefik-dashboard.rule=Host(`traefik.svc.dev`)
      - traefik.http.routers.traefik-dashboard.service=dashboard@internal

      - traefik.http.routers.traefik-dashboard-api.tls=true
      - traefik.http.routers.traefik-dashboard.tls.certresolver=step-ca
      - traefik.http.routers.traefik-dashboard-api.entrypoints=http,https
      - traefik.http.routers.traefik-dashboard-api.rule=Host(`traefik.svc.dev`) && PathPrefix(`/api`)
      - traefik.http.routers.traefik-dashboard-api.service=api@internal
    hostname: traefik
    networks:
      traefik:
        ipv4_address: 10.8.10.252
    command: 
      - --api=true
      - --api.dashboard=true

      - --log.level=ERROR

      - --accesslog=true
      - --accesslog.fields.defaultmode=keep
      - --accesslog.fields.names.RouterName=keep
      - --accesslog.fields.headers.defaultMode=keep
      - --accesslog.fields.headers.names.RouterName=keep
      - --accesslog.fields.headers.names.RequestHost=keep

      - --providers.file=true
      - --providers.file.watch=true
      - --providers.file.directory=/etc/traefik/config
      - --providers.file.debugloggeneratedtemplate=true

      - --providers.docker=true
      - --providers.docker.watch=true
      - --providers.docker.network=traefik
      - --providers.docker.useBindPortIP=false
      - --providers.docker.endpoint=unix:///var/run/docker.sock

      - --serverstransport.insecureskipverify=true

      - --entrypoints.ssh.address=:22

      - --entrypoints.http.address=:80
      - --entrypoints.http.http.redirections.entryPoint.to=https
      - --entrypoints.http.http.redirections.entryPoint.scheme=https
      - --entryPoints.http.http.redirections.entrypoint.permanent=true

      - --entrypoints.https.address=:443
      - --entryPoints.https.http3.advertisedport=443
      - --entryPoints.https.http.tls.certResolver=step-ca
      - --entryPoints.https.http.tls.domains[0].main=svc.dev
      - --entryPoints.https.http.tls.domains[0].sans=*.svc.dev

      - [email protected]
      - --certificatesresolvers.step-ca.acme.storage=/certs/acme.json
      - --certificatesresolvers.step-ca.acme.caserver=https://ca.svc.dev/acme/acme/directory
      - --certificatesresolvers.step-ca.acme.tlschallenge=true
      - --certificatesresolvers.step-ca.acme.dnschallenge=false
      - --certificatesresolvers.step-ca.acme.dnschallenge.provider=httpreq
      - --certificatesresolvers.step-ca.acme.httpchallenge=false
    volumes:
      - step-ca:/step-ca:ro
      - ./certs/:/certs/:rw
      - ./config/:/etc/traefik/config/:ro
      - /var/run/docker.sock:/var/run/docker.sock
    logging:
      driver: json-file
      options:
        max-size: 32m
    extra_hosts:
      - ca.svc.dev:10.8.10.254
      - dns.svc.dev:10.8.10.253
    environment:
      - TZ=Asia/Shanghai
      - LEGO_CA_CERTIFICATES=/step-ca/certs/root_ca.crt
    container_name: traefik

volumes:
  certs:
    name: certs
    external: true
  step-ca:
    name: step-ca
    external: true

networks:
  traefik:
    external: true

If you’re not familiar with Step-CA, you can check out my article “Setting up a Private ACME Server with Step-CA”.

Lines 3-4: Use the host’s dnsmasq to resolve services, mainly when Step-CA needs to request HTTPS services during tlsChallenge
Lines 10-12: Set default TLS certificate related configuration
Lines 18, 24: Set TLS certificate resolver for routes
Lines 67-69: Set resolver and domain for Entrypoint
Lines 75-77: Since I redirected all HTTP traffic to HTTPS, I can only enable tlsChallenge
Lines 79, 92: Mount the root certificate generated by the Step-CA container, otherwise TLS handshake will fail

As long as DNS resolution is normal, Step-CA will be able to issue certificates for Traefik after startup!

In the above configuration, I intentionally made an error demonstration, which is the wildcard certificate domain configured in lines 11-12 and 68-69, because only dnsChallenge supports wildcard certificate issuance!

The general process of tlsChallenge and httpChallenge is shown in the following diagram:

tlsChallenge

After the certificate application is successful, the corresponding crt and key will be stored in the directory file configured by --certificatesresolvers.step-ca.acme.storage:

{
  "step-ca": {
    "Account": {
      "Email": "[email protected]",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:[email protected]"
          ],
          "orders": "https://ca.svc.dev/acme/acme/account/FFqNin75x7Kk5Y6R3FcUdsAGSgL486yr/orders"
        },
        "uri": "https://ca.svc.dev/acme/acme/account/FFqNin75x7Kk5Y6R3FcUdsAGSgL486yr"
      },
      "PrivateKey": "MIIJKL...........Pt1jfvTA=",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "traefik.svc.dev"
        },
        "certificate": "LS0tLS...........RFLS0tLS0K",
        "key": "LS0tLS...........VEUgT21XS0VZLS0tLS0K",
        "Store": "default"
      }
    ]
  }
}

Summary⌗

Following the above steps, you can implement Traefik requesting certificates from an internal private ACME Server. However, there is still one issue: only dnsChallenge supports wildcard certificate applications.

This is often needed in projects, such as using subdomains as Bucket mappings in MinIO!

To implement wildcard certificate applications, you must integrate a DNS that supports API updates in the internal network, such as ACME DNS. Traefik’s dnsChallenge Provider also supports acme-dns.

For more content about ACME DNS and wildcard certificate applications, stay tuned for the next article!

I hope this is helpful, Happy hacking…

Implementing Automatic HTTPS with Traefik and Step-CA

Background⌗

Automatic HTTPS⌗

Summary⌗