Introduction⌗

I frequently deploy services on my internal network using Traefik as a reverse proxy with self-signed TLS certificates. When accessing these services from a local browser, everything works smoothly. However, when services deployed in containers need to communicate with each other, things get much more complicated. Let me share the challenges you might encounter!

Domain Name Resolution⌗

The services I deploy locally are primarily for my own machine access, rarely involving internal network access. I use Dnsmasq as my DNS service, which resolves all .test TLDs to 127.0.0.1.

This works fine when accessing from a local browser, but problems arise with container-to-container communication. For example, if Outline needs to access MinIO service, and MinIO is behind Traefik reverse proxy with the domain name minio.test, the traffic flow looks like this:

`Outline` ---> `Traefik` ---> `MinIO`

This creates an issue: when Outline requests the domain minio.test, it gets 127.0.0.1 from Dnsmasq, which within the container means it’s requesting itself, unable to route traffic to the Traefik container.

There are two viable solutions to this problem:

Configure Dnsmasq to resolve records to Traefik’s IP address;
Set extra_hosts in the Docker Compose configuration file, equivalent to hardcoding entries in the container’s /etc/hosts file.

Either approach requires setting a fixed IP for the Traefik container!

For example, I use Valet as my PHP development environment locally, and the configuration in the ~/.config/valet/dnsmasq.d directory is as follows:

tree
.
├── tld-infra.conf # Infrastructure service resolution
├── tld-svc.dev.conf # Valet related project resolution
└── tld-test.conf # Other test service resolution

1 directory, 3 files

docker network inspect traefik
[
    {
        "Name": "traefik",
        "Id": "1bab78892e73451c7ed7f73a9ac4415da15d690885d293edb6aeb287e35e156b",
        "Created": "2023-10-22T18:07:03.173509102+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.8.10.0/24",
                    "Gateway": "10.8.10.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b34f15b5a78146fbe5265e4653e843cb7201b8642747aa03dd4a11c9fc8122be": {
                "Name": "traefik",
                "EndpointID": "466ea1148e4ad1fe84757ee3c489e44b5c75b037d5988f43daaeb80b99f82012",
                "MacAddress": "02:42:0a:08:0a:fe",
                "IPv4Address": "10.8.10.254/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

For instance, in my setup, all *.test traffic goes through traefik, and the Dnsmasq configuration is as follows:

address=/.test/10.8.10.254
listen-address=127.0.0.1

This way, both containers and the host machine directly request the Traefik container, avoiding unknown issues that might arise when the resolution address is loopback!

Certificates⌗

When the Outline container needs to request MinIO over HTTPS with a self-signed certificate, the TLS handshake fails because the X.509 certificate isn’t trusted! The solution is simple, but implementation may vary depending on the base image!

For Linux-based systems, you typically place CA certificates in the container’s /usr/local/share/ca-certificate directory, then enter the container and execute:

update-ca-certificates

If the update-ca-certificates command isn’t installed in the container, you can directly append the CA certificate content to the /etc/ssl/certs/ca-certificates.crt file:

cat /usr/local/share/ca-certificates/ca.crt >> /etc/ssl/certs/ca-certificates.crt

This approach is generally applicable, but the obvious issue is that you either need to manually execute this when creating the container or rewrite the Dockerfile to execute these steps during the build phase. This way, you don’t have to manually perform the above operations every time you recreate the container!

Remember that after updating certificates, you need to restart the container (not recreate it), otherwise the changes won’t take effect!

Besides the above solutions, certain specific services also support configuration through environment variables:

For Node.JS images, you can use the NODE_EXTRA_CA_CERTS environment variable to specify the CA location, so Node.JS processes in the container will automatically load the CA;
Gitlab Runner can trust CA certificates through the CA_CERTIFICATES_PATH and CI_SERVER_TLS_CA_FILE environment variables.
For Python images, if using the requests library, use REQUESTS_CA_BUNDLE; if using the httpx library, use the SSL_CERT_FILE environment variable to specify the CA location, so Python processes in the container will automatically load the CA;

Conclusion⌗

When developing locally and trying to simulate a production-like experience, there are quite a few pitfalls that require patience and debugging skills. But if you persist, you’ll find that not a single hair was lost in vain…

I hope this is helpful, Happy hacking…

Trusting Self-Signed CA Certificates in Local Docker Environment

Introduction⌗

Domain Name Resolution⌗

Certificates⌗

Conclusion⌗