DevOps - Using Consul for Configuration Management

Consul Service⌗
I previously wrote an article “Managing Service Configuration”, which mainly explained how to use Consul with confd to implement automatic updates of service configurations. So I won’t go into too much detail here about how to set up the Consul service.
Creating Configuration File Templates⌗
It’s recommended to store project configuration templates together with the project in the Git repository for easier management and deployment. For example, for our Laravel project, we create a .env.tpl
file in the project root directory with the following content:
APP_NAME={{ key "project/itdb/production/APP_NAME" }}
APP_ENV={{ key "project/itdb/production/APP_ENV" }}
APP_KEY={{ key "project/itdb/production/APP_KEY" }}
APP_DEBUG={{ key "project/itdb/production/APP_DEBUG" }}
APP_URL={{ key "project/itdb/production/APP_URL" }}
LOG_CHANNEL={{ key "project/itdb/production/LOG_CHANNEL" }}
DB_CONNECTION={{ key "project/itdb/production/DB_CONNECTION" }}
DB_HOST={{ key "project/itdb/production/DB_HOST" }}
DB_PORT={{ key "project/itdb/production/DB_PORT" }}
DB_DATABASE={{ key "project/itdb/production/DB_DATABASE" }}
DB_USERNAME={{ key "project/itdb/production/DB_USERNAME" }}
DB_PASSWORD={{ key "project/itdb/production/DB_PASSWORD" }}
BROADCAST_DRIVER={{ key "project/itdb/production/BROADCAST_DRIVER" }}
CACHE_DRIVER={{ key "project/itdb/production/CACHE_DRIVER" }}
QUEUE_CONNECTION={{ key "project/itdb/production/QUEUE_CONNECTION" }}
SESSION_DRIVER={{ key "project/itdb/production/SESSION_DRIVER" }}
SESSION_LIFETIME={{ key "project/itdb/production/SESSION_LIFETIME" }}
REDIS_HOST={{ key "project/itdb/production/REDIS_HOST" }}
REDIS_PASSWORD={{ key "project/itdb/production/REDIS_PASSWORD" }}
REDIS_PORT={{ key "project/itdb/production/REDIS_PORT" }}
MAIL_DRIVER={{ key "project/itdb/production/MAIL_DRIVER" }}
MAIL_HOST={{ key "project/itdb/production/MAIL_HOST" }}
MAIL_PORT={{ key "project/itdb/production/MAIL_PORT" }}
MAIL_USERNAME={{ key "project/itdb/production/MAIL_USERNAME" }}
MAIL_PASSWORD={{ key "project/itdb/production/MAIL_PASSWORD" }}
MAIL_ENCRYPTION={{ key "project/itdb/production/MAIL_ENCRYPTION" }}
If you find this tedious, you can also use the following method to generate it:
{{ range ls "configs/crm" }}
{{ .Key }}={{ .Value }}{{ end }}
However, with this approach, the configuration items in the final output file will be sorted alphabetically.
The key specified after “key” is the complete path of the KEY you added in Consul
Consul Template⌗
Consul Template is an open-source tool implemented in Golang by HashiCorp that converts KEY/VALUE pairs into configuration files according to templates. Official download link: https://releases.hashicorp.com/consul-template
After downloading, we need to deploy it to the remote server where we need to generate configuration files.
Simple Deployment⌗
consul-template \
-consul-addr=http://consul.example.com:8500 \
-template /path/to/.env.tpl:/path/to/target/.env:'php /path/to/project/artisan config:cache'
Please replace
/path/to/*/
above with your actual paths
This way, Consul Template will continue running, monitoring changes to KEY/VALUE pairs in Consul and modifying local configuration files accordingly. However, this approach is not ideal.
Managing with Systemd⌗
Create a /etc/consul-template/config.hcl
configuration file with the following content:
consul {
address = "http://consul.example.com:8500"
retry {
# This enabled retries. Retries are enabled by default, so this is
# redundant.
enabled = true
# This specifies the number of attempts to make before giving up. Each
# attempt adds the exponential backoff sleep time. Setting this to
# zero will implement an unlimited number of retries.
attempts = 12
# This is the base amount of time to sleep between retry attempts. Each
# retry sleeps for an exponent of 2 longer than this base. For 5 retries,
# the sleep times would be: 250ms, 500ms, 1s, 2s, then 4s.
backoff = "250ms"
# This is the maximum amount of time to sleep between retry attempts.
# When max_backoff is set to zero, there is no upper limit to the
# exponential sleep between retry attempts.
# If max_backoff is set to 10s and backoff is set to 1s, sleep times
# would be: 1s, 2s, 4s, 8s, 10s, 10s, ...
max_backoff = "1m"
}
}
# This is the signal to listen for to trigger a reload event. The default
# value is shown below. Setting this value to the empty string will cause CT
# to not listen for any reload signals.
reload_signal = "SIGHUP"
# This is the signal to listen for to trigger a graceful stop. The default
# value is shown below. Setting this value to the empty string will cause CT
# to not listen for any graceful stop signals.
kill_signal = "SIGINT"
# This is the maximum interval to allow "stale" data. By default, only the
# Consul leader will respond to queries; any requests to a follower will
# forward to the leader. In large clusters with many requests, this is not as
# scalable, so this option allows any follower to respond to a query, so long
# as the last-replicated data is within these bounds. Higher values result in
# less cluster load, but are more likely to have outdated data.
max_stale = "10m"
# This is the log level. If you find a bug in Consul Template, please enable
# debug logs so we can help identify the issue. This is also available as a
# command line flag.
log_level = "warn"
# This is the path to store a PID file which will contain the process ID of the
# Consul Template process. This is useful if you plan to send custom signals
# to the process.
pid_file = "/var/run/consul-template.pid"
# This is the quiescence timers; it defines the minimum and maximum amount of
# time to wait for the cluster to reach a consistent state before rendering a
# template. This is useful to enable in systems that have a lot of flapping,
# because it will reduce the the number of times a template is rendered.
wait {
min = "5s"
max = "10s"
}
# This block defines the configuration for a template. Unlike other blocks,
# this block may be specified multiple times to configure multiple templates.
# It is also possible to configure templates via the CLI directly.
template {
# This is the source file on disk to use as the input template. This is often
# called the "Consul Template template". This option is required if not using
# the `contents` option.
source = "/path/on/disk/to/.env.tpl"
# This is the destination path on disk where the source template will render.
# If the parent directories do not exist, Consul Template will attempt to
# create them, unless create_dest_dirs is false.
destination = "/path/on/disk/where/template/will/.env"
# This options tells Consul Template to create the parent directories of the
# destination path if they do not exist. The default value is true.
create_dest_dirs = true
# This is the optional command to run when the template is rendered. The
# command will only run if the resulting template changes. The command must
# return within 30s (configurable), and it must have a successful exit code.
# Consul Template is not a replacement for a process monitor or init system.
command = "php /path/to/project/artisan config:cache"
# This is the maximum amount of time to wait for the optional command to
# return. Default is 30s.
command_timeout = "60s"
# Exit with an error when accessing a struct or map field/key that does not
# exist. The default behavior will print "<no value>" when accessing a field
# that does not exist. It is highly recommended you set this to "true" when
# retrieving secrets from Vault.
error_on_missing_key = false
# This is the permission to render the file. If this option is left
# unspecified, Consul Template will attempt to match the permissions of the
# file that already exists at the destination path. If no file exists at that
# path, the permissions are 0644.
perms = 0600
# This option backs up the previously rendered template at the destination
# path before writing a new one. It keeps exactly one backup. This option is
# useful for preventing accidental changes to the data without having a
# rollback strategy.
backup = true
# These are the delimiters to use in the template. The default is "{{" and
# "}}", but for some templates, it may be easier to use a different delimiter
# that does not conflict with the output file itself.
left_delimiter = "{{"
right_delimiter = "}}"
# This is the `minimum(:maximum)` to wait before rendering a new template to
# disk and triggering a command, separated by a colon (`:`). If the optional
# maximum value is omitted, it is assumed to be 4x the required minimum value.
# This is a numeric time with a unit suffix ("5s"). There is no default value.
# The wait value for a template takes precedence over any globally-configured
# wait.
wait {
min = "2s"
max = "10s"
}
}
If you need to deploy multiple projects on one server and want Consul Template to monitor configuration information for multiple projects, you can add multiple template
configuration blocks. For more detailed parameters, refer to: https://github.com/hashicorp/consul-template#configuration-file-format.
Create a /etc/systemd/system/consul-template.service
file with the following content:
[Unit]
Description=Consul Template Service
Documentation=https://github.com/hashicorp/consul-template
After=syslog.target network.target
ConditionFileIsExecutable=/usr/lib/consul-template/consul-template
[Service]
Type=simple
Restart=on-failure
StartLimitBurst=10
StartLimitInterval=5
PIDFile=/var/run/consul-template.pid
ExecStart=/usr/lib/consul-template/consul-template --config=/etc/consul-template/config.hcl
ExecReload=/bin/kill -s SIGHUP $MAINPID
ExecStop=/bin/kill -s SIGINT $MAINPID
[Install]
WantedBy=multi-user.target
sudo chmod 754 /etc/systemd/system/consul-template.service
sudo systemctl daemon-reload
sudo systemctl enable consul-template.service
sudo systemctl start consul-template.service
This completes the process management of Consul Template with Systemd.
I hope this is helpful, Happy hacking…