Hashicorp Nomad service discovery
With the recently released Nomad 1.3 a service registry backed by Nomad was added. This is making building complex services dependent on others really easy. Earlier we typically would have added Hashicorp Consul as service registry. In a production environment that would have meant to a add a Consul cluster running along with the Nomad cluster. Obviously this comes with some costs even when running such a cluster is really made simple by Hashicorp.
However, it’s built in now as of Nomad 1.3 with many improvements coming with 1.3.x and 1.4. This includes such features as simplified east-west load balancing and service health checks.
This time I want to show how simple it is to add a service that is required by some other service.
With my last post I already showed how to deploy a containerized workload and expose it as Nomad service. This time we add a Postgresql database service and connect the application service to that database service using the Nomad service provider and service discovery.
Add database service
For the application service we had added a group api with the task api.
All the details have been stripped so we can just focus on the job description structure.
job "continuousdelivery" {
group "api" {
count = 3
# ...
task "api" {
driver = "docker"
# ...
env {
SERVER_PORT = "8080"
MANAGEMENT_SERVER_PORT = "8081"
SPRING_JPA_GENERATE_DDL = true
}
}
}
}Ok, so now first we need to add a database service. We will add a new task group as we don’t want to tie the new workload to the deployment rules of the api group. We will only need a single database service while we want to scale-out the api service and all api instances will then connect to this one database service. Also, all tasks in a group are co-located on the same Nomad client, so they can share some resources. This is not needed with this setup, both tasks are scheduled independently. Those details about task group specifications are well documented in the job specification documents.
job "continuousdelivery" {
group "db" { (1)
count = 1
network {
port "postgresdb" {
to = "5432"
}
}
task "db" {
driver = "docker"
config {
image = "bitnami/postgresql:11"
ports = [ "postgresdb" ]
}
env {
POSTGRESQL_DATABASE="app"
POSTGRESQL_USERNAME="spring"
POSTGRESQL_PASSWORD="boot"
}
service {
name = "continuousdelivery-db"
provider = "nomad"
port = "postgresdb"
}
}
}
group "api" {
count = 3
task "api" {
driver = "docker"
# ...
env {
SERVER_PORT = "8080"
MANAGEMENT_SERVER_PORT = "8081"
SPRING_JPA_GENERATE_DDL = true
}
template { (2)
destination="application.env"
env = true
data = <<EOH
SPRING_DATASOURCE_URL=jdbc:postgresql://{{ range nomadService "continuousdelivery-db" }}{{ .Address }}:{{ .Port }}{{ end }}/app
SPRING_DATASOURCE_USERNAME="spring"
SPRING_DATASOURCE_PASSWORD="boot"
EOH
}
}
}
}| 1 | A new group specification db is added including a network port mapping, task and service stanza |
| 2 | A template stanza is added to the existing task specification. It generates the environment setup to connect the api service to the db service. |
We have used a template stanza for creating a dynamic configuration using environment variables.
By promoting the generated configuration as environment with env=true, the environment variables are injected into the workload’s
environment.
The template will even be updated when the service registry is updated and re-schedule the api task in case the db task was updated or going down and rescheduled for example on a different Nomad client (node).
Let’s deploy the updated job specification.
nomad job run \ (1)
--var-file credentials.hcl \ (2)
--var-file application.hcl \ (3)
continuousdelivery.hclWe can now view the registered services with the following command
> nomad service list
Service Name Tags
continuousdelivery-api [api]
continuousdelivery-db [db]To get a bit more insight into the many objects that were created while deploying the job descriptor we can run the following command:
> nomad status continuousdelivery
ID = continuousdelivery
Name = continuousdelivery
Submit Date = 2022-07-25T10:49:31+02:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost Unknown
api 0 0 3 0 0 0 0
db 0 0 1 0 0 0 0
Latest Deployment
ID = 91b03490
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Auto Revert Desired Placed Healthy Unhealthy Progress Deadline
api true 3 3 3 0 2022-07-25T11:00:26+02:00
db false 1 1 1 0 2022-07-25T10:59:44+02:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
84683324 59556282 api 0 run running 2m57s ago 2m2s ago
99f62cff 59556282 api 0 run running 2m57s ago 2m10s ago
a1947721 59556282 db 0 run running 2m57s ago 2m43s ago
eafe5928 59556282 api 0 run running 2m57s ago 2m9s agoIf you are eager to see how the dynamic configuration was created, we can look into the generated configuration template output
> nomad alloc fs 84683324 api/application.env (1)
SPRING_DATASOURCE_URL=jdbc:postgresql://192.168.1.18:30658/app
SPRING_DATASOURCE_USERNAME="spring"
SPRING_DATASOURCE_PASSWORD="boot"| 1 | We used the allocation ID 84683324 from the previous command to view the allocation’s file system. |
In case you wanted to view the complete code on GitHub, here is the link to the commits related to this post.
I hope with this post I have been able to add some valuable input for you to understand how (almost) trivial it is to work with Nomad’s service discovery.
Not knowing how close Nomad’s service registry will come to the Consul service registry already available for years, it may make sense to look into the Consul way of registering and discovering services. Expect a follow-up post on this very soon.