Hashicorp Nomad with Consul service discovery

2022-07-26 4 min read martin

With my previous post I demonstrated how simple it is to connect services through Nomad’s service discovery support. I promised to follow-up showing this with Consul providing the service registry and use Consul’s service discovery.

Le’s look into the service stanza documentation. The provider key allows 2 different values. We already used nomad implementing the Nomad service registry. Before Nomad 1.3 only consul was a supported service registry. This was the default for this key and would register the service with the Consul service registry. So we would go ahead and just would set provider = "consul" for the api and the db service.

But really that would not make up for a valuable post. Though it certainly demonstrates the powerfulness of Nomad.

For Nomad 1.4 it was announced that it will extend their service registry implementation, so it will be able to offer service health checks. I can only assume that these health checks will very likely look similar to the implementation provided by the Consul integration. However, I will add a service check and we will see if Nomad 1.4 will come up with something different as soon it was released.

service {
    name = "continuousdelivery-db"
    provider = "consul"
    port = "postgresdb"
    tags = [ "db" ]
    check {
        name     = "db-check"
        type     = "tcp"
        interval = "60s"
        timeout  = "4s"
        check_restart {
            grace = "20s"
        }
    }
}

service {
    name        = "continuousdelivery-api"
    provider    = "consul"
    port        = "http"
    tags        = [ "api" ]
    check {
        name     = "continuousdelivery-api-check"
        type     = "http"
        port     = "management_http"
        path     = "/actuator/health"
        interval = "60s"
        timeout  = "10s"
        check_restart {
            grace = "60s"
        }
    }
}

A service check can provide a kind of health check for indicating that a service is able to accept requests. With that health status, the service registry can filter service to healthy ones during service discovery so only healthy service instances are returned. Here we have added different types checks. For the database service we only check if a TCP socket is active while for the api service we use a http check that is able to call HTTP URLs.

After updating the service registry to be based on Consul we also have to update the service discovery part in the template stanza that was generating the environment variables for the application configuration.

template {
    destination="application.env"
    env = true
    data = <<EOH
    SPRING_DATASOURCE_URL=jdbc:postgresql://{{ range service "continuousdelivery-db" }}{{ .Address }}:{{ .Port }}{{ end }}/app (1)
    SPRING_DATASOURCE_USERNAME="spring"
    SPRING_DATASOURCE_PASSWORD="boot"
    EOH
}

1	We used the HCL function `nomadService` for querying the Nomad service registry before and modified it to use `service` to query the Consul service registry.

Before we can deploy this updated job specification we have to add a Consul instance so Nomad can register services with Consul. First install the Consul software, then run the Consul agent using the configuration below. I have chosen to run the Consul agent in dev mode which runs the agent in server and client mode as a single process.

server.hcl

#See https://www.consul.io/docs/agent#client-node-with-multiple-interfaces-or-ip-addresses
# Binding to private IP address for accessibility from container
# Binding to loopback IP address for (simple default) accessibility by nomad client(s)
client_addr = "{{ GetPrivateInterfaces | exclude \"type\" \"ipv6\" | join \"address\" \" \" }} 127.0.0.1"

For starting the Consul server run the following command.

server.hcl

consul agent -dev -config-file server.hcl

Let’s deploy the updated job specification.

nomad job run \ (1)
    --var-file credentials.hcl \ (2)
    --var-file application.hcl \ (3)
    continuousdelivery.hcl

It will take a while until all services have been updated and are showing healthiness. Consul has a web console built-in that is served by the running agent and we can open using the URL http://localhost:8500.

In case you wanted to view the complete code on GitHub, here is the link to the commits related to this post.

Expect more on that exciting stuff to be published soon.