serviceprober

service-prober

A configurable Prometheus blackbox prober for complex services, e.g. those services whose conversations are too complex to be modeled by the simple expect-like semantics of the stock prometheus-blackbox-exporter.

It currently has modules for IMAP-based tests (round-trip message delivery checks), and for script-based HTTP tests suitable for interacting with complex web apps (though there is no Javascript support).

There is one major difference in how the service-prober works with respect to the stock prometheus-blackbox-prober: while the prometheus-blackbox-prober shares its configuration between the prober and Prometheus itself (prober defines probe types, the actual targets are defined in Prometheus configuration), the service-prober configuration completely defines all probes and all targets. To Prometheus, service-prober appears just like any other job to scrape.

Another difference with the prometheus-blackbox-prober is that here the probe execution intervals are not tied to the Prometheus scraping interval: instead, probes have their own execution schedule. This is because service probes can be heavyweight, and we want to stagger them across time instead of synchronizing them with the scrape.

Configuration

The service-prober configuration is meant to be compact and expressive, and provides a few features meant to facilitate the generation of a large number of probe permutations with different parameters.

The configuration must be in JSON format, and must consist of a top-level object containing two attributes:

• vars, to define arbitrary global variables;
• probes, a list of probe specifications.

Each probe specification is an object that can have the following attributes:

• type, specifying the probe type, which must be one of the supported probe types (see below);
• name, the probe name, must be unique across all probes to let Prometheus distinguish them;
• interval, how often to execute the probe;
• timeout, the timeout on each probe execution;
• loop, a list of variable names used to generate probe permutations (see Permutations below);
• params, an object containing type-specific parameters to configure the exact behavior of the probe.

Variable expansion

The service-prober configuration supports variable expansion in all string values. The syntax is shell-like, ${variable}, and it is possible to navigate hierarchies of objects using dot-notation to separate attributes, e.g. given the following variables:

{
  "servers": {
    "foo": {
      "ip": "1.2.3.4",
      "zone": "us"
    },
    "bar": {
      "ip": "2.3.4.5",
      "zone": "eu"
    }
  }
}

one could write ${servers.foo.ip} to obtain 1.2.3.4. Note that there is no syntax to access array (list) elements, only objects.

Referencing non-existing variables results in a fatal error.

Permutations

The loop attribute on probe specifications allows one to generate permutations of parameters, resulting in multiple probes being created out of a single (parameterized) specification.

Each member of the loop list should reference a variable from vars (using the dot-notation to navigate hierarchies, as outlined in the Variable expansion section), which must be an array. The values of this array will be used to replace the original variable, each time generating a new probe.

This is best clarified with examples. Suppose we have the following configuration, using a hypothetical ping probe type (which does not exist, but whose behavior should be intuitive) for simplicity:

{
  "vars": {
    "servers": ["1.2.3.4", "2.3.4.5"]
  },
  "probes": [
    {
      "type": "ping",
      "name": "ping/${servers}",
      "loop": ["servers"],
      "params": {
        "addr": "${servers}"
      }
    }
  ]
}

We have a servers variable with a list of IP addresses of our servers. We also define a ping probe, which uses a loop attribute and has some parameterized values: let's assume that params.addr points the probe at the IP address to ping.

What this does is generate two probes: for each value of the servers variable, service-prober creates a new global variable context where the servers variable is no longer the whole array, but just the selected value. This lets us use ${servers} in the probe attributes, which will evaluate to the different values of the original servers array every time. Thus we'll have a probe named ping/1.2.3.4, with params.addr set to 1.2.3.4, and another probe named ping/2.3.4.5 which points instead at 2.3.4.5.

(Yes, the pluralization issue with servers is a bit annoying).

The arrays used for permutations don't just have to be strings, they can be complex objects.

If multiple loop variables are used, probes will be created for all possible permutations of the combination of parameters.

To exemplify the previous two points, let's examine a slightly more complex configuration, this time using the imap_login probe (which actually exists), and trying to test IMAP user login on two different servers with two separate user accounts:

{
  "vars": {
    "servers": ["imap1.example.com", "imap2.example.com"],
    "credentials": [
      {
        "username": "[email protected]",
        "password": "password1"
      }, {
        "username": "[email protected]",
        "password": "password2"
      }
    ]
  },
  "probes": [
    {
      "type": "imap_login",
      "name": "imap_login/${servers}/${credentials.username}",
      "loop": ["servers", "credentials"],
      "params": {
        "addr": "${servers}:993",
        "username": "${credentials.username}",
        "password": "${credentials.password}"
      }
    }
  ]
}

This time we'll get 4 probes, one for each possible combination of the values of servers and credentials.

It is very important, as it is done in these examples, to always add the loop values to the name attribute of the probe specification. If this is not done, there will be multiple probes with the same name, violating the metric uniqueness requirements.

Probe types

imap_login

The imap_login probe just performs a simple IMAP login check, and closes the connection immediately after authentication. It is not terribly useful as this functionality is already provided by the prometheus-blackbox-prober, however it is provided for illustration purposes.

Configuring this probe requires the following params:

• dns_map, to override DNS results (see DNS overrides below)
• addr, host:port of the IMAP server to connect to
• username, for IMAP authentication
• password, for IMAP authentication
• ssl, SSL configuration for the IMAP connection (see SSL options below)

NOTE: this probe does not support STARTTLS, or other authentication mechanisms beyond SASL AUTH PLAIN.

imap_roundtrip

The imap_roundtrip probe performs a round-trip check for SMTP and IMAP: it will try to deliver a test email message via authenticated SMTP, and will monitor an IMAP mailbox until it is received (or the timeout expires).

The configuration requires the following params:

• dns_map, to override DNS results (see DNS overrides below)
• message_body, overrides the default test message body
• smtp, with SMTP-related parameters:
  • addr, host:port of the SMTP server to connect to
  • username, for SMTP authentication
  • password, for SMTP authentication
  • ssl, SSL configuration for the SMTP connection (see SSL options below)
• imap, with IMAP-related parameters:
  • addr, host:port of the IMAP server to connect to
  • username, for IMAP authentication
  • password, for IMAP authentication
  • ssl, SSL configuration for the IMAP connection (see SSL options below)
  • expected_folder, the IMAP folder to open (default INBOX)
  • expected_headers, an associative array of header / regexp pairs, to run checks on the delivered message

NOTE: this probe does not support STARTTLS, not for IMAP nor for SMTP. Additionally, only the AUTH PLAIN mechanism is supported.

The expected_headers attribute can be used for implicit pipeline testing, by verifying for instance that the messages are DKIM-signed, or that they went through the spam filters, etc.:

{
  "probes": [
    {
      "type": "imap_roundtrip",
      "name": "imap-roundtrip-example",
      "params": {
        "imap": {
          "addr": "imap.example.com:993",
          "username": "username",
          "password": "password",
          "expected_headers": {
            "DKIM-Signature: "^v=1;.*d=example.com; s=selector;",
            "X-Spam-Status": "^[nN][oO]"
          },
        },
        "smtp": {
          "addr": "mail.example.com:465",
          "username": "username",
          "password": "password"
        }
      }
    }
  ]
}

http

The http prober can run a series of HTTP interactions, pretending to be a browser (to the extent that it keeps track of cookies across requests), and verify that they proceed according to the configured expectations.

The prober executes a script, consisting of multiple steps. At every step, it is possible to make new requests, click links in the page (found using CSS selectors), or submit forms (same).

The probe configuration requires the following params:

• ssl, SSL configuration (see SSL options below)
• dns_map, to override DNS results (see DNS overrides below)
• script, the script to execute

The script is a list of steps, each supporting the following attributes:

• type, the step type, one of open (discard current state, request a new page), click (find a link on the page and click it), or submit (submit a form on the page, with the desired values)
• url, the URL to request when the step type is open
• method, used with the open step type (default GET)
• headers, used with the open step type, is an associative array of additional headers to be set in the HTTP request
• basic_auth, used with the open step type, is an associative array with username and password fields, which sets the Authorization header in the HTTP request
• selector, the CSS selector to use when the step type is click or submit, to identify the A or FORM element respectively
• form_values, an associative array of form values to submit, when the step type is submit
• expected_url, when present, is a regular expression checked against the URL of the current page at the end of the step (after redirects etc)
• expected_data, when present, should be a regular expression that will be matched against the body of the current page, at the end of the step.

So, for instance, we could define a probe to log into a hypothetical web application:

{
  "vars": {
    "username": "[email protected]",
    "password": "password1"
  },
  "probes": [
    {
      "type": "http",
      "name": "pannello",
      "interval": "10s",
      "timeout": "10s",
      "params": {
        "script": [
          {
            "type": "open",
            "url": "https://webapp.example.com/",
            "expected_url": "https://webapp.example.com/login"
          }, {
            "type": "submit",
            "selector": ".form-signin",
            "form_values": {
              "username": "${username}",
              "password": "${password}"
            },
            "expected_url": "https://webapp.example.com/",
            "expected_data": "<title>[^<]*Webapp"
          }, {
            "type": "click",
            "selector": "a.logout-link",
            "expected_data": "Successfully Logged Out"
          }
        ]
      }
    }
  ]
}

The above probe will try to access webapp.example.com, expect a login form, fill it in with username and password, navigate to the web app itself, and log out (of course the example is using sample CSS selectors, the interaction details will be very specific to the application).

If you don't need the CSS-selector features, you can also just use the http probe to make simple requests with the open step.

openvpn_ping

The openvpn_ping prober attempts to connect to an openvpn server, and after performing a successful handshake it will attempt to send and receive ICMP Echo packets through the tunnel to verify that the gateway can route traffic.

The probe configuration requires the following params. Most of them belong to the subset of openvpn configuration options understood by the minimal implementation of the protocol we use:

• pingTarget, the IP that should be pinged through the VPN gateway
• remote, the IP of the OpenVPN remote,
• proto, the protocol in use (either tcp or udp)
• port, the port the OpenVPN server is listening on
• cipher, the preferred --data-cipher for the VPN data channel.
• auth, the HMAC algorithm used to authenticate the packets in the data channel. This is ignored if an AEAD cipher is picked.
• cert, the path to a valid certificate that the client will use to athenticate against the service.
• key, the path to a valid key that the client will use to authenticate against the service
• ca, the path to the certificate authority that the client will use to authenticate the server.

As an example, we can then combine the variable expansion capabilities to define a parametrized probe that will check several remotes on different ports and protocols:

{
  "vars": {
    "remotes": [
      {
        "ip": "10.10.10.10",
        "proto": "tcp",
        "port": "1194"
      },
      {
        "ip": "11.11.11.11",
        "proto": "udp",
        "port": "1194"
      }
    ]
  },
  "probes": [
    {
      "type": "openvpn_ping",
      "name": "openvpn_ping/${remotes.ip}:${remotes.port}/${remotes.proto}",
      "loop": [
        "remotes"
      ],
      "params": {
        "pingTarget": "1.1.1.1",
        "remote": "${remotes.ip}",
        "proto": "${remotes.proto}",
        "port": "${remotes.port}",
        "cipher": "AES-256-GCM",
        "auth": "SHA1",
        "cert": "/path/to/cert.pem",
        "key": "/path/to/cert.pem",
        "ca": "/path/to/ca.crt"
      }
    }
  ]
}

Common configuration features

DNS overrides

All probes provide the capability to override DNS results, by specifying a dns_map, which consists of a simple set of hostname / IP address pairs. The service-prober will look up hostnames in the dns_map before going to DNS, so this mechanism can be used to target specific servers, e.g.:

{
  "vars": {
    "servers": ["1.2.3.4", "2.3.4.5"]
  },
  "probes": [
    {
      "type": "imap_login",
      "name": "imap_login/${servers}",
      "loop": ["servers"],
      "dns_map": {
        "imap.example.com": "${servers}"
      },
      "params": {
        "addr": "imap.example.com:993",
        "username": "[email protected]",
        "password": "password1"
      }
    }
  ]
}

This configuration creates two probes, both will attempt to log in to "imap.example.com", but each one will resolve that name to a different IP address from servers. However, SSL validation will be performed correctly with a server_name of "imap.example.com".

SSL options

By default, service-prober will validate all SSL connections using the system-wide CA roots, and will perform CN / subjectAltName validation of the certificates.

All SSL connections can be configured with the same set of options, specified as an object with the following attributes:

• ca, a file containing a CA certificate in PEM format, used to validate the server's certificate
• cert, key, specifying a client certificate and private key
• server_name, to control SNI and to override the expected server name for validation purposes
• skip_validation, set to true to disable all client-side SSL validation

By default, the server_name is set to the hostname used for the connection, so there's no need to override it when using DNS remappings to target the connection at a specific IP.

Usage

Building

With a recent Go version installed, build the service-prober binary with:

go build ./cmd/service-prober

Running

You can check all the available options with

service-prober --help

Most importantly you'll need the --config option to point at your configuration.

The service-prober binary has a few operation modes worth mentioning, which might help in writing a new configuration. It is possible to just run a configuration check with the --validate option:

service-prober --config=my-config.json --validate

The exit status will reflect the configuration correctness.

You can also ask the tool, with the --oneshot option, to simply run all probes once, and then exit, without starting a HTTP server to export metrics at all. This is useful to check that a configuration actually performs as expected, before putting it in production. In this scenario, to run just a subset of the probes you can use the --only option and a regular expression, e.g.:

service-prober --config=my-config.json --oneshot --only=^imap

The above command will only run the probes whose name starts with "imap".