Federated Alerta

Alerta servers can forward alerts, actions and deletes to other Alerta servers or downstream systems using the forwarder built-in plugin.

In this way, Alerta servers can be organised in a hierarchy of sub-ordinate servers forwarding to servers above them to create a hierarchy of Alerta servers.

Also individual Alerta servers can be kept in sync with others each other to create a “active-active” architectures where any server can be used as the “primary” and if a failure occurs any other server can take over.

Alerta can be configured to send some or all of the following:

  • received alerts

  • all alert actions or just specific actions. eg. ack, shelve, open, close

  • custom alert actions eg. createIssue or ringBell

  • alert deletions

Note

Forwarding heartbeats is not currently possible but may be supported in a future release.

Configuration

BASE_URL

Set the BASE_URL to the public endpoint that will be used by the web UI, alert generating systems, and forwarding Alerta servers

PLUGINS

Simply add the fowarder plugin to this list of configured plugins to enable and load it when the API starts.

FWD_DESTINATIONS

Forwarder destinations have three parts:

  • remote BASE_URL

  • authentication credentials

  • forwarding filter

Example

FWD_DESTINATIONS = [
    ('http://localhost:9000', {'username': 'user', 'password': 'pa55w0rd', 'timeout': 10}, ['alerts', 'actions']),  # BasicAuth
    # ('https://httpbin.org/anything', dict(username='foo', password='bar', ssl_verify=False), ['*']),
    ('http://localhost:9001', {
        'key': 'e3b8afc0-db18-4c51-865d-b95322742c5e',
        'secret': 'MDhjZGMyYTRkY2YyNjk1MTEyMWFlNmM3Y2UxZDU1ZjIK'
    }, ['actions']),  # Hawk HMAC
    ('http://localhost:9002', {'key': 'demo-key'}, ['delete']),  # API key
    ('http://localhost:9003', {'token': 'bearer-token'}, ['*']),  # Bearer token
]

Forwarding Loops

The configured remote BASE_URL must match the BASE_URL as set at the remote server otherwise forwarding loops are possible (see next).

There is no problem with Alerta servers forwarding alerts and actions to each other. This is the basis of an “active-active” failover configuration.

However, if misconfigured, servers can endlessly forward to each other and back again which would eventually lead to resource exhaustion and failure.

The solution is to borrow the concept of a “forwarding loop” header from mail servers. If the X-Alerta-Loop header contains the Alerta server name (ie BASE_URL) of the receiving server then the receiving server should not process the request.

And if an alert or action is to be forwarded to a remote Alerta server that is already in the “forwarding loop” header then that server should be skipped.

https://qmail.mivzakim.net/qmail-manual-html/misc/RFCLOOPS.html

Authentication

Alerta supports multiple authentication methods for the remote systems when forwarding:

  • HMAC

  • API Key

  • BasicAuth

  • Bearer token

The recommended authentication method for Alerta-to-Alerta forwarding is HMAC because it is the most secure, it does not require a user to be associated with the credentials and the credentials do not expire.

The other authentication methods are available for use when forwarding to non-Alerta systems.

To generate HMAC key and secret, it is useful to use a UUID for the key and base64 encoded string for the secret so that they are visibly different.

On macOs, run

1$ uuidgen | tr '[:upper:]' '[:lower:]'         <= create HMAC "key"
258e7c66f-b990-4610-9496-60eb3c63339b
3
4$ date | md5 | base64                        <= create HMAC "secret"
5MzVlMzQ5NWYzYWE2YTgxYTUyYmIyNDY0ZWE2ZWJlYTMK

Forwarding Filters

The types of entities to be forwarded are configurable:

  • * - everything ie. alerts, all actions (incl. custom), deletes

  • alerts - alerts

  • actions - all actions

  • individual actions - eg. ack

  • custom actions eg. createIssue

  • delete

Examples

['*']
['alerts', 'ack', 'unack', 'close', 'delete']
['alerts', 'delete']

Non-Alerta Forwarding

It is possible to make use of Alerta forwarding to forward alerts to non-Alerta systems. However, any forwarding destination will need to implement the following endpoints:

  • POST /alert - alert receiver

  • PUT /alert/{alertId}/action - action alerts

  • DELETE /alert/{alertId} - delete alerts

Responses should have HTTP 200 OK status code on success. HTTP response bodies are not parsed so they will not effect the result. However, it is good practise to add meaningful messages to the payloads which will be useful when debugging.

Troubleshooting

  • Enable detailed logging with DEBUG=True.

  • Use a dummy endpoint such as https://httpbin.org/anything

Future Enhancements

  • heartbeat forwarding

  • circuit-breaker retry logic

  • configurable endpoints