Running elements

Zerolog provide command line tool to start every element. This section will explain how to start each element of zerolog

Running forwarder

The forwarder is in charge of receiving logs from external services and to forward them to receivers. Thanks to zeromq PUB/SUB pattern, you can also use topic to filter messages. In this way you can associate receivers with specifics logs messages.

Like for all elements, zerolog provide a command to start a forwarder :

usage: zerolog forwarder [-h] [-f FRONTEND] [-b BACKEND] [-m MONITOR]

optional arguments:
    -h, --help            show this help message and exit
    -f FRONTEND, --frontend FRONTEND
                          frontend port for sending data to receivers
    -b BACKEND, --backend BACKEND
                          backend port for receiving data
    -m MONITOR, --monitor MONITOR
                          monitor port for publishing data

Forwarder’s parameters are all optional and will use default values if not provided, so you can run a forwarder using only :

zerolog forwarder

Only monitor socket will not be set if the -m option is not provided. The monitor socket only role is to output messages received by the forwarder, so you can easily check if logs messages are able to hit the forwarder and are correctly sending back messages to receivers.

Running receiver

The receiver is in charge to filter messages incomming from forwarder and dispatch them to a pool of workers. By default, receiver will not filter on any topic and will receive all messages from forwarder.

Command usage :

usage: zerolog receiver [-h] [-t TOPIC] [-l LOG_CONFIG]
                     [--output-port OUTPUT_PORT | --output-socket OUTPUT_SOCKET]
                     forwarder_address forwarder_port

positional arguments:
forwarder_address     address of running forwarder
forwarder_port        port of running forwarder

optional arguments:
  -h, --help         show this help message and exit
  -t TOPIC, --topic TOPIC
                     optional topic to subscribe
  -l LOG_CONFIG, --log-config LOG_CONFIG
                     location of logging configuration file (python
                     standard logging configuration file)
  --output-port OUTPUT_PORT
                     output TCP port for workers (default to 6200)
  --output-socket OUTPUT_SOCKET
                     output unix socket for workers

forwarder_address and forwarder_port are both mandatory. The first one need to be a valid address for connecting to your forwarder (e.g: "127.0.0.1" and the second one is the frontend port of your forwarder.

topic parameter is optional and default is to accept any message from forwarder. But you can configure your can use specific topic for filtering messages, for example if you only want nginx logs for this receiver. This allow you to use only one forwarder for all messages and all processes.

--log-config parameter is pretty obvious here. This one can be usefull for sending an email, or logging to sentry (for example) if the receiver crash.

output-port and output-socket parameters will be used to bind the ventilator socket (the one in charge of sending messages to workers). You can’t set them both, but you can choose between unix socket (if workers are on the same machine for example) or classic TCP port

Running workers

The worker is in charge of processing logs messages and apply your own logic on them.

Command :

usage: zerolog worker [-h] [-d DIRECTORY] backend worker_class

positional arguments:
    backend               backend to connect for receiving message (e.g:
                          tcp://127.0.0.1:6200)
    worker_class          class to use as worker

optional arguments:
    -h, --help            show this help message and exit
    -d DIRECTORY, --directory DIRECTORY
                          directory to append to sys.path for import (optional)

As you can see, this command aim to allow you to start your own workers. Only restriction is that your workers must take a backend parameter in __init__ method, and have a run method.

For simplicity you can simply extends the BaseWorker class avaible in zerolog.

backend parameter here is a full zeromq connection string, allowing you to connect to any backend (TCP or unix socket)

worker_class represent a full import path to the wanted worker class (e.g: zerolog.worker.BaseWorker.

--directory parameter allow you to append a specific directory to python path. With this you can use a worker_class from anywhere and without installing anything, a simple python file could be used to declare your worker