Zerolog API

Forwarder

zerolog.forwarder.start_forwarder(pub_port, receive_port, mon_port=None, backend_socket=None, frontend_socket=None)

Start a zeromq proxy for forwarding messages from TCP socket to zmq PUB socket

Parameters:
  • pub_port (int) – port number to use for publishing messages to workers
  • receive_port (int) – port number to use for receiving messages
  • mon_port (optional) (int) – port to use for monitor socket
  • backend_socket (optionnal) (str) – socket type to use for backend socket
  • frontend_socket (optionnal) (str) – socket type to use for frontend socket

Receiver

Reciver module

Contain default receiver. Receiver will be in charge to get data from forwarder

class zerolog.receiver.Receiver(forwarder_address, forwarder_port, topic=None, *args, **kwargs)

Bases: object

Main receiver class.

You can use different patterns for receiver and workers :

  • Workers connect to receiver over TCP
  • Workers connect to receiver over ipc protocol (same machine)

For more flexibility this class allow you to pass an output_port parameter or a output_socket, the last one allowing you to use ipc protocol between workers and receiver.

Warning

If both output_port and output_socket are set, an error will be raised. If none are set, default is to use TCP bind on port 6200

Goal of this class

This class is very basic since there is no specific action here, only data data forwarding from forwarder to workers. Yet it’s configurable enough to cover major use cases, and can be easily override.

But keep in mind that for large application this class will process a very large amount of data. Keep it simple and put logic in workers, there here for that prupose

Parameters:
  • forwarder_address (str) – IP or domain of forwarder
  • forwarder_port (int) – port to listen for incomming forwarder messages
  • topic (str) – publish topic to listen (default to all)
  • output_port (int) – port to bind for sending data to workers
  • output_socket (str) – location of ipc socket to use for sending data to workers
  • logging_config (str) – file path to logging configuration
Raise:

TypeError
recv_data()

Receive data from forwarder and return them. Channel will not be return

Returns:data from forwarder
Return type:bytes
run()

Main receiver loop

setup_output_socket(output_port=None, output_socket=None)

Setup PUSH output socket

Worker

Base worker implmentation

class zerolog.worker.BaseWorker(backend, *args, **kwargs)

Bases: object

Base worker class.

This class cannot be used “as is”, it will raises an ImplementationError in process_data methode. BaseWorker provide only a skeleton to help you to develop your own workers

Note

For conveniance, default recv method for backend socket is recv_string() from pyzmq. But you can change it easily by overloading recv_func in your worker, for example:

def __init__(self, backend, *args, **kwargs):
    super(MyWorkerClase, self).__init__(backend, *args, **kwargs)
    self.recv_func = self.backend.recv_json()
Parameters:backend (str) – backend zeromq string to connect to receiver. e.g: ipc://unix.socket
process_data(data)

Process data

Parameters:data (mixed) – data received from backend
Raises:NotImplementedError
run()

Main loop for receive messages and process them

self.recv_func is used to receive data from backend