Zerolog API¶
Forwarder¶
-
zerolog.forwarder.
start_forwarder
(pub_port, receive_port, mon_port=None, backend_socket=None, frontend_socket=None)¶ Start a zeromq proxy for forwarding messages from TCP socket to zmq PUB socket
Parameters: - pub_port (int) – port number to use for publishing messages to workers
- receive_port (int) – port number to use for receiving messages
- mon_port (optional) (int) – port to use for monitor socket
- backend_socket (optionnal) (str) – socket type to use for backend socket
- frontend_socket (optionnal) (str) – socket type to use for frontend socket
Receiver¶
Reciver module
Contain default receiver. Receiver will be in charge to get data from forwarder
-
class
zerolog.receiver.
Receiver
(forwarder_address, forwarder_port, topic=None, *args, **kwargs)¶ Bases:
object
Main receiver class.
You can use different patterns for receiver and workers :
- Workers connect to receiver over TCP
- Workers connect to receiver over ipc protocol (same machine)
For more flexibility this class allow you to pass an
output_port
parameter or aoutput_socket
, the last one allowing you to useipc
protocol between workers and receiver.Warning
If both
output_port
andoutput_socket
are set, an error will be raised. If none are set, default is to use TCP bind on port 6200Goal of this class
This class is very basic since there is no specific action here, only data data forwarding from forwarder to workers. Yet it’s configurable enough to cover major use cases, and can be easily override.
But keep in mind that for large application this class will process a very large amount of data. Keep it simple and put logic in workers, there here for that prupose
Parameters: - forwarder_address (str) – IP or domain of forwarder
- forwarder_port (int) – port to listen for incomming forwarder messages
- topic (str) – publish topic to listen (default to all)
- output_port (int) – port to bind for sending data to workers
- output_socket (str) – location of ipc socket to use for sending data to workers
- logging_config (str) – file path to logging configuration
Raise: TypeError
-
recv_data
()¶ Receive data from forwarder and return them. Channel will not be return
Returns: data from forwarder Return type: bytes
-
run
()¶ Main receiver loop
-
setup_output_socket
(output_port=None, output_socket=None)¶ Setup PUSH output socket
Worker¶
Base worker implmentation
-
class
zerolog.worker.
BaseWorker
(backend, *args, **kwargs)¶ Bases:
object
Base worker class.
This class cannot be used “as is”, it will raises an
ImplementationError
inprocess_data
methode.BaseWorker
provide only a skeleton to help you to develop your own workersNote
For conveniance, default recv method for backend socket is
recv_string()
from pyzmq. But you can change it easily by overloadingrecv_func
in your worker, for example:def __init__(self, backend, *args, **kwargs): super(MyWorkerClase, self).__init__(backend, *args, **kwargs) self.recv_func = self.backend.recv_json()
Parameters: backend (str) – backend zeromq string to connect to receiver. e.g: ipc://unix.socket
-
process_data
(data)¶ Process data
Parameters: data (mixed) – data received from backend Raises: NotImplementedError
-
run
()¶ Main loop for receive messages and process them
self.recv_func
is used to receive data from backend
-