Plugins

This section describes how to create plugins, specify the connections between them, and then run the complete data logging pipeline.

Plugins are python classes which extend one of the base plugin types. The class can be defined in any python source code file, and multiple plugin classes may be defined in a single file. The directory containing plugin source code can be specified in a configuration file, or in command line parameters.

Data Sources

Data source classes must extend DataSource. In addition, their class name must have the suffix “DataSource”.

The following code is a simple example of a functional data source plugin. It sends the string “Hello, World!” to any connected sinks once every 10 seconds:

plugin_demos/helloworld_datasource.py

import asyncio
from datalogd import DataSource

# Must extend datalogd.DataSource, and class name must have DataSource suffix
class HelloWorldDataSource(DataSource):
    def __init__(self, sinks=[]):
        # Do init of the superclass (DataSource), connect any specified sinks
        super().__init__(sinks=sinks)
        # Queue first call of update routine
        asyncio.get_event_loop().call_soon(self.say_hello)

    def say_hello(self):
        "Send ``Hello, World!`` to connected sinks, then repeat every 10 seconds."
        self.send("Hello, World!")
        asyncio.get_event_loop().call_later(10, self.say_hello)

Note the use of the asyncio event loop to schedule method calls, as opposed to a separate loop/sleep thread or similar.

Data sources will have their close() method called when the application is shutting down so that any resources (devices, files) used can be released.

Data Sinks

Data sinks accept data using their receive() method, and do something with it, such as committing it to a database. They must extend DataSink, and must use the suffix “DataSink”.

A simple example data sink which prints received data in uppercase to the console is:

plugin_demos/shout_datasink.py

from datalogd import DataSink

# Must extend datalogd.DataSink, and class name must have DataSink suffix
class ShoutDataSink(DataSink):
    # Override the receive() method to do something useful with received data
    def receive(self, data):
        "Accept ``data`` and shout it out to the console."
        print(str(data).upper())

Data sinks will have their close() method called when the application is shutting down so that any resources (devices, files) used can be released.

Data Filters

Data filters are in fact both DataSources and DataSinks, and thus share the functionality of both. They must extend DataFilter, and must use the suffix “DataFilter”.

The NullDataFilter is the most simple example of a filter, which accepts data and passes it unchanged to any connected sinks. A slightly more functional filter which performs some processing on the data is:

plugin_demos/helloworld_datafilter.py

from datalogd import DataFilter

# Must extend datalogd.DataFilter, and class name must have DataFilter suffix
class HelloWorldDataFilter(DataFilter):
    # Override the receive() method to do something useful with received data
    def receive(self, data):
        # Send modified data to all connected sinks
        self.send(str(data).replace("Hello", "Greetings"))

Connecting

To connect the above source, filter and sink to a complete “Hello, World!” data logger, the connection graph could be specified as:

digraph {
    source [class=HelloWorldDataSource];
    filter [class=HelloWorldDataFilter];
    sink   [class=ShoutDataSink];
    source -> filter -> sink;
}

Running

See the Command Line Parameters section for details on specifying configuration from the command line.

$ datalogd --plugindir plugin_demos --graph 'digraph{a[class=HelloWorldDataSource];b[class=HelloWorldDataFilter];c[class=ShoutDataSink];a->b->c;}'

INFO:main:Initialising DataLogDaemon.
INFO:DataLogDaemon:Loaded config from: /etc/xdg/datalogd/datalogd.conf
INFO:pluginlib:Loading plugins from standard library
INFO:pluginlib:Adding plugin_demos as a plugin search path
INFO:DataLogDaemon:Detected source plugins: NullDataSource, LibSensorsDataSource, RandomWalkDataSource, SerialDataSource, HelloWorldDataSource
INFO:DataLogDaemon:Detected filter plugins: NullDataFilter, AggregatorDataFilter, CSVDataFilter, KeyValDataFilter, TimeStampDataFilter, HelloWorldDataFilter
INFO:DataLogDaemon:Detected sink plugins: NullDataSink, FileDataSink, InfluxDBDataSink, LoggingDataSink, MatplotlibDataSink, PrintDataSink, ShoutDataSink
INFO:DataLogDaemon:Initialising node a:HelloWorldDataSource()
INFO:DataLogDaemon:Initialising node b:HelloWorldDataFilter()
INFO:DataLogDaemon:Initialising node c:ShoutDataSink()
INFO:DataLogDaemon:Connecting a:HelloWorldDataSource -> b:HelloWorldDataFilter
INFO:DataLogDaemon:Connecting b:HelloWorldDataFilter -> c:ShoutDataSink
INFO:main:Starting event loop.
GREETINGS, WORLD!
GREETINGS, WORLD!
GREETINGS, WORLD!
^CINFO:main:Stopping event loop.