Random Data to CSV File

This generates random data, adds a timestamp, formats as CSV, and writes to a file. The default settings for the FileDataSink is to flush out to the file every 10 seconds.

recipes/randomwalk_timestamp_csv_file.config

[datalogd]
connection_graph =
  digraph {
    a [class=RandomWalkDataSource];
    b [class=TimeStampDataFilter];
    d [class=CSVDataFilter, labels=None];
    s [class=FileDataSink, filename="randomwalk.csv"];
    a -> b -> d -> s;
  }

$ datalogd -c recipes/randomwalk_timestamp_csv_file.config

INFO:main:Initialising DataLogDaemon.
INFO:DataLogDaemon:Loaded config from: /etc/xdg/datalogd/datalogd.conf, recipes/randomwalk_timestamp_csv_file.config
INFO:pluginlib:Loading plugins from standard library
INFO:DataLogDaemon:Detected source plugins: NullDataSource, LibSensorsDataSource, RandomWalkDataSource, SerialDataSource
INFO:DataLogDaemon:Detected filter plugins: NullDataFilter, AggregatorDataFilter, CSVDataFilter, KeyValDataFilter, TimeStampDataFilter
INFO:DataLogDaemon:Detected sink plugins: NullDataSink, FileDataSink, InfluxDBDataSink, LoggingDataSink, MatplotlibDataSink, PrintDataSink
INFO:DataLogDaemon:Initialising node a:RandomWalkDataSource()
INFO:DataLogDaemon:Initialising node b:TimeStampDataFilter()
INFO:DataLogDaemon:Initialising node d:CSVDataFilter(labels=None)
INFO:DataLogDaemon:Initialising node s:FileDataSink(filename=randomwalk.csv)
INFO:DataLogDaemon:Connecting a:RandomWalkDataSource -> b:TimeStampDataFilter
INFO:DataLogDaemon:Connecting b:TimeStampDataFilter -> d:CSVDataFilter
INFO:DataLogDaemon:Connecting d:CSVDataFilter -> s:FileDataSink
INFO:main:Starting event loop.

$ cat randomwalk.csv

2020-04-17 22:02:51.887071+09:30,-1.0,2020-04-17 22:02:51.887071+09:30,-2.0
2020-04-17 22:02:52.888208+09:30,0.0,2020-04-17 22:02:52.888208+09:30,0.0
2020-04-17 22:02:53.889423+09:30,-1.0,2020-04-17 22:02:53.889423+09:30,-2.0
2020-04-17 22:02:54.889818+09:30,-1.0,2020-04-17 22:02:54.889818+09:30,-4.0
2020-04-17 22:02:55.891065+09:30,-2.0,2020-04-17 22:02:55.891065+09:30,-6.0
2020-04-17 22:02:56.892261+09:30,-3.0,2020-04-17 22:02:56.892261+09:30,-8.0
2020-04-17 22:02:57.893478+09:30,-4.0,2020-04-17 22:02:57.893478+09:30,-10.0
2020-04-17 22:02:58.894673+09:30,-3.0,2020-04-17 22:02:58.894673+09:30,-12.0
2020-04-17 22:02:59.895897+09:30,-2.0,2020-04-17 22:02:59.895897+09:30,-14.0
...

There are no labels for the column headers, and the file will grow infinitely large with time. It might be better to aggregate the data and update the file with only the latest data. Note the use of mode="w" and flush_interval=None, which causes the file to be opened, written, and closed on each reciept of data. In this way, each block of aggregated data will overwrite the old one.

recipes/randomwalk_timestamp_aggregator_csv_file.config

[datalogd]
connection_graph =
  digraph {
    a [class=RandomWalkDataSource];
    b [class=TimeStampDataFilter];
    c [class=AggregatorDataFilter, buffer_size=3600, send_every=10];
    d [class=CSVDataFilter];
    s [class=FileDataSink, filename="randomwalk.csv", mode="w", flush_interval=None];
    a -> b -> c -> d -> s;
  }

$ datalogd -c recipes/randomwalk_timestamp_aggregator_csv_file.config

INFO:main:Initialising DataLogDaemon.
INFO:DataLogDaemon:Loaded config from: /etc/xdg/datalogd/datalogd.conf, recipes/randomwalk_timestamp_aggregator_csv_file.config
INFO:pluginlib:Loading plugins from standard library
INFO:DataLogDaemon:Detected source plugins: NullDataSource, LibSensorsDataSource, RandomWalkDataSource, SerialDataSource
INFO:DataLogDaemon:Detected filter plugins: NullDataFilter, AggregatorDataFilter, CSVDataFilter, KeyValDataFilter, TimeStampDataFilter
INFO:DataLogDaemon:Detected sink plugins: NullDataSink, FileDataSink, InfluxDBDataSink, LoggingDataSink, MatplotlibDataSink, PrintDataSink
INFO:DataLogDaemon:Initialising node a:RandomWalkDataSource()
INFO:DataLogDaemon:Initialising node b:TimeStampDataFilter()
INFO:DataLogDaemon:Initialising node c:AggregatorDataFilter(buffer_size=3600, send_every=30)
INFO:DataLogDaemon:Initialising node d:CSVDataFilter()
INFO:DataLogDaemon:Initialising node s:FileDataSink(filename=randomwalk.csv)
INFO:DataLogDaemon:Connecting a:RandomWalkDataSource -> b:TimeStampDataFilter
INFO:DataLogDaemon:Connecting b:TimeStampDataFilter -> c:AggregatorDataFilter
INFO:DataLogDaemon:Connecting c:AggregatorDataFilter -> d:CSVDataFilter
INFO:DataLogDaemon:Connecting d:CSVDataFilter -> s:FileDataSink
INFO:main:Starting event loop.

$ cat randomwalk.csv

timestamp,analog_randomwalk0,timestamp,analog_randomwalk1
2020-04-17 22:27:36.264577+09:30,1.0,2020-04-17 22:27:36.264577+09:30,0.0
2020-04-17 22:27:37.265669+09:30,1.0,2020-04-17 22:27:37.265669+09:30,2.0
2020-04-17 22:27:38.266249+09:30,2.0,2020-04-17 22:27:38.266249+09:30,0.0
2020-04-17 22:27:39.267433+09:30,3.0,2020-04-17 22:27:39.267433+09:30,-2.0
2020-04-17 22:27:40.268600+09:30,2.0,2020-04-17 22:27:40.268600+09:30,0.0
2020-04-17 22:27:41.269554+09:30,2.0,2020-04-17 22:27:41.269554+09:30,0.0
2020-04-17 22:27:42.270715+09:30,3.0,2020-04-17 22:27:42.270715+09:30,-2.0
2020-04-17 22:27:43.271873+09:30,4.0,2020-04-17 22:27:43.271873+09:30,0.0
2020-04-17 22:27:44.272887+09:30,4.0,2020-04-17 22:27:44.272887+09:30,0.0
2020-04-17 22:27:45.274042+09:30,3.0,2020-04-17 22:27:45.274042+09:30,-2.0
2020-04-17 22:27:46.275201+09:30,2.0,2020-04-17 22:27:46.275201+09:30,0.0
2020-04-17 22:27:47.276220+09:30,1.0,2020-04-17 22:27:47.276220+09:30,0.0
2020-04-17 22:27:48.277396+09:30,1.0,2020-04-17 22:27:48.277396+09:30,0.0
2020-04-17 22:27:49.278583+09:30,0.0,2020-04-17 22:27:49.278583+09:30,-2.0
2020-04-17 22:27:50.279763+09:30,1.0,2020-04-17 22:27:50.279763+09:30,-2.0
2020-04-17 22:27:51.280956+09:30,0.0,2020-04-17 22:27:51.280956+09:30,-2.0
2020-04-17 22:27:52.282120+09:30,0.0,2020-04-17 22:27:52.282120+09:30,-2.0
2020-04-17 22:27:53.283198+09:30,0.0,2020-04-17 22:27:53.283198+09:30,-2.0
2020-04-17 22:27:54.284388+09:30,1.0,2020-04-17 22:27:54.284388+09:30,-2.0
2020-04-17 22:27:55.285719+09:30,0.0,2020-04-17 22:27:55.285719+09:30,-4.0
2020-04-17 22:27:56.286249+09:30,0.0,2020-04-17 22:27:56.286249+09:30,-4.0
2020-04-17 22:27:57.287438+09:30,-1.0,2020-04-17 22:27:57.287438+09:30,-6.0
2020-04-17 22:27:58.288604+09:30,-2.0,2020-04-17 22:27:58.288604+09:30,-8.0
2020-04-17 22:27:59.289765+09:30,-2.0,2020-04-17 22:27:59.289765+09:30,-10.0
2020-04-17 22:28:00.290927+09:30,-1.0,2020-04-17 22:28:00.290927+09:30,-10.0
2020-04-17 22:28:01.292095+09:30,0.0,2020-04-17 22:28:01.292095+09:30,-12.0
2020-04-17 22:28:02.292911+09:30,0.0,2020-04-17 22:28:02.292911+09:30,-10.0
2020-04-17 22:28:03.294073+09:30,0.0,2020-04-17 22:28:03.294073+09:30,-12.0
2020-04-17 22:28:04.295262+09:30,-1.0,2020-04-17 22:28:04.295262+09:30,-12.0
2020-04-17 22:28:05.296247+09:30,-2.0,2020-04-17 22:28:05.296247+09:30,-10.0
...