Random Data to CSV File¶
This generates random data, adds a timestamp, formats as CSV, and writes to a
file. The default settings for the
FileDataSink
is to flush out to the file
every 10 seconds.
[datalogd]
connection_graph =
digraph {
a [class=RandomWalkDataSource];
b [class=TimeStampDataFilter];
d [class=CSVDataFilter, labels=None];
s [class=FileDataSink, filename="randomwalk.csv"];
a -> b -> d -> s;
}
$ datalogd -c recipes/randomwalk_timestamp_csv_file.config
INFO:main:Initialising DataLogDaemon.
INFO:DataLogDaemon:Loaded config from: /etc/xdg/datalogd/datalogd.conf, recipes/randomwalk_timestamp_csv_file.config
INFO:pluginlib:Loading plugins from standard library
INFO:DataLogDaemon:Detected source plugins: NullDataSource, LibSensorsDataSource, RandomWalkDataSource, SerialDataSource
INFO:DataLogDaemon:Detected filter plugins: NullDataFilter, AggregatorDataFilter, CSVDataFilter, KeyValDataFilter, TimeStampDataFilter
INFO:DataLogDaemon:Detected sink plugins: NullDataSink, FileDataSink, InfluxDBDataSink, LoggingDataSink, MatplotlibDataSink, PrintDataSink
INFO:DataLogDaemon:Initialising node a:RandomWalkDataSource()
INFO:DataLogDaemon:Initialising node b:TimeStampDataFilter()
INFO:DataLogDaemon:Initialising node d:CSVDataFilter(labels=None)
INFO:DataLogDaemon:Initialising node s:FileDataSink(filename=randomwalk.csv)
INFO:DataLogDaemon:Connecting a:RandomWalkDataSource -> b:TimeStampDataFilter
INFO:DataLogDaemon:Connecting b:TimeStampDataFilter -> d:CSVDataFilter
INFO:DataLogDaemon:Connecting d:CSVDataFilter -> s:FileDataSink
INFO:main:Starting event loop.
$ cat randomwalk.csv
2020-04-17 22:02:51.887071+09:30,-1.0,2020-04-17 22:02:51.887071+09:30,-2.0
2020-04-17 22:02:52.888208+09:30,0.0,2020-04-17 22:02:52.888208+09:30,0.0
2020-04-17 22:02:53.889423+09:30,-1.0,2020-04-17 22:02:53.889423+09:30,-2.0
2020-04-17 22:02:54.889818+09:30,-1.0,2020-04-17 22:02:54.889818+09:30,-4.0
2020-04-17 22:02:55.891065+09:30,-2.0,2020-04-17 22:02:55.891065+09:30,-6.0
2020-04-17 22:02:56.892261+09:30,-3.0,2020-04-17 22:02:56.892261+09:30,-8.0
2020-04-17 22:02:57.893478+09:30,-4.0,2020-04-17 22:02:57.893478+09:30,-10.0
2020-04-17 22:02:58.894673+09:30,-3.0,2020-04-17 22:02:58.894673+09:30,-12.0
2020-04-17 22:02:59.895897+09:30,-2.0,2020-04-17 22:02:59.895897+09:30,-14.0
...
There are no labels for the column headers, and the file will grow infinitely
large with time. It might be better to aggregate the data and update the file
with only the latest data. Note the use of mode="w"
and flush_interval=None
, which causes
the file to be opened, written, and closed on each reciept of data. In this way,
each block of aggregated data will overwrite the old one.
[datalogd]
connection_graph =
digraph {
a [class=RandomWalkDataSource];
b [class=TimeStampDataFilter];
c [class=AggregatorDataFilter, buffer_size=3600, send_every=10];
d [class=CSVDataFilter];
s [class=FileDataSink, filename="randomwalk.csv", mode="w", flush_interval=None];
a -> b -> c -> d -> s;
}
$ datalogd -c recipes/randomwalk_timestamp_aggregator_csv_file.config
INFO:main:Initialising DataLogDaemon.
INFO:DataLogDaemon:Loaded config from: /etc/xdg/datalogd/datalogd.conf, recipes/randomwalk_timestamp_aggregator_csv_file.config
INFO:pluginlib:Loading plugins from standard library
INFO:DataLogDaemon:Detected source plugins: NullDataSource, LibSensorsDataSource, RandomWalkDataSource, SerialDataSource
INFO:DataLogDaemon:Detected filter plugins: NullDataFilter, AggregatorDataFilter, CSVDataFilter, KeyValDataFilter, TimeStampDataFilter
INFO:DataLogDaemon:Detected sink plugins: NullDataSink, FileDataSink, InfluxDBDataSink, LoggingDataSink, MatplotlibDataSink, PrintDataSink
INFO:DataLogDaemon:Initialising node a:RandomWalkDataSource()
INFO:DataLogDaemon:Initialising node b:TimeStampDataFilter()
INFO:DataLogDaemon:Initialising node c:AggregatorDataFilter(buffer_size=3600, send_every=30)
INFO:DataLogDaemon:Initialising node d:CSVDataFilter()
INFO:DataLogDaemon:Initialising node s:FileDataSink(filename=randomwalk.csv)
INFO:DataLogDaemon:Connecting a:RandomWalkDataSource -> b:TimeStampDataFilter
INFO:DataLogDaemon:Connecting b:TimeStampDataFilter -> c:AggregatorDataFilter
INFO:DataLogDaemon:Connecting c:AggregatorDataFilter -> d:CSVDataFilter
INFO:DataLogDaemon:Connecting d:CSVDataFilter -> s:FileDataSink
INFO:main:Starting event loop.
$ cat randomwalk.csv
timestamp,analog_randomwalk0,timestamp,analog_randomwalk1
2020-04-17 22:27:36.264577+09:30,1.0,2020-04-17 22:27:36.264577+09:30,0.0
2020-04-17 22:27:37.265669+09:30,1.0,2020-04-17 22:27:37.265669+09:30,2.0
2020-04-17 22:27:38.266249+09:30,2.0,2020-04-17 22:27:38.266249+09:30,0.0
2020-04-17 22:27:39.267433+09:30,3.0,2020-04-17 22:27:39.267433+09:30,-2.0
2020-04-17 22:27:40.268600+09:30,2.0,2020-04-17 22:27:40.268600+09:30,0.0
2020-04-17 22:27:41.269554+09:30,2.0,2020-04-17 22:27:41.269554+09:30,0.0
2020-04-17 22:27:42.270715+09:30,3.0,2020-04-17 22:27:42.270715+09:30,-2.0
2020-04-17 22:27:43.271873+09:30,4.0,2020-04-17 22:27:43.271873+09:30,0.0
2020-04-17 22:27:44.272887+09:30,4.0,2020-04-17 22:27:44.272887+09:30,0.0
2020-04-17 22:27:45.274042+09:30,3.0,2020-04-17 22:27:45.274042+09:30,-2.0
2020-04-17 22:27:46.275201+09:30,2.0,2020-04-17 22:27:46.275201+09:30,0.0
2020-04-17 22:27:47.276220+09:30,1.0,2020-04-17 22:27:47.276220+09:30,0.0
2020-04-17 22:27:48.277396+09:30,1.0,2020-04-17 22:27:48.277396+09:30,0.0
2020-04-17 22:27:49.278583+09:30,0.0,2020-04-17 22:27:49.278583+09:30,-2.0
2020-04-17 22:27:50.279763+09:30,1.0,2020-04-17 22:27:50.279763+09:30,-2.0
2020-04-17 22:27:51.280956+09:30,0.0,2020-04-17 22:27:51.280956+09:30,-2.0
2020-04-17 22:27:52.282120+09:30,0.0,2020-04-17 22:27:52.282120+09:30,-2.0
2020-04-17 22:27:53.283198+09:30,0.0,2020-04-17 22:27:53.283198+09:30,-2.0
2020-04-17 22:27:54.284388+09:30,1.0,2020-04-17 22:27:54.284388+09:30,-2.0
2020-04-17 22:27:55.285719+09:30,0.0,2020-04-17 22:27:55.285719+09:30,-4.0
2020-04-17 22:27:56.286249+09:30,0.0,2020-04-17 22:27:56.286249+09:30,-4.0
2020-04-17 22:27:57.287438+09:30,-1.0,2020-04-17 22:27:57.287438+09:30,-6.0
2020-04-17 22:27:58.288604+09:30,-2.0,2020-04-17 22:27:58.288604+09:30,-8.0
2020-04-17 22:27:59.289765+09:30,-2.0,2020-04-17 22:27:59.289765+09:30,-10.0
2020-04-17 22:28:00.290927+09:30,-1.0,2020-04-17 22:28:00.290927+09:30,-10.0
2020-04-17 22:28:01.292095+09:30,0.0,2020-04-17 22:28:01.292095+09:30,-12.0
2020-04-17 22:28:02.292911+09:30,0.0,2020-04-17 22:28:02.292911+09:30,-10.0
2020-04-17 22:28:03.294073+09:30,0.0,2020-04-17 22:28:03.294073+09:30,-12.0
2020-04-17 22:28:04.295262+09:30,-1.0,2020-04-17 22:28:04.295262+09:30,-12.0
2020-04-17 22:28:05.296247+09:30,-2.0,2020-04-17 22:28:05.296247+09:30,-10.0
...