CSV collector

the collector settings are used by the collector to successfully parse csv files basic settings delimiter description the delimiter of the csv file required yes csvheader description indicates if the csv files contain a header or not if a header is present, the header record will be skipped when parsing the csv file required yes timestampcolumn description index of column holding the timestamp (starts from 0) required yes timestamplayout description the layout of the timestamp of the measurements, default rfc3339 will be used more info https //pkg go dev/time#layout required no statuscolumn description index of column holding the status (starts from 0) required no trimstring description the string to trim (leading and trailing) from the csv cells used (values, mappings, status, …) required no basepath description base directory, upon which the default incoming , processed and error directory are constructed required yes example c \plantdata\\ incomingdirectorypath description override for the directory that needs to be monitored by the csv collector required no example c \plantdata\incoming processeddirectorypath description override for the directory in which processed files will be put processed files are files which had at least one measurement sent to influx required no example c \plantdata\processed processed files are files which had at least one measurement sent to influx when a csv measurement name is not found in the measurements configuration, the file will still be marked as processed successfully errordirectorypath description override for the directory in which files that contained an error will be put invalid csv files will be stored here required no example c \plantdata\failed filemask description file mask of the files to use required yes the file mask for which the collector will monitor files advanced settings processinterval description delay in milliseconds for processing new files after the were detected required no default 10000 processdelay description delay in milliseconds for processing new files after the were detected required no default 3000 startupdelay description initial delay for checking the incoming directory in milliseconds required no default 9000 processedmaxage description the maximum time in minutes to keep processed files 0 will keep the files indefinitely required no default 43200 failedmaxage description the maximum time in minutes to keep failed files 0 will keep the files indefinitely required no default 129600 sourcetimezone description the timezone for the data points coming from the source device can be overwritten using this setting available timezones can be found here https //en wikipedia org/wiki/list of tz database time zones required no measurement settings the measurement settings section configures how csv data is interpreted and mapped into measurements this includes defining how each column in the csv file is used, setting up filters for data inclusion, and specifying data quality indicators valuecolumn description column of the value in the csv file for this measurement required yes example 6 filter description filter to parse only the rows that match the filter the filter is a json object where the key is the column index (as a string) and the value is the value to match if multiple values in the same column should be matched, separate them with a semicolon required yes details key csv column index as a string value names used for mapping, separated by semicolons if multiple example in the following example, the filter will only parse rows where the value of the first column is example value or the value of the second column is match with this or also match with this { "0" "example value", "1" "match with this;also match with this" } statusgood description all values that are mapped to status ‘good’ multiple statuses can be configured using semicolon separated values required no example “valid;correct” checkout the data collection methods section for more information timestamplayout description the layout of the timestamp, defaults to the timestamplayout setting on the collector more info https //pkg go dev/time#layout required no tagsincsv description \ this allows to add tags to the measurement with the tag value coming from csv data the structure of this setting is similar as the filter setting the json key is the column index (as string) where the tag value is present, and the json value is the tag key (or tag name) under which the data should be stored upon the measurement required no example {"4" "description"} examples example 1 this example demonstrates setting up a collector and measurement for a csv with multiple sensor data given the following csv format, with files being provided by a data source located in europe/brussels collector settings setting value delimiter , csvheader true timestampcolumn 0 timestamplayout 2/01/2006 15 04 05 000 statuscolumn 4 trimstring no value advanced setting value sourcetimezone europe/brussels measurement settings setting value name area pump99 valuecolumn 2 filter { “1” “p99” } statusgood good tagsincsv {} timestamplayout no value setting value name area temperature100 valuecolumn 2 filter { “1” “tt100” } statusgood good tagsincsv {} timestamplayout no value filtering csv data for measurements the csv collector can create points for multiple measurements at the same record (optionally originating from different value columns) measurements on the csv collector, are required to have a value column index using valuecolumn and can optionally have a mapping for a name or multiple names in the csv record using filter when no name in the csv is available for the measurement, the measurement can be saved with filter {} when there is a name or there are multiple names available in the csv (in different columns) which should match for this measurement, then the measurement can be saved with filter {"column index" "name in csv in that column"} or {"column index" "name in csv in that column", "column index" "name in csv in that column"} for multiple mappings in filter , all mappings should be valid for the record to create a point for the measurement! for one mapping, multiple possible names can be given by separating the names with a semicolon {"column index" "name1;name2", "column index" "name in csv in that column"} settings make sure to set the according csv collector /#basic settings , prior to adding measurements with the proper csv collector /#measurement settings