Ingesting td-agent.log with the DataSet Fluentd Plugin

For users who use our Fluentd plugin to send logs to DataSet, you might want to ingest the td-agent.log OR fluentd.log to DataSet for troubleshooting purposes. You can use tail function to read the td-agent.log and forward the output to DataSet. Here is a sample config file:

<source>
 @type tail
 path /var/log/td-agent/td-agent.log
 pos_file /var/log/td-agent/td-agent.log.pos
 format none
 tag scalyr.tdagent.log
</source>

<filter scalyr.tdagent.log>
 @type record_transformer
 <record>
 parser tdagent
 </record>
</filter>

<match scalyr.**>
 @type scalyr
 api_write_token XXXX
 server_attributes {
 "serverHost": "fluentd_host"
 }
</match>

Notes:

• Please adjust the tag (i.e. scalyr.tdagent.log) based on your own config
• If you are getting "Permission denied @ rb_file_s_stat" error, it can be resolved by running the following commands.

chown -R root:td-agent /var/log
chmod -R 775 /var/log/

Restarting Fluentd to apply the changes, and log in to DataSet to confirm that td-agent.log is successfully ingested.