Skip to main content

Fluentd

Migrating from Elastic Search to Siglens using Fluentd

1. Install Fluentd​

Setup for Elasticsearch version 7.9.3​

Fluentd comes with various plugins, including an Elasticsearch plugin (fluent-plugin-elasticsearch). Follow these steps to install the compatible version:

  1. Uninstall the default Elasticsearch plugin: Make sure you are either in the fluentd-command-prompt (on Windows) or the path variables are set up or in the fluentd directory.

    fluent-gem uninstall fluent-plugin-elasticsearch

  • For td-agent:

    td-agent-gem uninstall fluent-plugin-elasticsearch
  1. Install the compatible Elasticsearch plugin (version 4.3.3):
  •  fluent-gem install fluent-plugin-elasticsearch -v 4.3.3
  •  sudo td-agent-gem install fluent-plugin-elasticsearch -v 4.3.3
  1. Verify the installation:
  • fluent-gem list | grep fluent-plugin-elasticsearch

    For td-agent:

    td-agent-gem list | grep fluent-plugin-elasticsearch

  1. Install the compatible Elasticsearch gem (version 7.9): First, uninstall any installed elasticsearch gem.

    fluent-gem uninstall elasticsearch

  • For td-agent:

    td-agent-gem uninstall elasticsearch

  • Then, install the elasticsearch gem version 7.9:

    fluent-gem install elasticsearch -v 7.9

  • For td-agent:

    td-agent-gem install elasticsearch -v 7.9

You might need to setup or install ruby modules or development toolkit. If required, it will be automatically prompted and installed.

2. Configure Fluentd​

Sample Configuration file​

<source>
  @type tail
  path D:\Siglens\SplunkExport.json
  pos_file D:\Siglens\fluentd_logs\SplunkExport1.log.pos
  tag my.logs
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

<match my.logs>
  @type elasticsearch
  host http://localhost:8081/elastic
  logstash_format true
  include_tag_key true
  tag_key @log_name
  verify_es_version_at_startup false
  default_elasticsearch_version 7.9
  request_timeout 45s # defaults to 5s
</match>

3. Run Fluentd​

Navigate to the Fluentd directory and run fluentd -c <<path-of-fluentd-config>>. On Linux, prepend the command with sudo. If using td-agent, replace fluentd with td-agent. On Windows, run the command as an Administrator.

  • Linux: 
    sudo fluentd -c /home/fluentd_config.conf
  • Windows: Open the fluentd command prompt as an Administrator and run 
    fluentd -c /home/fluentd_config.conf