Vector

Ingesting logs into Siglens using Vector

1. Install Vector

Linux

Install Vector for Linux:

Debian and Ubuntu

Add the Vector repo and install using APT:

curl -1sLf 'https://setup.vector.dev' \
| sudo -E bash
sudo apt-get install vector

CentOS, Redhat, and Amazon Linux

Add the Vector repo and install using YUM:

curl -1sLf 'https://setup.vector.dev' \
| sudo -E bash
sudo yum install vector

macOS

Install Vector using Homebrew:

brew tap vectordotdev/brew && brew install vector

Windows

Install Vector using the official installer for Windows:

powershell Invoke-WebRequest https://packages.timber.io/vector/0.36.1/vector-x64.msi -OutFile vector-0.36.1-x64.msi
msiexec /i vector-0.36.1-x64.msi

Other

Install Vector using the Vector installer:

curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash

2. Configure Vector

Download the sample events file using the following command:

curl -s -L https://github.com/siglens/pub-datasets/releases/download/v1.0.0/2kevents.json.tar.gz -o 2kevents.json.tar.gz && tar -xvf 2kevents.json.tar.gz

Create a vector config file with the Siglens Vector sink.

YAML

vector.yaml

data_dir: /var/lib/vector

sources:
  read_from_file:
    type: file
    include:
      - 2kevents.json # Path to the log file

sinks:
  siglens:
    type: elasticsearch
    inputs:
      - read_from_file
    endpoints:
      - http://localhost:8081/elastic/
    mode: bulk
    healthcheck:
      enabled: false

JSON

vector.json

{
  "data_dir": "/var/lib/vector",
  "sources": {
    "read_from_file": {
      "type": "file",
      "include": [
        "2kevents.json"
      ]
    }
  },
  "sinks": {
    "siglens": {
      "type": "elasticsearch",
      "inputs": [
        "read_from_file"
      ],
      "endpoint": "http://localhost:8081/elastic/",
      "mode": "bulk",
      "healthcheck": {
        "enabled": false
      }
    }
  }
}

Please note that you might need to add transforms to your Vector configuration according to the structure of your data to ensure it is processed correctly.

For in-depth information on Vector configuration, visit the official vector documentation.

3. Run Vector

vector --config vector.yaml