Overview of the test, dataset

The objective of this benchmark is to assess performance of these two databases under conditions mirroring real world scenarios. Our approach involves running both databases on identical hardware configurations, ingesting identical dataset and running the same set of queries. For this reason we opted for the NYC Taxi Dataset.

The NYC Taxi Dataset is a massive dataset of over 1 Billion trips made by NYC Taxis and Uber, all originating in New York City since 2009.

The queries used were as follows:

Q1: Find the total number of rides.
Q2: Find the average ride cost grouped by the number of passengers.
Q3: Find the number of rides grouped by the number of passengers and the pick-up location.
Q4: Find the number of rides grouped by the number of passengers, pick-up location, and trip distance.

Introductions of the databases

Grafana Loki:

Loki is a Prometheus inspired multi-tenant, highly available, horizontally scalable log aggregation system. In contrast to Prometheus, Loki collects logs using push rather than pull methods and places more emphasis on logs than metrics.

Loki simply indexes metadata about your logs, such as a set of labels for each log stream, not the actual contents of the logs. Loki-based logging stack consists of 3 components viz. Agent ( for example, Promtail), Loki (main server) and Grafana.

SigLens:

SigLens is a column oriented database built from scratch for observability data. It offers dynamic compressions over each column requiring zero configuration.

SigLens uses columnar micro indices that are 1/100th size of an actual database index. This reduces the number of machines needed. SigLens uses AgileAggregationTree algorithm that makes aggregations queries run at sub-second speeds.

Test Setup

We used an AWS im4gn.2xlarge instance for both SigLens and Loki, it had the following hardware configuration:

8 vCPUs
32 GB of RAM
3.5 TB of storage
10 Gigabit networking

Common steps for both:

Setup the server, mount the storage, configure aws s3 cli by following these steps:

Step 1: Let’s find out if the NVM storage device is present on this server. To find that, ssh into your server and run lsblk. You should see something like the following, with the nvme1n1 item having 3.4 TB of storage.

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0  21.3M  1 loop /snap/amazon-ssm-agent/7529
loop1          7:1    0  49.1M  1 loop /snap/core18/2794
loop2          7:2    0  59.3M  1 loop /snap/core20/2019
loop3          7:3    0 109.6M  1 loop /snap/lxd/24326
loop4          7:4    0  35.5M  1 loop /snap/snapd/20102
nvme0n1      259:0    0     8G  0 disk
├─nvme0n1p1  259:2    0   7.9G  0 part /
└─nvme0n1p15 259:3    0    99M  0 part /boot/efi
nvme1n1      259:1    0   3.4T  0 disk

Step 2: Let’s mount the storage device and create directories.

sudo mkfs.xfs /dev/nvme1n1
sudo mkdir /mnt/nvme1n1
sudo mount /dev/nvme1n1 /mnt/nvme1n1

Step 3: Verify Storage

You can check that it's mounted by running df -h and you should see something like this:

Filesystem       Size  Used Avail Use% Mounted on
/dev/root        7.6G  1.5G  6.2G  20% /
tmpfs             16G     0   16G   0% /dev/shm
tmpfs            6.2G  948K  6.2G   1% /run
tmpfs            5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p15   98M  6.3M   92M   7% /boot/efi
tmpfs            3.1G  4.0K  3.1G   1% /run/user/1000
/dev/nvme1n1     3.5T   25G  3.4T   1% /mnt/nvme1n1

Step 4: Update Permissions

sudo chmod 777 /mnt/nvme1n1

Step 5: Configure AWS CLI

Note: This AWS config step is only needed if you are using a private S3 bucket, if you use our public S3 bucket you can skip this step.

sudo apt-get install awscli -y
aws configure

Cleanup and Prepare Dataset:

Note: You can skip this step and use our s3 bucket. It has the dataset in the correct format. If you want to prepare the dataset yourself then follow these steps else jump directly to the benchmark step to use our s3 bucket.

mkdir -p /mnt/nvme1n1/data
cd /mnt/nvme1n1/data

Download the dataset from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page into above dir. We used the 2011-2017 yellow taxi trip parquet files

Benchmark Grafana Loki

You'll want three terminals. Terminal 1 will run Loki, Terminal 2 will run Promtail, and Terminal 3 will run ingestion and queries.

Install Loki and Promtail

sudo apt install unzip -y
                                
mkdir /mnt/nvme1n1/loki
cd /mnt/nvme1n1/loki
                                
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.4/loki-linux-arm64.zip"
unzip loki-linux-arm64.zip
chmod a+x loki-linux-arm64
                                
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.4/promtail-linux-arm64.zip"
unzip promtail-linux-arm64.zip
chmod a+x promtail-linux-arm64
                                
wget https://raw.githubusercontent.com/grafana/loki/v2.9.4/cmd/loki/loki-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.9.4/clients/cmd/promtail/promtail-local-config.yaml

Update the default configurations

In loki-local-config.yaml, update these configurations with these new values:

path_prefix: /mnt/nvme1n1/loki/internal/path
chunks_directory: /mnt/nvme1n1/loki/internal/chunks
rules_directory: /mnt/nvme1n1/loki/internal/rules

Increase the timeout limit by adding these into the server section:

http_server_read_timeout: 12h
http_server_write_timeout: 12h

Add this to your config to avoid ingestion throttling and increase the number of buckets that queries are allowed to return.

limits_config:
    per_stream_rate_limit: 4G
    per_stream_rate_limit_burst: 4G
    ingestion_rate_mb: 4096
    ingestion_burst_size_mb: 4096
    max_query_series: 1000000
    query_timeout: 12h

In promtail-local-config.yaml change __path__ to:

__path__: /mnt/nvme1n1/data/*.json

Run Loki and Promtail

In Terminal 1, start Loki with:

./loki-linux-arm64 -config.file=loki-local-config.yaml

In Terminal 2, start Promtail with:

./promtail-linux-arm64 -config.file=promtail-local-config.yaml

Ingest the Data

In Terminal 3, run:

mkdir /mnt/nvme1n1/data
cd /mnt/nvme1n1/data
                                    
for year in {2011..2017}; do
    for month in {01..12}; do
        basefile="yellow_tripdata_$year-$month"
                                    
        aws s3 cp s3://siglens-benchmark-datasets/nyc-taxi-benchmark-data/json/$basefile.json.gz .
        gunzip $basefile.json.gz
    done
done

Since Promtail is already running and configured to ingest all JSON files in this data/ directory, it will start ingesting the logs. You can check the progress of the ingestion with this bash:

total=$(curl -s http://localhost:9080/metrics | grep -v '^#' | awk '/promtail_file_bytes_total/ {sum += $2} END {print sum}'); \
ingested=$(curl -s http://localhost:9080/metrics | grep -v '^#' | awk '/promtail_read_bytes_total/ {sum += $2} END {print sum}'); \
percent=$(echo "$ingested $total" | awk '{printf "%.2f\n", ($1 / $2) * 100}'); \
echo "percent ingested: $percent%"

Run the Queries in Loki

After ingestion is complete, run the queries.

# Query 1
curl -G -s "http://localhost:3100/loki/api/v1/query" \
    --data-urlencode "query=sum(count_over_time({job=\"varlogs\"} | json [24h])) by (airport_fee)" \
    --data-urlencode "time=$(date +%s)000000000" \
    --data-urlencode "stats=true"
                                
# Query 2
curl -G -s "http://localhost:3100/loki/api/v1/query" \
    --data-urlencode "query=avg_over_time({job=\"varlogs\"} | json | unwrap total_amount [24h]) by (passenger_count)" \
    --data-urlencode "time=$(date +%s)000000000" \
    --data-urlencode "stats=true"
                                
# Query 3
curl -G -s "http://localhost:3100/loki/api/v1/query" \
    --data-urlencode "query=sum(count_over_time({job=\"varlogs\"} | json [24h])) by (passenger_count, PULocationID)" \
    --data-urlencode "time=$(date +%s)000000000" \
    --data-urlencode "stats=true"
                                
# Query 4
curl -G -s "http://localhost:3100/loki/api/v1/query" \
    --data-urlencode "query=sum(count_over_time({job=\"varlogs\"} | json [24h])) by (passenger_count, PULocationID, trip_distance)" \
    --data-urlencode "time=$(date +%s)000000000" \
    --data-urlencode "stats=true"

Grafana Loki Query Results

We ran each query for 5 times, and took the lowest number of all those runs.

	Try 1	Try 2	Try 3	Try 4	Try 5	Lowest
Q1	295,679 ms	292,636 ms	283,091 ms	281,638 ms	279,073 ms	279,073 ms
Q2	3,998,457 ms	3,976,538 ms	4,007,424 ms	4,002,602 ms	3,994,540 ms	3,976,538 ms
Q3	405,964 ms	413,498 ms	421,700 ms	416,008 ms	411,589 ms	405,964 ms
Q4	N/A	N/A	N/A	N/A	N/A	N/A

Note: For query 4, we had increased Loki’s timeout limit to 12 hours, but after 4.5 hours the query returned and didn’t have any response (not even an empty json response), so it seems that Loki canceled the query for some reason.

Benchmark SigLens

You'll want three terminals. Terminal 1 will run SigLens, Terminal 2 will do some setup and view the logs, and Terminal 3 will send the queries. Terminal 3 can run on your local machine if you set up the server to accept HTTP traffic, but Terminals 1 and 2 should be on the server. Start with Terminal 1.

Install Go and SigLens

SigLens is built in golang. In this step we will install the go dependencies, install SigLens and start it.

sudo apt update
sudo apt install golang -y
git clone https://github.com/siglens/siglens.git
cd siglens

## Start SigLens
sudo go run cmd/siglens/main.go --config server.yaml

Wait until SigLens is running. You'll see these lines in the terminal once it's up:

INFO[2023-12-06 18:10:38] Extracting config from configFile: server.yaml
INFO[2023-12-06 18:10:38] Defaulting to 2160hrs (90 days) of retention...
INFO[2023-12-06 18:10:38] ----- Siglens server type SingleNode starting up -----
INFO[2023-12-06 18:10:38] ----- Siglens Ingestion server starting on 0.0.0.0:8081 -----
INFO[2023-12-06 18:10:38] ----- Siglens Query server starting on 0.0.0.0:5122 -----
INFO[2023-12-06 18:10:38] ----- Siglens UI starting on 0.0.0.0:5122 -----

Prepare Agile Aggregations

In a continuous running ingest, you don’t need to do this step. However, since this is a static dataset ingestion, we need to set up the aggregations columns. SigLens creates aggregation trees during ingestion using these columns. These trees are used during query processing

Switch to Terminal 2 and run the following:

curl -X POST -d '{
    "tableName": "trips",
    "groupByColumns": ["airport_fee", "passenger_count", "PULocationID", "trip_distance"],
    "measureColumns": ["total_amount"]
}' http://localhost:5122/api/pqs/aggs
echo ""

You should get this response:

{"message":"All OK","status":200}

Ingest data into SigLens

Now let's ingest some data into SigLens. In Terminal 2 run

cd /mnt/nvme1n1/siglens/tools/sigclient 
mkdir dataset

Run the following script to download, decompress, and ingest the data into SigLens.

for year in {2011..2017}; do
    for month in {01..12}; do
        {
            basefile="yellow_tripdata_$year-$month"
            aws s3 cp s3://siglens-benchmark-datasets/nyc-taxi-benchmark-data/json/$basefile.json.gz dataset/
            gunzip dataset/$basefile.json.gz
            python3 ../nyc-taxi-benchmark/ingester.py dataset/$basefile.json
        } &
    done
    wait
done

Run Queries against SigLens

Run the following in Terminal 3. If Terminal 3 is on your local machine, make sure to replace localhost with the IP of the server. You can remove the | python3 -m json.tool if you want, it just formats the JSON response. Check the log file siglens.log for the query times.

curl -X POST -d '{
    "searchText": "SELECT airport_fee, count(*) FROM trips GROUP BY airport_fee",
    "indexName": "trips",
    "startEpoch": "now-24h",
    "endEpoch": "now",
    "queryLanguage": "SQL"
}' http://localhost:5122/api/search | python3 -m json.tool

curl -X POST -d '{
    "searchText": "SELECT passenger_count, avg(total_amount) FROM trips GROUP BY passenger_count",
    "indexName": "trips",
    "startEpoch": "now-24h",
    "endEpoch": "now",
    "queryLanguage": "SQL"
}' http://localhost:5122/api/search | python3 -m json.tool

curl -X POST -d '{
    "searchText": "SELECT passenger_count, PULocationID, count(*) FROM trips GROUP BY passenger_count, PULocationID",
    "indexName": "trips",
    "startEpoch": "now-24h",
    "endEpoch": "now",
    "queryLanguage": "SQL"
}' http://localhost:5122/api/search | python3 -m json.tool

curl -X POST -d '{
    "searchText": "SELECT passenger_count, PULocationID, trip_distance, count(*) FROM trips GROUP BY passenger_count, PULocationID, trip_distance",
    "indexName": "trips",
    "startEpoch": "now-24h",
    "endEpoch": "now",
    "queryLanguage": "SQL"
}' http://localhost:5122/api/search | python3 -m json.tool

SigLens Query Results:

We ran each query for 5 times, and took the lowest number from those runs.

	Try 1	Try 2	Try 3	Try 4	Try 5	Lowest
Q1	60 ms	72 ms	63 ms	41 ms	42 ms	41 ms
Q2	43 ms	43 ms	44 ms	43 ms	43 ms	43 ms
Q3	116 ms	304 ms	442 ms	509 ms	471 ms	116 ms
Q4	9,815 ms	9,229 ms	10,316 ms	9,093 ms	9,092 ms	9,092 ms

Result Comparison

As you have seen, we used the exact same hardware, exact same dataset, exact same queries. We took the lowest of the 5 runs for each query type for both of these databases and compiled that result in this table:

Query Type	Grafana Loki	SigLens	Comparison
Q1	286,423 ms	55 ms	SigLens 5208 x faster than Grafana Loki
Q2	3,995,912 ms	43 ms	SigLens 92,928x faster than Grafana Loki
Q3	413,752 ms	368 ms	SigLens 1124x faster than Grafana Loki
Q4	N/A	9,509 ms	N/A

SigLens proves to be 1124x-92,928x faster on all query types. These speed advantages of SigLens are due to its dynamic columnar compressions, dynamic columnar micro indexing, AgileAggsTree. These innovative techniques help run your filter and aggregation queries on complex datasets at lightning fast speeds.

Try SigLens in less than 1 minute

SigLens gets installed in less than a minute with docker or helm chart. Give it a try and experience the difference.

Try SigLens

GitHub Repo

Benchmark Questions?

Both of these databases were given the exact same treatment, there was no unfair advantage given to SigLens.

You can verify these test results yourself by following the detailed steps we have outlined above. If you feel something is incorrect please submit an issue or a pull request at https://github.com/siglens/siglens.