We've had some benchmarking requests from multiple organizations struggling with ingest performance from Elasticsearch, so we're publishing them here. The latest Gravwell release marks a significant improvement in ingest and indexing performance and this post covers the nitty gritty details. Better ingest performance means reduced infrastructure cost, less dropped data, and faster time-to-value. See how Gravwell stacks up.

When doing some comparisons of Gravwell to other systems, it's important to remember that Gravwell is an unstructured system -- more of a data lake than a data warehouse. So while some competitors convert the data into a fully structured document for ingest and indexing, Gravwell does this all on the fly with a highly flexible data definition. So even if a log entry contains a stack trace (we have seen that... a lot) Gravwell can still ingest it and you can still query it. Anomalous input does not break core Gravwell indexing. We'll talk more about that distinction in a future post, but for now let's jump right into the results.

The core results from the Elasticsearch benchmark are generated from a 3.2GB geoname dataset in JSON format using the esrally tool. This is structured data. To better serve our customers unique needs, Gravwell has a modular indexing strategy. The worst case "shotgun" approach is to just full text everything.

Elasticsearch and Gravwell are very different systems at their core and each solves different problems. Apples-to-apples comparisons aren't really fair, so keep that in mind. However, many people use Elasticsearch for log and event correlation and analytics so....here we go: 

Keep reading for the technical details or skip to raw output at this gist. 

The other thing to point out is the system utilization during ingest. Here are two screenshots of htop running during these benchmarks.

The Gravwell paradigm of "ingest anything" means standing up Gravwell can be insanely fast and easy compared to complicated structure-on-ingest solutions. Get your log analytics set up in hours instead of days.

Test System

For this blog we will be leveraging an Intel Core i7-7700k. This system is not a server class system with quad,  hexa, or even octo channel memory, but it has characteristics similar to what you might find in a low cost 1U blade server. We will also be running our instance of Gravwell inside Docker and using the BTRFS filesystem.

The exact specs are:

We will show a variety of configurations with different indexing schemes and engines so that you can see how the various configurations effect ingest and query performance.

System Tuning

Not much tuning is needed with a modern Linux system with Gravwell, but a few tweaks can reduce the strain and wear and tear on hardware components when using the index engine. Mostly we are trying to reduce the number of page writebacks that occur when the host system has a very fast disk available. Gravwell uses a synchronization system to ensure that data makes it to the disk, but the host kernel will also attempt to push dirty pages back to the disk as it can. Fast disks, like NVME flash, allow the kernel to perform these writebacks very quickly which can cause needless flash wear and tear. If you do not tune these parameters your performance won't change much, but you might see a lot more data written to your SSD.

Four parameters in the virtual memory subsystem should be tweaked:

1. dirty_ratio
2. dirty_background_ratio
3. dirty_writeback_centisecs
4. dirty_expire_centisecs

Here is a handy script that will throw some good values in and set them up to be reloaded at every reboot.

#!/bin/bash
user=$(whoami)
if [ "$user" != "root" ]; then
    echo "must run as root"
fi

echo 70 > /proc/sys/vm/dirty_ratio
echo 60 > /proc/sys/vm/dirty_background_ratio
echo 2000 > /proc/sys/vm/dirty_writeback_centisecs
echo 3000 > /proc/sys/vm/dirty_expire_centisecs

echo "vm.dirty_ratio = 70" >> /etc/sysctl.conf
echo "vm.dirty_background_ratio = 60" >> /etc/sysctl.conf
echo "vm.dirty_writeback_centisecs = 2000" >> /etc/sysctl.conf
echo "vm.dirty_expire_centisecs = 3000" >> /etc/sysctl.conf

Methodology

We will be using Gravwell version 3.1.0 and ingesting text data using the singleFIle ingester from https://github.com/gravwell/ingesters on commit e6c6fadfdf507e9bffc5e3cb930b9d07ca5c0719. Ingest will be performed with the "-ignore-ts" flag so that ingest rates are not limited by the single ingesters ability to resolve timestamps dynamically.

For each configuration we will flush the file system caches on the host prior to ingest and query using the command:

sync; echo 3 > /proc/sys/vm/drop_caches

The docker system is mounted on its own partition using the BTRFS filesystem with transparent file compression. When we use the transparent file compression we will use the ZSTD compression algorithm with compress-force=true. The exact /etc/fstab entry for storage volume is:

/dev/nvme0n1p4 /var/lib/docker btrfs ssd,compress-force=zstd 0 0

Well Configurations

Each data set will be ingested using four different well configurations:

1. No indexing at all, only a temporal index
2. Fulltext indexing, worst case
3. Complete field indexing using the bloom engine
4. Complete field indexing using the index engine

Raw Well

This well utilizes no data indexing at all, just a raw unstructured ingest. The well DOES build a temporal index, but because the data does not have timestamps the temporal index is minimal.  Every query will need to scrape and structure the entire dataset.

[Storage-Well "raw"]
    Location=/opt/gravwell/storage/raw
    Tags=raw
    Enable-Transparent-Compression=true

Fulltext Well

This well utilizes the fulltext accelerator and will break every log entry into its components for indexing using the index engine. This is by far the most abusive indexing strategy, but is also the most forgiving. Fulltext indexing allows you accelerate searches on subcomponents, not just complete fields. The fulltext indexing system can often result in an index that is as large or larger than the original data.

[Storage-Well "fulltext"]
    Location=/opt/gravwell/storage/fulltext
    Tags=fulltext
    Accelerator-Name=fulltext
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

Field Extraction With Bloom Engine Well

The field extraction system extracts each field and uses Gravwell's bloom engine to perform query acceleration. The bloom engine is much more efficient when comes to disk overhead, but can be less efficient for query. Bloom filters provide the ability to say that a field value is definitely not in a set, or might be in a set. Gravwell uses the bloom engine to rapidly evaluate whether or not it needs to process a block of data, effectively reducing the search space. The bloom engine performs very well when querying sparse data, but is not very effective when querying fields that occur regularly.

[Storage-Well "bench"]
    Location=/opt/gravwell/storage/bench
    Tags=bench
    Accelerator-Name=<dataset dependent>
    Accelerator-Args=<dataset dependent>
    Accelerator-Engine-Override=bloom
    Enable-Transparent-Compression=true

Field Extraction With Index Engine Well

Field extraction with the indexing system uses a full indexing solution combined with field extraction. The indexing engine can consume significant disk space (especially if paired with full text indexing), but provides an effective acceleration across all query types, sparse and dense. However, that disk overhead can translate to dramatic query speedups, especially when dealing with extremely large data sets.

[Storage-Well "bench2"]
    Location=/opt/gravwell/storage/bench2
    Tags=bench2
    Accelerator-Name=<dataset dependent>
    Accelerator-Args=<dataset dependent>
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

Installing gravwell.conf and Starting

To start, make sure that you have the latest Gravwell docker image by performing a docker pull gravwell/gravwell:latest, then create the container without starting it:

docker create --name benchmark --rm gravwell/gravwell:latest

Pull the default gravwell.conf from the container, edit it to add our wells (there will be 8 plus the default), push it back in, and then start the container:

docker cp benchmark:/opt/gravwell/etc/gravwell.conf /tmp/
vim /tmp/gravwell.conf
docker cp /tmp/gravwell.conf benchmark:/opt/gravwell/etc/gravwell.conf
docker start benchmark

The exact gravwell.conf we will be using is:

[global]
### Web server HTTP/HTTPS settings
Web-Port=80
Insecure-Disable-HTTPS=true

### Other web server settings
Remote-Indexers=net:127.0.0.1:9404
Persist-Web-Logins=True
Session-Timeout-Minutes=1440
Login-Fail-Lock-Count=4
Login-Fail-Lock-Duration=5

[Default-Well]
    Location=/opt/gravwell/storage/default/
    Disable-Compression=true

[Storage-Well "rawgeo"]
    Location=/opt/gravwell/storage/rawgeo
    Tags=rawgeo
    Enable-Transparent-Compression=true

[Storage-Well "fulltextgeo"]
    Location=/opt/gravwell/storage/fulltext
    Tags=fulltextgeo
    Accelerator-Name=fulltext
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

[Storage-Well "benchgeo"]
    Location=/opt/gravwell/storage/benchgeo
    Tags=benchgeo
    Accelerator-Name=fields
    Accelerator-Args="-d \"\t\" [0] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]"
    Accelerator-Engine-Override=bloom
    Enable-Transparent-Compression=true

[Storage-Well "bench2geo"]
    Location=/opt/gravwell/storage/bench2geo
    Tags=bench2geo
    Accelerator-Name=fields
    Accelerator-Args="-d \"\t\" [0] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]"
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

[Storage-Well "rawapache"]
    Location=/opt/gravwell/storage/rawapache
    Tags=rawapache
    Enable-Transparent-Compression=true

[Storage-Well "fulltextapache"]
    Location=/opt/gravwell/storage/fulltextapache
    Tags=fulltextapache
    Accelerator-Name=fulltext
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

[Storage-Well "benchapache"]
    Location=/opt/gravwell/storage/benchapache
    Tags=benchapache
    Accelerator-Name=fields
    Accelerator-Args="-q -d \" \" [0] [1] [2] [3] [5] [6] [7] [8] [9] [10]"
    Accelerator-Engine-Override=bloom
    Enable-Transparent-Compression=true

[Storage-Well "bench2apache"]
    Location=/opt/gravwell/storage/bench2apache
    Tags=bench2apache
    Accelerator-Name=fields
    Accelerator-Args="-q -d \" \" [0] [1] [2] [3] [5] [6] [7] [8] [9] [10]"
    Accelerator-Engine-Override=index
    Enable-Transparent-Compression=true

The docker image has the ingest and control secrets configured via environment variables, so we do not need to set them in the gravwell.conf. We can now upload our license and check on our well configurations through the GUI.

Data Sets and Results

For familiarity, we will be using some of the same datasets employed by Elastic.co to benchmark elasticsearch. The datasets are all available on github and represent a nice cross section of data types that might be stored in Gravwell. Here are links to benchmarking info on Elasticsearch and Lucene itself.

Some of the data sets have very high key variability (many completely unique fields) and some have relatively low key variability (reasonable amount of key repetition). In most networking and security contexts key variability is relatively modest as the same machines will typically do the same things over and over.

The command used to ingest each data set using the singleFile ingester is roughly:

Installing the singleFile ingester is as simple as running "go get -u github.com/gravwell/ingesters/singleFIle" if you have the go tools installed.

Geonames

The geonames data set contains a tab delimited data set of locations around the world. The fields include unique identifiers, country codes, city names, locations, elevations, coordinates, etc. This dataset has a very high level of key variability. Every single entry contains multiple unique values, including IDs, coordinates, locations, elevations, and populations. The geonames data set is a good example of ingesting data that is particularly abusive to an index.

The geonames dataset is comprised of 11.9 million records and 1.5GB. The raw data is available at via geonames.org and is described on their readme page. Gravwell will use the fields acceleration module to handle the tab delimited data for the bench and bench2 wells.

The Accelerator-Name parameter will be fields and the Accelerator-Args parameter will use the tab delimiter and will index the full 18 fields.

Accelerator-Name=fields
Accelerator-Args="-d \"\t\" [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]"

Apache Access Logs

A common log format used to benchmark indexing systems is the Apache access log. There aren't very many large corpus of access logs available, so we used a generator from github to generate 10 million apache access logs that take up 2.1 GB of space. The generator used to generate the log data is available https://github.com/kiritbasu/Fake-Apache-Log-Generator.

The Accelerator-Name parameter will be fields and the Accelerator-Args parameter will use the space delimiter and supported quoted fields, we will index the full 9 fields.

Accelerator-Name=fields
Accelerator-Args="-q -d \" \" [0] [1] [2] [3] [4] [5] [6] [7] [8] [9]"

Interestingly, the uncompressed storage usage on indexing the whole fields uses more disk space than just indexing the entire entry with fulltext. There are a pretty standard set of useragent components and URL components and when you combine them they become pretty unique keys. When you treat them as words, there isn't much variability so fulltext is actually a bit more efficient in terms of disk space here.

Query Performance

To measure query performance we will run two queries across each well, the first query finds the raw record using a sparse data field (we will use an IP or city name), the second query uses a dense field (we will use a request and country code). Thow two queries will show the performance characteristics of each indexing method. For each query result we will provide the total time to execute as well as the total number of entries that enter the search pipeline. Examining the total number of entries that enter the pipeline is a good metric for determining how effective the indexing system was at reducing the search space.

GeoNames

For the GeoName data set we will execute two queries for each well, the first is a query that retrieves a single record using a highly unique key, the second is one that retrieves some subset of the records using a reasonably common key. We will use the fields module with inline filtering for all wells except the fulltext well where we will use the grep module and the "-w" flag.

Sparse Query Results

For this test we will retrieve a single record using a unique identifier, we will be pulling back the geonames entry for the Moscow, ID police department with the unique ID 9671688.

The exact query for the fields wells is:

fields -d "\t" [0]==9671688

The query for the fulltext well is:

grep -w 9671688

Dense Query Results

For this test we will retrieve a subset of records using a relatively common identifier, we will be pulling every item in the "America/Boise" region. Because the field contains a word separator or fulltext query will have to filter terms. There are 25,263 entries out of the 11.9 million that match this query.

The exact query for the fields wells is:

fields -d "\t" [17]=="America/Boise"

The query for the fulltext well is:

grep -w -s America Boise | grep "America/Boise"

Apache Access

For the Apache access log data set we will again execute two queries for each well, the first is a query that retrieves a single record filtering for a specific IP, the second is one that retrieves some subset of the records using a reasonably common key. We will use the fields module with inline filtering for all wells except the fulltext well where we will use the grep module and the "-w" flag.

For this test we will retrieve a single record by searching for a specific IP; for the well using fulltext well we will use "grep -w" to find a specific record, for all others we will use the fields module to extract a specific field.

Sparse Query Results

For the sparse search, we are going to look for logs from a specific IP. Because we used a generator and have less than 4 billion records, there is a reasonable likelihood that each IP is only in the log set once or twice. We are looking for the IP 104.170.11.4, but if you are following along you will need to pick an IP in your set.

The exact query for the fields wells is:

fields -d " " -q [0]=="104.170.11.4"

The query for the fulltext well is:

grep -w "104.170.11.4"

Dense Query Results

For the dense query search, we are going to look for entries with a return code of 500. The 500 error code represents about 2% of the log entries in our data set. Because the "word" 500 might be in other places, the "grep -w" module can't stand on its own, so we have to follow it up with another grep to ensure we are only getting entries with a return code of 500.

The exact query for the fields wells is:

fields -d " " -q [6]==500 | count

The query for the fulltext well is:

grep -w 500 | grep " HTTP/1.0\" 500"

It is important to note how the bloom engine utterly failed for this dense query, this is because the bloom engine can only say if some is or is not in a set. For this query and data set, the return code of 500 was evenly spread throughout the entire dataset; it was likely in every single block, making the bloom engine acceleration not much better than raw. It is important to think about your queries and your data. The bloom engine is great for finding needles in a haystack, but pretty terrible at dealing with subsets that are relatively dense.

The results for these queries are a lot less spectacular than the sparse queries due to the even disbursement of our target dataset. Most of the work is done pulling data from the disk and decompressing blocks, so while the index engine ensured that very few entries entered the pipeline, the storage system still had to do a lot of work to get blocks off the disk.

Conclusion

Gravwell provides significant flexibility in how and what it indexes. Logging and data analytics systems are typically sized for a median data flow, whether that be EPS or MB/s, however the real world often throws wrenches and it can be extremely important to have the ingest and indexing headroom to deal with sudden bursts of data.

The hybrid indexing system also enables users of the system to not need to know exactly how data is indexed; as long as they leverage inline filters and the appropriate filtering modules Gravwell will use the configured accelerators to speed up queries. Queries that span multiple tags and wells will transparently invoke their respective acceleration engines without user involvement.