We've had some benchmarking requests from multiple organizations struggling with ingest performance from Elasticsearch, so we're publishing them here. The latest Gravwell release marks a significant improvement in ingest and indexing performance and this post covers the nitty gritty details. Better ingest performance means reduced infrastructure cost, less dropped data, and faster time-to-value. See how Gravwell stacks up.

We are excited to introduce autoextractors with Gravwell version 3.0.2.  Autoextractors make it easy for regex gurus and binary ninjas to generate extractions and share them in a portable format.  Autoextractors can dramatically simplify the process of performing field extractions from unstructured data without complicated time-of-ingest data definitions; they can built and shared by ninjas and and used by us mere mortals.

In this detailed technical guide we’ll cover analyzing Bro security analytics with Gravwell. Bro is a passive network security sensor designed to provide a plugin friendly detection framework. There are a myriad of commercial Bro vendors and almost as many ways to format and store the output. Gravwell provides an efficient and simple interface for acquiring, storing, and querying Bro data.

DNS auditing is an integral part of any I.T. security program. Name resolutions can act as a great tip for discovering malware, command and control streams, or misbehaving employees. Acquiring DNS audit data can be difficult with some DNS servers (*cough* Windows *cough*); for this post we are going to show an extremely easy method of getting DNS audit data directly into Gravwell.

To celebrate the release of the Gravwell Community Edition we are also releasing a standalone collectd ingester. Collectd is an excellent tool for monitoring the health of hardware, systems, and applications. For this post we will be demonstrating the installation and configuration of collectd to monitor the health and status of a few machines. We will be providing dashboard import codes so that you can quickly and easily import our ready made dashboards. The collectd ingester is part of the core suite of ingesters and is open source on github.

Back when we released the first version of Gravwell we immediately began sharing with friends and colleagues. Those initial testers primarily used Gravwell to monitor their home networks. They found rogue devices, neighbors leaching WiFi, poorly behaving IOT devices, and even some children that were playing video games when they should have been in bed. There was one problem, our friends wanted to give Gravwell to their friends but we aren't really a consumer software company. Our from-scratch secret sauce is what enables such game-changing pricing for larger enterprises but because we don't price on the data drip model it doesn't work for very small deployments. All that changes with the community edition...

Update (1/24/2019)

This post is mostly about building your own docker images. If you're interested in getting up and running fast using Gravwell+Docker, head over to our docs that cover our pre-built images: 

https://dev.gravwell.io/docs/#!configuration/docker.md

Overview

For this blog post we are going to go over the deployment of a distributed Docker-based Gravwell cluster. We will use Docker and a few manageability features to very quickly build and deploy a cluster of Gravwell indexers. By the end of the post we will have deployed a 6 node Gravwell cluster, a load balancing federator, and a couple ingesters. Also, the six node “cluster” is also going to absolutely SCREAM, collecting over 4 million entries per second on a single Ryzen 1700 CPU. You read that right, we are going to crush the ingest rate of every other unstructured data analytics solution available on a single $250 CPU.  Lets get started.

TL;DR

We are going to dive into Windows and show how to get logs flowing into Gravwell in under 5 minutes with the WinEvent ingester. Using the Windows queries we will audit login behavior, RDP usage, some Windows Defender, and identify when Bob from accounting is copying sensitive financial data to external storage devices. Also, Taylor Swift is involved; don't panic, just stay with me.

Overview

This Gravwell post is all about the wild world of Windows Event logging and analytics. Both Unix and Windows provide standardized central logging facilities; however, the structure and format of the stored logs are dramatically different. Syslog and most other logging systems with roots in Unix approach logging as an unstructured stream: a log entry is a string of text, no more, no less (we are going to ignore journald and its binary madness). Windows, however, logs all events in fully-formed XML and the logging system is integrated into the operating system itself.  We should also note that logging in Windows is... less than ideal.  If you are coming from the Unix world, throw out all your assumptions; things are different here.