John Floren

John's been writing Go since before it was cool and developing distributed systems for almost as long.
Find me on:

Recent Posts

Slice it Like Roast Beef: Parsing Raw ARP Messages in Gravwell

Feb 23, 2021 9:06:44 AM / by John Floren posted in Data Fusion, compound queries

One of Gravwell's great strengths is binary ingest: you can store things like raw packets, then parse them later when you know what you want to extract. This came in handy recently when I set up IPv6 on my home network and wanted to keep an eye on who's issuing Router Advertisement (RA) messages. A RA message by itself isn't very helpful, since you just get a MAC address and an IPv6 link-local address, but with a little bit of Gravwell query magic, I was able to parse out ARP packets to link the IPv6 address to an IPv4 address, which helps identify the machine.

Read More

Easy Custom Implementations with Gravwell Client Library

Feb 8, 2021 11:32:37 AM / by John Floren posted in developer, API, golang

Version 3.7.0 of the Gravwell open source repository introduces an exciting new feature: a Go library for interacting directly with Gravwell! Our Data Fusion platform has always been about meeting custom analytics needs and not forcing clients onto limited rails for dashboarding, searching, etc. Out-of-the-box only gets you so far, and beyond is where our customers get into doing some really, really cool stuff.

Open sourcing the Gravwell client library makes it much faster for users to get any custom code up and running - which means less time to ingestion, automation, alerting, and other juicy data goodness. This post will show how to instantiate & authenticate a client, then give a few examples of what you can do.

Read More

4.1.0 Feature Spotlight: Upload Data from the Gravwell UI

Jan 12, 2021 8:54:36 AM / by John Floren posted in Software Updates, ingester, pcap

Sometimes, you just need to get data into Gravwell without setting up any ingesters--maybe you want to analyze a collection of log files somebody emailed you, or maybe you've got a pcap file from Wireshark. We've had command-line tools for this for years, but with Gravwell 4.1.0 we're pleased to announce a new feature: a flexible and easy-to-use interface for ingesting data inside the web interface! This UI lets you drag-and-drop line-delimited logs, packet capture files, or entries downloaded from a Gravwell query; Gravwell will figure out what you gave it and parse it appropriately.

Read More

Amp Up Your Data Analysis with the new Zeek Kit

Nov 16, 2020 9:30:00 AM / by John Floren posted in Security, docker, Bro, kits, DNS, zeek

Zeek can give you so much insight into what's going on in your network, but it can feel like drinking from the firehose - dozens of files full of terse log entries, and no easy way to cross-reference or merge them. That's where Gravwell's new Zeek kit comes in. It's a suite of pre-built queries, dashboards, and more which can help you make sense of what Zeek's telling you with a few clicks.

Read More

Gravwell Weather Data Kit - Look Ma, No Ingester!

Aug 3, 2020 2:48:20 PM / by John Floren posted in ingester, HOWTO, first time, kits

Maybe you've just signed up for Gravwell Community Edition and are not quite sure where to start. There are a lot of features in Gravwell, and a lot of different ingesters for pulling in data. Gravwell 4.0 includes a kit that can collect data without any external ingester--and it helps you analyze everyone's favorite topic, the weather!

Read More

Gravwell 3.3.11

May 8, 2020 2:27:58 PM / by John Floren posted in Community Edition

Today we released Gravwell 3.3.11, hot on the heels of last week's 3.3.10. In our previous post, we'd said that 3.3.9 was the final planned release before our big 3.4.0 version, but there were a few important fixes we wanted to get out ASAP! These two releases were almost entirely bug-fixes, except for two little features we snuck in; we'll start by talking about the bug-fixes first and save the fun stuff for the end!

Read More

Gravwell Ingester Preprocessors

Mar 30, 2020 10:05:08 AM / by John Floren posted in ingester

Gravwell's ingesters can pull data from a wide variety of sources and we advocate keeping raw data formats for root cause analysis, but sometimes it's nice to massage the data a little before sending it to the indexers. Maybe you're getting JSON data sent over syslog and would like to strip out the syslog headers. Maybe you're getting gzip-compressed data from an Apache Kafka stream. Maybe you'd like to be able to route entries to different tags based on the contents of the entries. Gravwell's ingest preprocessors make this possible by inserting one or more processing steps before an entry is sent upstream to the indexer.

Read More

New Release with Netflow v9 Support for Gravwell

Jan 8, 2020 9:30:00 AM / by John Floren posted in Network Analytics, Integrations

Gravwell has officially supported Netflow v5 and IPFIX for some time. As of Gravwell 3.3.3, we're happy to announce that we now support Netflow v9 as well! This post will talk about the essential differences between Netflow v9 and IPFIX, how we implemented support, and how to get up and running with Netflow v9 ingest. We'll also talk about some pretty serious efficiency improvements we made in our IPFIX/Netflow v9 parsing module.

Read More

Announcing Gravwell's Office 365 Management Log Ingester

Dec 17, 2019 11:45:00 AM / by John Floren posted in Events, EventLog, Microsoft, Windows, Integrations

If your enterprise is using Office 365, your users are generating log entries every time they log in, upload files to OneDrive, send an email--the logging is pretty extensive! You can analyze these log events in the O365 console, but wouldn't it be nice to pull them into Gravwell and correlate with other data sources? Thanks to the new Office 365 ingester, you can.

Read More

Introducing Gravwell Macros

Nov 27, 2019 9:24:55 AM / by John Floren posted in Software Updates

One of the exciting new features in Gravwell 3.3.0 is search macros. Anyone who's experimented much with Gravwell knows you may often end up crafting a long and complex regular expression which you'll want to use over and over, but such a long regex makes the query hard to work with. Macros let you turn that long regular expression (or any other part of a search query) into a much shorter name you can use again and again.

Read More