(This post is part one of a two-part technology series around building and using an IPMI ingester and kit. Part two coming soon.)
In many data aggregation and analysis tools, the ecosystem is fully closed source, and often even data ingest protocols are proprietary. This means that if you want to ingest a novel data format of your own, you’re either, a) $%*! out of luck, or b) forced to collapse your data into some form of low performance, textual, line-delimited data that a generic log ingester will work with.
At Gravwell HQ, we take a different approach. All of our ingesters are open source and freely available under a BSD license, and our ingest framework is open and available as a Go library. In this post, we’ll be taking a tour of how we wrote a real and officially supported Gravwell ingester: the new Gravwell IPMI Ingester. We’ll cover how we manage configuration files, set up and manage indexer connections, and transform IPMI data into a flexible JSON schema before sending it out.
Recent Posts
In today's blog, we’ll give a short overview of the transaction module introduced in our most recent update: Gravwell 4.1.5. The transaction module is a powerful module that can rewrite individual entries into grouped entries based on any number of keys--essentially, the transaction module allows you to collate entries based on a given criteria.
Gravwell 4.1 introduces a new module - Enrich - that can add static data to every entry in a query. Sometimes you need to add static data to a dataset, such as the standard deviation itself across all entries in the dataset or annotations about the query, or you may want to fuse several data points from a resource. The enrich module provides this simple but important feature.
The Gravwell team is happy to announce the release of Gravwell 4.1.0 - Gamma Burst.
A few highlights of what's included in the new release:
- Compound Query support
- Web UI based ingester
- A new “enrich” module
- Temporal mode in the “dump” module
- Internal performance and stability improvements
(Current users - visit the download page for instructions on updating. For a complete list of changes, see the Gravwell 4.1.0 release notes)
We’ll have a series of blog posts discussing the various features of Gravwell 4.1.0, but we wanted to get started with our favorites - Compound Queries.
What’s in a Domain Name? That which we call a CNAME by any other AAAA record would still be used by malware to steal your data. This article introduces the Gravwell CoreDNS Kit, which provides dashboards, queries, and other resources to help you quickly analyze data from a CoreDNS instance using the Gravwell CoreDNS plugin.
Gravwell is designed to work with your data, in your infrastructure, and within your constraints. Whether you have petabytes of packet capture, data-at-rest sensitivity requirements, or are simply integrating existing infrastructure, Gravwell is built to enable a workflow that meets your needs. Today we’ll look at an example integration with multiple Google Stenographer installations, our new Gravwell Packet Fleet ingester, and a powerful new feature in Gravwell Big Bang - Actionables.