Easy Custom Implementations with Gravwell Client Library

Feb 8, 2021 11:32:37 AM / by John Floren posted in developer, API, golang

Version 3.7.0 of the Gravwell open source repository introduces an exciting new feature: a Go library for interacting directly with Gravwell! Our Data Fusion platform has always been about meeting custom analytics needs and not forcing clients onto limited rails for dashboarding, searching, etc. Out-of-the-box only gets you so far, and beyond is where our customers get into doing some really, really cool stuff.

Open sourcing the Gravwell client library makes it much faster for users to get any custom code up and running - which means less time to ingestion, automation, alerting, and other juicy data goodness. This post will show how to instantiate & authenticate a client, then give a few examples of what you can do.

Read More

Fellow Go Devs, Here's Our Experience Moving to Go Modules From Dep

Oct 17, 2019 2:14:57 PM / by John Floren posted in developer, golang

When we started Gravwell years ago, we knew it was going to be a significant undertaking requiring some serious tooling under the hood. Building a custom data lake and analytics platform from scratch that can scale to hundreds of TB/day ain't easy. We chose Go for a lot of reasons and that choice has paid dividends in terms of what we've been able to accomplish in so short a time.

This post is about our tooling, and some of the lessons we have learned along the way in managing a large Go codebase. A few weeks ago Gravwell made the switch to Go modules on both our open source github repositories and our internal repo. Let's talk about about our planned workflow going forward and a few caveats we've run into.


Read More

Amazon Kinesis Streams and Gravwell

Dec 5, 2017 1:11:03 PM / by John Floren posted in developer, API, Amazon, AWS, ingester, Kinesis

Amazon’s Kinesis Streams service provides a powerful way to aggregate data (logs, etc.) from a large number of sources and feed that data into multiple data consumers. For instance, a large enterprise might use one Kinesis stream to gather log data from their cloud infrastructure and another stream to aggregate sales data from the web services running on that infrastructure. Once the data is in the stream, it remains available for up to a day (or optionally longer) for any number of applications to read it back for processing and analysis. This is particularly useful to customers that want to deploy and destroy virtual machines on a whim; data is stored in the stream, rather than the ephemeral VMs.

Read More