Blog

Announcing the new Gravwell HTTP Ingester

Feb 7, 2019 11:22:19 AM / by John Floren

Gravwell recently introduced a new ingester which accepts entries via HTTP POST requests. Now it's easy to send arbitrary data to Gravwell via scripts using only the curl command. In this blog post, we'll use the HTTP ingester to build a weather-monitoring dashboard!

Environment

In this blog post, we assume the hostname "gravwell" resolves to the machine running Gravwell.

OpenWeatherMap

We used OpenWeatherMap (openweathermap.org) to gather our weather data. Their API allows us to automatically pull down JSON-formatted weather information for free. If you're following along on your own Gravwell system, you'll need an OpenWeatherMap API key, which can be obtained at https://openweathermap.org/appid

Setting up the HTTP Ingester

If you're using the debian repos it's as easy as:

sudo apt install gravwell-http-ingester

The HTTP POST ingester is available in the Gravwell Debian repository or for standalone download at https://dev.gravwell.io/docs/#!quickstart/downloads.md. We installed the ingester on the same machine as the primary Gravwell instance, for simplicity.

With the ingester installed, we needed to set up a URL to send data to. We modified /opt/gravwell/etc/gravwell_http_ingester.conf, deleting the default Listener declaration and making a new one for weather data:

[Listener "weather"]
     URL="/weather"
     Tag-Name=weather

The ingester defaults to listening for connections on port 8080. This configuration specifies that POST requests to /weather should be ingested with the tag "weather".

After restarting the ingester (systemctl restart gravwell_http_ingester.service), it was ready to go. (You may need to set the Ingest-Secret and Cleartext-Backend-Target fields of the config file if the ingester is not co-resident with the main Gravwell instance)

Getting Weather Data

To actually fetch the weather data and send it to Gravwell, we wrote a simple shell script. The script pulls weather data for three locations around the country, in this case specified via latitude/longitude coordinates (see the OpenWeatherMap API docs for more options):

#!/bin/sh
curl "http://api.openweathermap.org/data/2.5/weather?lat=35.1091&lon=-106.5508&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-
curl "http://api.openweathermap.org/data/2.5/weather?lat=46.901&lon=-119.6306&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-
curl "http://api.openweathermap.org/data/2.5/weather?lat=47.1430&lon=-99.7791&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-

Note that YOUR_KEY_HERE should be replaced with an OpenWeatherMap API Key (https://openweathermap.org/appid).

To get samples at regular intervals, we marked the script as executable and set up a cron job to run every minute:

* * * * * /home/john/bin/pushweather.sh >/dev/null 2>&1

Analyzing the data

By simply running the query `tag=weather`, we can verify that weather data is coming in:

rawdata

 

The obvious next step is to look at temperature for each city. Note that OpenWeatherMap reports temperature in degrees Kelvin, so we use the eval module to convert to Fahrenheit in the pipeline:

tag=weather json main.temp main.humidity main.pressure name | eval setEnum("temp", 1.8*(temp-273.15)+32) | mean temp by name | chart mean by name
temps

 

By setting the resulting chart to "Area Chart" in the graph options menu (the jagged line icon on the right-hand side, above the gear icon), we obtain a nice contrasty graph showing temperature trends in three cities.

Building a dashboard

To build a convenient dashboard of weather info, we first create a new dashboard named "Weather" with a default timeframe of a day, then re-run the temperature search above and add it to the dashboard:

addtodash

 

addtodash2

 

Opening the dashboard should now show a tile containing the temperature chart, which we can resize for better viewing:

dash1

 

We can add similar searches to extract barometric pressure and humidity for a particular city:

dash2

 

Summary

The HTTP POST Ingester makes it easy to get arbitrary data into Gravwell with just a 'curl' command. We keep finding new ways to apply it, and we hope you'll take some time to experiment on your own. If you find a particularly cool use for the HTTP Ingester (or Gravwell in general!), we'd love to hear about it at info@gravwell.io!

Topics: Gravwell Story, ingester, Community Edition

John Floren

Written by John Floren

John's been writing Go since before it was cool and developing distributed systems for almost as long.