Gravwell recently introduced a new ingester which accepts entries via HTTP POST requests. Now it's easy to send arbitrary data to Gravwell via scripts using only the curl command. In this blog post, we'll use the HTTP ingester to build a weather-monitoring dashboard!
Environment
In this blog post, we assume the hostname "gravwell" resolves to the machine running Gravwell.
OpenWeatherMap
We used OpenWeatherMap (openweathermap.org) to gather our weather data. Their API allows us to automatically pull down JSON-formatted weather information for free. If you're following along on your own Gravwell system, you'll need an OpenWeatherMap API key, which can be obtained at https://openweathermap.org/appid
Setting up the HTTP Ingester
If you're using the debian repos it's as easy as:
sudo apt install gravwell-http-ingester
The HTTP POST ingester is available in the Gravwell Debian repository or for standalone download at https://dev.gravwell.io/docs/#!quickstart/downloads.md. We installed the ingester on the same machine as the primary Gravwell instance, for simplicity.
With the ingester installed, we needed to set up a URL to send data to. We modified /opt/gravwell/etc/gravwell_http_ingester.conf, deleting the default Listener declaration and making a new one for weather data:
[Listener "weather"]
URL="/weather"
Tag-Name=weather
The ingester defaults to listening for connections on port 8080. This configuration specifies that POST requests to /weather should be ingested with the tag "weather".
After restarting the ingester (systemctl restart gravwell_http_ingester.service), it was ready to go. (You may need to set the Ingest-Secret and Cleartext-Backend-Target fields of the config file if the ingester is not co-resident with the main Gravwell instance)
Getting Weather Data
To actually fetch the weather data and send it to Gravwell, we wrote a simple shell script. The script pulls weather data for three locations around the country, in this case specified via latitude/longitude coordinates (see the OpenWeatherMap API docs for more options):
#!/bin/sh
curl "http://api.openweathermap.org/data/2.5/weather?lat=35.1091&lon=-106.5508&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-
curl "http://api.openweathermap.org/data/2.5/weather?lat=46.901&lon=-119.6306&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-
curl "http://api.openweathermap.org/data/2.5/weather?lat=47.1430&lon=-99.7791&APPID=YOUR_KEY_HERE" | curl gravwell:8080/weather -X POST -d @-
Note that YOUR_KEY_HERE should be replaced with an OpenWeatherMap API Key (https://openweathermap.org/appid).
To get samples at regular intervals, we marked the script as executable and set up a cron job to run every minute:
* * * * * /home/john/bin/pushweather.sh >/dev/null 2>&1
Analyzing the data
By simply running the query `tag=weather`, we can verify that weather data is coming in:
The obvious next step is to look at temperature for each city. Note that OpenWeatherMap reports temperature in degrees Kelvin, so we use the eval module to convert to Fahrenheit in the pipeline:
tag=weather json main.temp main.humidity main.pressure name | eval setEnum("temp", 1.8*(temp-273.15)+32) | mean temp by name | chart mean by name
By setting the resulting chart to "Area Chart" in the graph options menu (the jagged line icon on the right-hand side, above the gear icon), we obtain a nice contrasty graph showing temperature trends in three cities.
Building a dashboard
To build a convenient dashboard of weather info, we first create a new dashboard named "Weather" with a default timeframe of a day, then re-run the temperature search above and add it to the dashboard:
Opening the dashboard should now show a tile containing the temperature chart, which we can resize for better viewing:
We can add similar searches to extract barometric pressure and humidity for a particular city:
Summary
The HTTP POST Ingester makes it easy to get arbitrary data into Gravwell with just a 'curl' command. We keep finding new ways to apply it, and we hope you'll take some time to experiment on your own. If you find a particularly cool use for the HTTP Ingester (or Gravwell in general!), we'd love to hear about it at info@gravwell.io!
