Gravwell Query of the Week

Check for "Bouncing" Ingesters

Written by Gravwell | Apr 19, 2021 11:56:13 PM

This query checks the logs in the Gravwell tag for ingester connect/disconnect events and charts the frequency of these events per ingester. If you see an ingester disconnecting and reconnecting frequently, it means one of two things: either there's something wrong with your connection, or the ingester is getting periodically kicked by the indexer(s) due to inactivity. Either situation warrants investigation! Normally, ingesters should only reconnect after a software upgrade.

tag=gravwell regex "INFO (?P<ingester>\S+).*\s+(?P<action>\S*connected)\s+(to|from)" | stats count by ingester | chart count by ingester

Below is an example of a table showing the results for this query: