Plotting Splunk with the same metric and dimension names shows NULL

When you try plotting on a graph Splunk metric split by a dimension with the same name (as the metric itself) will show NULL instead of the dimension.

Splunk timechart visualisation with breakdown by dimension with the same metric and dimension names will show NULL

The Problem

Let’s rewind a little.

Below is the payload that is sent to Splunk HEC and you will notice that there are 2 “statuses”:

  • "status": "success" – which is one of the dimensions and it can represent a collector/monitor status
  • "metric_name:status": 0 – which is the actual metric value that was collected by the collector/monitor
    "time": 1664970920,
    "event": "metric",
    "host": "",
    "index": "d_telegraf_metrics",
    "fields": {
        "collector": "collector_a",
        "status": "success",
        "metric_name:query_time_seconds": 10.869,
        "metric_name:status": 0

In the perfect world where you would probably rename one of these not to confuse the end-user in Splunk, but that (living in a perfect world) is not always the case.

As a result, we end up with NULLs in the graphs 🙁

The Solution

Lucky for us Splunk’s search language (SPL) is very powerful and flexible and with two little modifications to the “original” SPL (that was produced by the Metrics Analyzer), we can solve the issue.

All you need to do is :

  1. instead of prestats=true rename the metric function result using as command.
  2. update the avg function in the timechart command to use the renamed field name.

Original SPL:

original Splunk SPL that was causing NULL in graphs

Revised SPL:

fixed Splunk SPL that shows the breakdown by dimension with the same name as the metric

The Result

Splunk fixed timechart visualisation

More posts about Splunk

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.