When you try plotting on a graph Splunk metric split by a dimension with the same name (as the metric itself) will show NULL instead of the dimension.

The Problem

Let’s rewind a little.

Below is the payload that is sent to Splunk HEC and you will notice that there are 2 “statuses”:

• "status": "success" – which is one of the dimensions and it can represent a collector/monitor status
• "metric_name:status": 0 – which is the actual metric value that was collected by the collector/monitor

{
    "time": 1664970920,
    "event": "metric",
    "host": "host_5.splunk.com",
    "index": "d_telegraf_metrics",
    "fields": {
        "collector": "collector_a",
        "status": "success",
        "metric_name:query_time_seconds": 10.869,
        "metric_name:status": 0
    }
}

In the perfect world where you would probably rename one of these not to confuse the end-user in Splunk, but that (living in a perfect world) is not always the case.

As a result, we end up with NULLs in the graphs 🙁

The Solution

Lucky for us Splunk’s search language (SPL) is very powerful and flexible and with two little modifications to the “original” SPL (that was produced by the Metrics Analyzer), we can solve the issue.

All you need to do is :

1. instead of prestats=true rename the metric function result using as command.
2. update the avg function in the timechart command to use the renamed field name.

Original SPL:

Revised SPL:

The Result

More posts about Splunk