Not constantly. Not with a voice assistant persona. Just a short spoken summary when a completed turn arrived, so I could keep working without staring at the terminal.

Both Codex and Claude Code already emit useful telemetry. OpenTelemetry was already present in the workflow. The slightly unreasonable solution was to route that telemetry into a small local speech service.

The result looks like this:

The Collector remains the central routing point. Splunk’s existing exporters stay intact. Speech is an additional filtered branch.

The basic idea

The speech bridge is a small FastAPI service.

It accepts authenticated OTLP/HTTP protobuf logs and traces, correlates them into completed agent turns, summarises the turn with Gemini, generates audio through Cartesia, writes the returned WAV bytes to a temporary file, and plays them sequentially with macOS afplay.

You can find the code on GitHub.

Getting events out of the agents

For Codex (as it doesn’t emit its response via OTEL), I used lifecycle hooks.

The prompt event is sent when a prompt is submitted:

printf '%s\n' \
  '{"session_id":"synthetic-codex","turn_id":"turn-1","prompt":"Add tests"}' \
  | python3 scripts/codex_otel_hook.py UserPromptSubmit

The final response is sent on stop:

printf '%s\n' \
  '{"session_id":"synthetic-codex","turn_id":"turn-1","last_assistant_message":"Tests added"}' \
  | python3 scripts/codex_otel_hook.py Stop

The hook does not send directly to the speech service. It sends unauthenticated OTLP to the local Collector:

http://127.0.0.1:4318/v1/logs

The Collector then routes the relevant records to the authenticated speech exporter.

For Claude Code, I used its prompt and assistant-response events together with enhanced OTLP traces. Claude’s event model is slightly different, so the speech bridge accepts both logs and traces.

The important point is that the agent does not need to know that speech exists. It just emits telemetry.

The Collector configuration

The agents do not send directly to the speech bridge. They send ordinary OTLP to the local OpenTelemetry Collector. The Collector keeps Splunk working as before, filters events by agent, and adds speech as a separate authenticated branch.

The configuration below is additive. It belongs in the existing Collector configuration (as described in Send Claude Code and Codex logs to Splunk); it is not a complete replacement for agent_config.yaml.

exporters:
  otlp_http/speech/codex:
    endpoint: http://host.docker.internal:17860
    headers:
      x-speech-ingest-token: "${SPEECH_INGEST_TOKEN}"
    sending_queue:
      enabled: true
      num_consumers: 1
      queue_size: 10000
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 10s
      max_elapsed_time: 0s

  otlp_http/speech/claude:
    endpoint: http://host.docker.internal:17860
    headers:
      x-speech-ingest-token: "${SPEECH_INGEST_TOKEN}"
    sending_queue:
      enabled: true
      num_consumers: 1
      queue_size: 10000
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 10s
      max_elapsed_time: 0s

processors:
  filter/drop_claude_logs:
    error_mode: ignore
    logs:
      log_record:
        - resource.attributes["service.name"] == "claude_code"
        - resource.attributes["service.name"] == "claude"
        - resource.attributes["service.name"] == "claude-code"

  filter/drop_non_claude_logs:
    error_mode: ignore
    logs:
      log_record:
        - resource.attributes["service.name"] != "claude_code" and resource.attributes["service.name"] != "claude" and resource.attributes["service.name"] != "claude-code"

  filter/drop_non_claude_traces:
    error_mode: ignore
    traces:
      span:
        - resource.attributes["service.name"] != "claude_code" and resource.attributes["service.name"] != "claude" and resource.attributes["service.name"] != "claude-code"

  batch/speech:
    timeout: 1s

service:
  pipelines:
    logs/codex:
      receivers: [otlp]
      processors: [filter/drop_claude_logs, transform/normalize, batch]
      exporters: [splunk_hec/codex, otlp_http/speech/codex]

    logs/claude:
      receivers: [otlp]
      processors: [filter/drop_non_claude_logs, transform/normalize, batch]
      exporters: [splunk_hec/claude, otlp_http/speech/claude]

    traces/claude:
      receivers: [otlp]
      processors: [filter/drop_non_claude_traces, batch/speech]
      exporters: [otlp_http/speech/claude]

The existing Collector already provides the otlp receiver, the Splunk exporters, transform/normalize, and the regular batch processor. This chapter only adds the speech-specific pieces.

What the pipelines do

The Codex pipeline drops known Claude service names, then sends the remaining logs to both Splunk and the Codex speech exporter. The Claude pipeline does the inverse. The accepted names are claude, claude_code, and claude-code because different Claude Code versions report different values.

The Claude trace pipeline applies the same filter to spans. Those traces help the bridge correlate a completed Claude turn. The separate speech exporters keep the Codex and Claude delivery queues independent.

The exporter endpoint is a base URL. OTLP/HTTP sends logs to /v1/logs and traces to /v1/traces. host.docker.internal is the Docker Desktop hostname that lets the Collector container reach the host-native FastAPI service.

Authentication and failure boundaries

The hooks send unauthenticated OTLP to the local Collector. The Collector adds x-speech-ingest-token when forwarding to the bridge. The token must be available inside the Collector container and must never be committed or printed.

services:
  splunk-otel:
    environment:
      - SPEECH_INGEST_TOKEN=${SPEECH_INGEST_TOKEN}

If the Collector cannot reach the bridge, its exporter queue retries delivery. If the bridge accepts the event but Gemini, Cartesia, or afplay fails, the speech worker increments failed, calls task_done(), and continues with the next FIFO item.

Validate before restarting

Validate the merged configuration before recreating the Collector:

docker compose --env-file /Users/ir/.config/otel2speech/collector.env -f /Users/ir/workspaces/splunk-pai-demo/docker/docker-compose.yaml config --quiet

docker compose --env-file /Users/ir/.config/otel2speech/collector.env -f /Users/ir/workspaces/splunk-pai-demo/docker/docker-compose.yaml up -d --force-recreate splunk-otel

The repository keeps this additive configuration in config/collector-speech-snippet.yaml. It is intentionally a snippet because the existing Collector configuration contains the Splunk receivers, exporters, and environment-specific settings.

Correlation is the real problem

Text-to-speech is easy.

Knowing when there is something worth speaking is harder.

Agent telemetry can arrive out of order. A response can arrive before the root trace. Multiple turns can be interleaved in the same session. There are also subagent and tool events that should not become spoken announcements.

The bridge therefore has a small in-memory correlator.

It tracks:

• source;
• session ID;
• turn ID;
• prompt;
• final response;
• trace and span identifiers;
• timestamps;
• whether the event belongs to the main agent.

A turn is emitted only when the required pieces are present. For Claude, the completed root span helps anchor the turn. For Codex, the prompt and final response lifecycle events provide the boundary.

The correlator also deduplicates events and filters out irrelevant agent activity.

One FIFO worker

Once a completed turn is emitted, it goes into one unbounded FIFO queue.

The worker processes exactly one turn at a time:

Sequential playback matters. If two agent turns complete close together, the audio should not overlap or race.

The worker also treats each turn as isolated work. A failure must not terminate the loop or prevent later turns from being spoken.

The statistics are deliberately simple:

• enqueued: normalised turns accepted;
• completed: successfully delivered turns, including dry-run;
• failed: turns where generation or delivery failed;
• last_source: source of the latest successful turn.

Every queue item calls task_done(), including failures.

Gemini is optional, but Cartesia is not

Gemini generates a short spoken summary of the completed turn.

The prompt asks for plain spoken language without Markdown or raw prompt quoting. It targets 20–30 words, but accepts any non-empty valid sentence within the configured maximum.

If Gemini is unavailable, lacks credentials, times out, or returns invalid output, the bridge uses a fixed fallback sentence:

Completed turn received and queued for speech; review the result when convenient, then continue with the next planned action safely.

That fallback is still sent through Gemini’s normal downstream path:

fallback sentence → Cartesia → afplay

This distinction matters. Gemini is a summary provider. It is not an audio backend.

Cartesia is the only speech synthesiser

The final design has one synthesis backend: Cartesia.

Codex and Claude retain distinct Cartesia voice IDs. Both use a Cartesia speed of 1.3 through SPEECH_RATE.

Cartesia returns audio bytes. The bridge writes those bytes to a temporary WAV file and calls:

afplay /path/to/temporary.wav

afplay is only a playback mechanism. It does not synthesise speech.

The temporary file is deleted in a finally block whether playback succeeds or fails.

There is no local macOS voice-synthesis fallback anymore. That makes failure behaviour much easier to reason about:

• missing Cartesia credentials: delivery fails;
• Cartesia request failure: delivery fails;
• empty Cartesia audio: delivery fails;
• afplay failure: delivery fails;
• worker increments failed and continues to the next FIFO item.

No alternate speech synthesis is attempted.

Keeping logs content-free

The bridge handles prompt and response text in memory while correlation and speech work are pending.

Application logs contain source names, counts, backend names, and failure categories. They do not log raw prompts, responses, or generated speech text.

This is particularly important because OTEL pipelines often contain full-content telemetry. The Collector configuration should be reviewed carefully, especially when adding new exporters.

The speech branch should be filtered and isolated. Splunk’s existing full-content exporters should remain intact.

Running it

The setup is intentionally manual:

uv sync
cp .env.example .env
uv run python scripts/ensure_shared_token.py
uv run uvicorn app.main:app --host 0.0.0.0 --port 17860

The important configuration is roughly:

SPEECH_INGEST_TOKEN=replace-with-a-long-random-token

GEMINI_API_KEY=
GEMINI_MODEL=gemini-3.5-flash

CARTESIA_API_KEY=
CARTESIA_MODEL=sonic-3.5
CARTESIA_CONTAINER=wav

SPEECH_RATE=1.3

CODEX_CARTESIA_VOICE_ID=...
CLAUDE_CARTESIA_VOICE_ID=...

Secrets stay in ignored environment files. The repository does not contain API keys or ingest tokens.

Why abuse OpenTelemetry this way?

This is not a conventional voice-agent architecture.

It is a small adapter built on top of telemetry that already exists. The agents produce lifecycle events, the Collector provides routing, and the speech bridge turns selected completed events into audio.

The advantages are practical:

• no agent-specific speech integration;
• no changes to the agent’s core execution loop;
• existing session and trace identifiers provide correlation;
• Collector routing keeps the speech service separate from the agents;
• the same telemetry can continue flowing to Splunk.

The tradeoffs are equally clear:

• the queue is in memory;
• restarting the bridge loses pending work;
• speech depends on network access to Gemini and Cartesia;
• playback is macOS-specific;
• telemetry ordering and filtering require careful testing.

For a production system, I would probably add durable state, explicit event schemas, retries, health metrics, and a proper audio service.

For a local coding workflow, this is enough.

OpenTelemetry was already telling me what the agents had done. I just gave it a voice.

Find the code on https://github.com/ilyaresh/otel2speech

The post How I (ab)used OpenTelemetry to make Claude Code and Codex speak to me appeared first on ISbyR.

This is the setup that ended up working. Two harnesses, one collector, one Splunk index (ai_usage) and two sourcetypes so I can tell them apart.

An Overview

Both harnesses speak OTLP. I send both to the same collector on 127.0.0.1:4318, then split them inside the collector by service.name and export to Splunk HEC with different sourcetype values.

Only logs. I deliberately left metrics and traces off. The log events already have a lot of info like token counts, durations, and tool details.

Claude Code config

Claude Code is configured with env vars. Add them in ~/.claude/settings.json under env so you don’t have to export them in the shell:

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_METRICS_EXPORTER": "none",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://127.0.0.1:4318",
    "OTEL_LOG_TOOL_DETAILS": "1"
  }
}

Notes on the bits that matter:

• CLAUDE_CODE_ENABLE_TELEMETRY=1 – is the master switch. Nothing happens without it.
• OTEL_METRICS_EXPORTER=none – logs only, on purpose.
• OTEL_EXPORTER_OTLP_ENDPOINT – is just the base URL. Claude Code appends the signal path itself.
• OTEL_LOG_TOOL_DETAILS=1 – includes which tool ran (Bash, Edit, MCP calls, etc.). That’s the part I actually care about.

Restart Claude Code after changing this, as the env block is read at startup.

Codex config

Codex config is a bit different to Claude. It’s not env vars — it’s a TOML block in ~/.codex/config.toml:

[otel]
environment = "local"
log_user_prompt = false

exporter = { otlp-http = {
  endpoint = "http://127.0.0.1:4318/v1/logs",
  protocol = "binary"
}}

trace_exporter = "none"
metrics_exporter = "none"

Two things I got wrong the first time:

• endpoint – it needs the full path /v1/logs, not just the base URL like Claude. If you give it the base URL, it won’t land.
• protocol = "binary" is protobuf. That lines up with Claude’s http/protobuf, so the same collector receiver handles both.

And a few other parameters worth mentioning:

• log_user_prompt = false – keeps my actual prompts out of Splunk. I only want the metadata, not the text I typed.
• trace/metrics_exporter = "none" – self-explanatory: we don’t want metrics or traces at this point

The collector

The collector is the Splunk distro of the OTEL collector, run in Docker. docker-compose.yaml:

services:
  splunk-otel:
    image: quay.io/signalfx/splunk-otel-collector:latest
    container_name: splunk-otel-collector
    environment:
      - SPLUNK_HEC_TOKEN=<your-hec-token>
      - SPLUNK_HEC_URL=https://host.docker.internal:38088/services/collector
      - SPLUNK_CONFIG=/etc/otel/collector/agent_config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8888:8888"   # collector's own metrics
    volumes:
      - ./agent_config.yaml:/etc/otel/collector/agent_config.yaml:ro
    restart: always

The important bit is host.docker.internal in SPLUNK_HEC_URL. The collector runs in a container; Splunk runs on the host, so localhost from inside the container would be the container itself. host.docker.internal is how the container reaches the host’s 38088.

HEC port would usually be 8088, but i had it already pre-occupied with something else :-).

The routing bit

This is the part that took the most fiddling. Both harnesses hit the same 4318, but I want them tagged differently in Splunk. The collector’s routing connector does the split, keyed on service.name. agent_config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  routing:
    default_pipelines: [logs/codex]          # anything not matched -> codex
    error_mode: ignore
    table:
      - statement: route() where IsMatch(resource.attributes["service.name"], "^claude(-|$).*")
        pipelines: [logs/claude]

processors:
  transform/normalize:
    error_mode: ignore
    log_statements:
      - set(log.time_unix_nano, log.observed_time_unix_nano) where log.time_unix_nano == 0
      - set(log.body, log.attributes) where log.attributes["event.name"] != nil
      - merge_maps(log.body, resource.attributes, "insert") where IsMap(log.body)
      - keep_keys(log.attributes, []) where IsMap(log.body)
  batch:

exporters:
  splunk_hec/codex:
    token: "${SPLUNK_HEC_TOKEN}"
    endpoint: "${SPLUNK_HEC_URL}"
    source: "codex"
    sourcetype: "otel:ai:codex"
    index: "ai_usage"
    tls:
      insecure_skip_verify: true

  splunk_hec/claude:
    token: "${SPLUNK_HEC_TOKEN}"
    endpoint: "${SPLUNK_HEC_URL}"
    source: "claude_code"
    sourcetype: "otel:ai:claude"
    index: "ai_usage"
    tls:
      insecure_skip_verify: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [routing]
    logs/codex:
      receivers: [routing]
      processors: [transform/normalize, batch]
      exporters: [splunk_hec/codex]
    logs/claude:
      receivers: [routing]
      processors: [transform/normalize, batch]
      exporters: [splunk_hec/claude]

How the routing actually resolves:

• Claude Code sends service.name = claude-code-desktop, which matches ^claude(-|$), so it goes down logs/claude.
• Codex sends service.name = codex-app-server, which does not match, so it falls through to default_pipelines: [logs/codex].

So Codex isn’t matched by a rule; it’s the default. That’s fine for two tools, but worth remembering: if a third thing ever sent OTLP to this collector without matching ^claude, it would get labelled as codex. The default is a catch-all, not a Codex-specific match.

insecure_skip_verify: true is because my Splunk HEC is a self-signed cert on localhost. Fine here, not something I’d do against a real endpoint.

Why the transform is there

Without transform/normalize the events landed in Splunk with an empty body and everything buried in OTEL attributes, which is annoying to search. The transform moves the log attributes into the body, merges in the resource attributes (that’s where service.name, versions, etc. live), then drops the now-duplicate attribute copy. The result is a flat JSON event where the fields are just… fields.

Testing it

Of course, after you send Claude Code and Codex logs to Splunk, you want to make sure they are there.

Quick sanity check in Splunk to see if anything is landing, and from both tools?

index=ai_usage earliest=-7d
| stats count, min(_time) as first, max(_time) as last by source, sourcetype
| eval first=strftime(first,"%Y-%m-%d %H:%M"), last=strftime(last,"%Y-%m-%d %H:%M")

In my case:

source        sourcetype        count   first             last
claude_code   otel:ai:claude    6659    2026-06-30 09:30  2026-07-07 09:16
codex         otel:ai:codex     6317    2026-06-30 09:17  2026-07-07 09:16

Both flowing, most recent event a couple of minutes old. Good enough.

A single Claude Code event looks like this (I’ve masked the user fields):

{
  "event.name": "hook_execution_complete",
  "hook_event": "PostToolUse",
  "hook_name": "PostToolUse:mcp__splunk-mcp-server__splunk_run_query",
  "service.name": "claude-code-desktop",
  "service.version": "1.18286.0",
  "session.id": "f11b6155-…",
  "total_duration_ms": "2",
  "os.type": "darwin",
  "user.email": "you@example.com"
}

And a Codex one:

{
  "event.name": "codex.websocket_request",
  "service.name": "codex-app-server",
  "service.version": "0.142.5",
  "model": "gpt-5.5",
  "conversation.id": "019f39b0-…",
  "duration_ms": "2",
  "auth_mode": "Chatgpt",
  "success": "true",
  "telemetry.sdk.language": "rust"
}

Different fields per tool are expected as they’re different products. The common keys (service.name, event.name, event.timestamp, user.email) are enough to build usage dashboards across both.

And you can have Splunk without some pretty dashboards :-)!

What I missed / gotchas

• Claude Code uses env vars, Codex uses a TOML block. I assumed both were env at first.
• Codex needs the full /v1/logs path on the endpoint. Claude only wants the base URL.
• Restart the harness after any config change. The telemetry config is read at startup, not live-reloaded.
• Codex is the default route, not a matched one. Fine for now, but the routing rule only explicitly matches Claude.
• Self-signed HEC on localhost → insecure_skip_verify: true. Don’t carry that into anything real.

Notes

• Don’t commit the compose file with SPLUNK_HEC_TOKEN in it. My local spike had real values inline while I was testing. Before sharing the repo, move them to .env or a secret store, keep .env out of git, and rotate the token if it has already left the machine.
• Metrics and traces are off here on purpose. If you want real OTEL metrics, not dashboard metrics derived from log events, flip OTEL_METRICS_EXPORTER / metrics_exporter back on and add a collector metrics pipeline. The log events already carry what I needed for this dashboard.
• Telemetry field names can change between versions. This worked on the versions listed at the top; check the current field names if a dashboard suddenly goes empty after an update.
• Pin the collector image before turning this into a repeatable setup. splunk-otel-collector:latest is fine for a spike, but it makes blog instructions drift over time.

References

• Codex config reference: https://developers.openai.com/codex/config-reference
• Claude Code monitoring: https://code.claude.com/docs/en/monitoring-usage
• Splunk OpenTelemetry Collector docs: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector
• Splunk HTTP Event Collector event format: https://help.splunk.com/en/splunk-enterprise/get-data-in/get-started-with-getting-data-in/10.0/get-data-with-http-event-collector/format-events-for-http-event-collector
• OpenTelemetry Collector Splunk HEC exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter

The post Send Claude Code and Codex logs to Splunk appeared first on ISbyR.

It is not a full architecture doc. It is more of a map of the words that kept coming up while building the bot.

Call Flow

To give a bit of context, below is the high-level flow that depicts the moving parts of a person calling an AI Voice Agent:

1. Inbound call — the caller dials a Twilio number over the PSTN. Twilio answers and holds the call.
2. Webhook — Twilio sends an HTTP POST to the FastAPI /voice endpoint to ask what to do with the call.
3. TwiML response — FastAPI replies with TwiML containing a <Connect><Stream> instruction, telling Twilio to open a media stream to the app.
4. WebSocket + audio — Twilio opens a WebSocket to /ws and starts streaming the caller’s audio as mu-law frames. Pipecat takes ownership of the connection.
5. Audio → STT — Pipecat passes the raw audio to the STT service (e.g. Cartesia Ink). VAD detects end-of-turn before the audio is sent.
6. Transcript → LLM — STT returns a text transcript. Pipecat appends it to the conversation and sends the full context to the LLM (e.g. Gemini).
7. Response text → TTS — the LLM streams its response back to Pipecat, which forwards the text to the TTS service (e.g. Cartesia Sonic).
8. Audio → Pipecat — TTS converts the text to speech and returns audio frames to Pipecat.
9. Audio stream → Twilio — Pipecat sends the synthesised audio back over the WebSocket as mu-law media messages.
10. Audio → caller — Twilio plays the audio to the caller over the PSTN. The caller hears the agent’s voice and the loop repeats for their next utterance.

So, without further ado, I present to you…

The ABCs of AI Voice Agents

AgentCore

AWS Bedrock AgentCore is a fully managed, enterprise-grade platform that helps developers build, deploy, and scale AI agents, and it was considered as a possible runtime.

The problem, however, was that Twilio couldn’t (or I couldn’t make it) connect to it directly. Twilio Media Streams has a specific WebSocket handshake and message format. AgentCore has its own auth and protocol expectations.

Relevant docs: AWS AgentCore

AI – Artificial Intelligence

For a Voice Agent, AI really means a few different things: speech recognition, text generation, text-to-speech, routing logic, and more. The LLM is only one part of the system.

ASR / STT – Automatic Speech Recognition and Speech-to-Text

These are often used interchangeably. It is the part that turns the caller’s audio into text that the LLM can understand.

In this project, the pipeline uses Cartesia STT (Ink model). Twilio ConversationRelay can also do ASR for you if you use that product instead of raw Media Streams. Other common vendors are Deepgram, Google Cloud STT, AssemblyAI, and OpenAI Whisper-style models.

The practical thing to remember: STT quality is not only about model accuracy. End-of-turn timing, background noise, phone audio, accents, and whether the user interrupts all matter.

Barge-in

Barge-in is when the caller interrupts the agent while the agent is speaking.

This is normal human behaviour. The system has to decide whether to stop speaking, listen, or update the conversation state.

CallSid / StreamSid

These are Twilio identifiers.

CallSid identifies the call. StreamSid identifies the media stream.

Relevant docs: Twilio CallSid and StreamSid

Cartesia

Cartesia is the speech and voice agent platform in this project.

In this project, Cartesia provides the speech components: STT for converting caller audio into text and TTS for converting the model’s response back into audio.

Cartesia can also handle more of the voice application layer: call orchestration, interruptions, deployment, logs, and call analytics (see Cartesia Line).

Cartesia Line

Cartesia Line is a code-first voice agent platform.

Cartesia Line can handle voice/call orchestration, Ink STT, Sonic TTS, interruptions, deployment, logs, and call analytics.

Line also has useful built-in concepts like transfer_call, send_dtmf, and handoffs (if you manage your telephony through them).

Note: If you are using your own Twilio number (which you can easily import with/connect to the Cartesia platform) and stream path, you still need access to the live Twilio REST API to manage the call controls.

Relevant docs: Cartesia Line

Conference

A conference is a Twilio call room with participants.

For warm transfer, the caller can be parked in a conference with hold music or a wait message while the human leg is being dialled. When the human agent accepts the transfer, they join the same conference.

The details matter:  startConferenceOnEnter,  endConferenceOnExit,  waitUrl, status callbacks, and what happens if a participant leaves.

Relevant docs: TwiML <Dial><Conference>.

ConversationRelay

Twilio ConversationRelay is a higher-level Twilio voice AI product (compared to Media Streams).

Instead of sending raw phone audio to your app, Twilio can handle STT and TTS and send structured WebSocket (text) messages to your application. Your app receives prompts/transcripts and sends text tokens back.

That can simplify the audio side. The tradeoff is that you are using Twilio’s ConversationRelay workflow rather than building your own Pipecat audio pipeline.

Relevant docs: Twilio ConversationRelay

DTMF – Dual-Tone Multi-Frequency.

This is the phone keypad signal: 0-9, star, and hash.

In this project, DTMF matters for warm transfer. The human agent answers, hears a brief, and presses 1 to accept the transfer. That is much safer than just dumping the caller onto whatever number was dialled.

Twilio <Gather> can collect DTMF digits.
Cartesia Line has a send_dtmf tool for pressing buttons when the voice agent itself needs to navigate a phone menu.

Relevant docs: Twilio <Gather>  and Cartesia Line  send_dtmf 

E.164

E.164 is the international phone number format, like +614….

It matters because APIs expect phone numbers to be normalised. Transfer tools should store and compare numbers in a predictable format.

HMAC

Hash-based Message Authentication Code is a cryptographic method used to simultaneously verify a message’s integrity and authenticity.

HMAC is used in the current app for the stream token. The /voice webhook validates Twilio’s signature, creates a call state, then mints a short-lived token tied to the CallSid. Twilio echoes that token in the WebSocket start.customParameters, and /ws validates it.

Why not just rely on Twilio’s webhook signature? Because the WebSocket upgrade does not carry the same signed form payload as the /voice POST.

IVR – Interactive Voice Response.

This is the classic “press 1 for…” phone menu. A voice AI receptionist can replace some IVR flows.

In our case, the human agent acceptance step is basically a tiny IVR: “press 1 to accept”.

Latency

Latency is a delay.

In a voice bot, latency is made up of many small delays:

• Twilio media to app.
• STT.
• VAD/end-of-turn.
• LLM first token.
• Tool calls, like Notion lookup.
• TTS first audio.
• Audio buffering back to Twilio.

The Notion (or other data store) lookup work is a good example. A live scan across Notion databases during the greeting made the call feel slow. The better approach was a quick caller lookup first, then background refresh/enrichment, with caching for slower data.

LLM – Large Language Model.

This is the reasoning layer. It reads the conversation, decides what to say, and calls tools when needed. In this project, the model has been Gemini.

The model should not be allowed to invent call control. It needs tools with clear rules: identify the caller, find the routing recipient, create a message, list assignments, cancel an assignment, and transfer a call.

Mark / Clear

mark is how the app asks Twilio to tell it when previously sent audio has finished playing. clear clears buffered audio.

These are Twilio Media Streams messages.

For transfer, mark was the safer way to avoid cutting off the “please hold” sentence when a caller is being transferred. Without it, the app can redirect the call before the caller hears “I will transfer you now”.

Relevant docs: Twilio mark and clear messages

Media Streams

Twilio Media Streams lets you stream call audio to your WebSocket server.

There are unidirectional streams and bidirectional streams. For the voice bot, bidirectional is the useful one because the app needs to receive the caller audio and send the agent audio back.

Twilio sends JSON messages like connected, start, media, mark, and stop. The audio payload is base64. When you send audio back to Twilio, you send media messages. When you want to know that buffered audio has finished playing, you send a mark and wait for Twilio to echo it.

Relevant docs: Twilio Media Streams

Notion

Notion is used as the operational data source in this project.

The bot looks up callers, contractors, clients, assignments, HR directory entries, routing recipients, and message logs from Notion.

Observability

Observability is how you understand what happened during a call.

For voice AI, logs alone are not enough. You want spans and metrics around:

• inbound webhook.
• WebSocket start/stop.
• STT latency.
• VAD/end-of-turn timing.
• LLM latency and tool calls.
• TTS latency.
• transfer lifecycle.
• call cleanup.

Relevant docs: Pipecat metrics

OAuth / JWT

OAuth and JWT are common auth patterns.

They matter in voice AI because every real-time endpoint is exposed to the internet. A public wss:// endpoint that accepts audio should not accept random traffic just because it looks like Twilio.

In this project, Twilio webhook signatures plus HMAC stream tokens were enough for the current Media Streams path. If using another runtime, hosted agent, or browser client, OAuth/JWT-style auth might be more relevant.

PCMU / mu-law / 8 kHz

PCMU, also called G.711 mu-law, is a common phone audio codec.

Twilio Media Streams sends and receives base64-encoded audio/x-mulaw at 8000 Hz for media frames. If you send the wrong audio format back to Twilio, the caller does not get useful audio.

This was one reason a direct “Twilio to some random WebSocket agent runtime” idea was not enough. The server must speak Twilio’s media message protocol and handle the expected audio format.

Pipecat

Pipecat is a framework for building real-time voice and multimodal agents.

In this build it provides the pipeline:

transport input -> STT -> user aggregator / VAD -> LLM -> TTS -> transport output -> assistant aggregator

The useful thing about Pipecat is that it gives names and structure to the pieces. The dangerous thing is assuming the transport protocol is magic. Twilio Media Streams still has its own JSON messages, audio format, CallSid, StreamSid, marks, and stop events.

Relevant docs: Pipecat

PII

Personally Identifiable Information.

Voice AI systems touch a lot of it: names, phone numbers, call recordings, transcripts, assignments, clients, and messages.

In this receptionist project, the agent should not volunteer surnames, phone numbers, or private details. It can use retrieved data to route the call, but it should speak only the minimum needed. For example, first names are usually enough for a transfer offer.

This is not just a prompt issue. Logs, traces, tool results, and evaluation datasets need the same discipline.

PSTN

Public Switched Telephone Network.

This is the normal phone network. When a caller dials a Twilio number from a mobile or landline, they are coming from the PSTN.

The annoying bit is that the PSTN is narrowband and phone-oriented. You are usually dealing with 8 kHz phone audio somewhere in the path, not clean studio audio.

Relevant vendors: Twilio, Telnyx, Vonage

RAG

Retrieval-Augmented Generation.

RAG means retrieving relevant knowledge and giving it to the model before it answers. In this project, Notion lookup is not exactly a generic vector RAG flow, but the idea is similar: fetch caller, contractor, client, assignment, HR directory, and routing context so the model can make the right decision.

The important bit is privacy. Retrieved data should be scoped. The agent should not volunteer surnames, phone numbers, or private records. If a contractor calls, they can get information relevant to their own assignments, not everyone else’s.

RTP

Real-time Transport Protocol.

RTP is how media packets often move in VoIP systems. You may not touch it directly if you are using Twilio Media Streams or ConversationRelay, but it is underneath a lot of telephony.

When people talk about codecs, jitter, packet loss, and media paths, RTP is usually close by.

SigV4

AWS Signature Version 4.

This came up when looking at AWS AgentCore. Direct Twilio-to-AgentCore looked tempting, but it did not fit.

The problems were:

• Twilio <Stream url> cannot practically sign the WebSocket request with AWS SigV4 headers.
• Twilio <Parameter> values arrive inside the later start.customParameters message, not in the initial WebSocket handshake where AgentCore auth would need them.
• Even if auth worked, the protocols did not match. Twilio sends Twilio Media Streams JSON with phone audio. Agent runtimes usually expect their own WebSocket protocol.

The conclusion was simple: you need a bridge or a single service that speaks Twilio on one side and your agent runtime on the other.

Relevant vendors/frameworks: AWS AgentCore, Twilio Media Streams, custom relay service.

SIP

Session Initiation Protocol.

SIP is a signalling protocol used to set up and control voice calls. If you are connecting phone systems, PBXs, trunks, and carriers, SIP eventually appears.

For this receptionist project, I mostly stayed at the Twilio Voice API/TwiML layer rather than managing SIP directly. That was enough for inbound calls, Media Streams, call redirects, and conferences.

State store

State is what the app remembers about the active call.

For a single container, in-memory state can work during local testing. In production, Redis or another shared store becomes important if there are multiple instances or restarts.

In this project state includes the caller, public base URL, allowed transfer numbers, active transfer state, stream token, and whether the call has already been redirected or ended.

Stop event

Twilio sends a stop event when the stream stops or the call ends.

This matters for cleanup. If the caller hangs up and the app does not cancel the pipeline, the model/STT/TTS work can keep running for no reason. Hangups need to cancel the active pipeline, clear state, and avoid trying to speak into a dead call.

Telnyx

Telnyx is another telephony provider.

We looked at Telnyx as an alternative to Twilio. The conclusion was similar for direct-to-AgentCore: Telnyx can stream audio and gives strong call control APIs, but stream_url and stream auth are still about connecting Telnyx to your server. They do not magically make a third-party agent runtime speak Telnyx’s protocol.

So Telnyx can be a good provider, but you still need a relay/bridge unless the voice agent framework directly supports its stream format.

Relevant docs: Telnyx Call Control

Tool calling

Tool calling is when the LLM asks the app to perform an action.

For this receptionist, tools are not optional decoration. They are how the agent does real work:

• identify the caller by phone number.
• find a routing recipient.
• list assignments.
• cancel an assignment.
• create a message.
• start a warm transfer.

The rules around tools matter. For example, transfer_call should only work for previously selected, call-capable recipients. The model should not be allowed to transfer to a random number it heard in the conversation.

Relevant frameworks: Gemini tool declarations, Pipecat tool handlers, Cartesia Line tools, OpenAI/Anthropic tool calling.

TTFB / TTFA

Time To First Byte and Time To First Audio.

For voice, TTFA is often the more useful mental model. The caller does not care when the server got a byte. They care when they hear something useful.

This is why streaming matters. Stream the LLM response if possible. Start TTS early if possible. Do not block the first greeting on slow CRM/database enrichment unless that data is needed to say hello.

TTS

Text-to-Speech.

This is the part that turns the agent’s text back into audio. In the current Pipecat setup, Cartesia TTS does this. Cartesia calls its TTS model family Sonic.

Twilio ConversationRelay can use TTS providers like Google, Amazon, Deepgram, and ElevenLabs depending on configuration and access.

The practical thing to remember: TTS is where latency becomes very obvious. A text chatbot can pause for a second and nobody really notices. A phone caller notices.

Turn-taking

Turn-taking is the question of when the caller’s turn ends and the agent’s turn starts.

Text chat mostly avoids this. Voice does not. A person can pause mid-sentence, say “um”, talk over the agent, press a key, or hang up while the model is thinking.

Good turn-taking needs VAD, STT partial/final transcripts, interruption handling, and prompts that tell the model what it has already said.

Twilio

Twilio is the telephony platform in this project.

It owns the public phone number, receives the inbound PSTN call, calls the /voice webhook, opens the Media Streams WebSocket, sends caller audio to the app, receives generated audio back, and performs the warm transfer call control.

The important thing I had to learn is that Twilio is not just a dumb audio pipe. Twilio has its own call state, identifiers, webhook signatures, TwiML instructions, media stream protocol, status callbacks, conferences, and REST APIs. If you want to move a live caller from an AI stream into a human warm transfer, you need to use Twilio call control deliberately.

Relevant products/frameworks: Twilio Programmable Voice, Twilio Media Streams, TwiML, Twilio REST API, Twilio Conferences, ConversationRelay.

TwiML

Twilio Markup Language.

TwiML is XML that tells Twilio what to do with a call. The webhook gets an inbound call, returns TwiML, and Twilio executes it.

Examples:

<Response> <Connect> <Stream url="wss://example.com/ws" /> </Connect> </Response>

or:

<Response> <Gather numDigits="1"> <Say>Press 1 to accept the call.</Say> </Gather> </Response>

The important bit: TwiML is not just a response format. It is call control.

Relevant docs: Twilio TwiML for Programmable Voice.

VAD

Voice Activity Detection.

VAD decides whether someone is speaking or not. That sounds small, but it is one of the things that makes a voice bot feel natural or painful.

If VAD ends the turn too early, the bot interrupts. If it waits too long, the call feels slow. If it treats background noise as speech, the agent can get confused.

In the Pipecat build, Silero VAD is used in the LLM user aggregator path. Cartesia Line handles turn-taking and interruption detection at the platform layer.

Relevant frameworks: Silero VAD, Pipecat VAD,

Warm transfer

Warm transfer means the caller is not blindly transferred.

The agent first gathers enough context, then contacts the human, gives them a short brief, and only connects the caller if the human accepts.

In this project, the warm transfer flow uses Twilio call control:

1. The model calls the transfer tool.
2. The app prepares transfer state.
3. The bot tells the caller to hold.
4. The app waits for a Twilio media mark confirming the hold prompt played.
5. The app redirects the live caller call into a conference.
6. The app calls the human agent.
7. The human hears a brief and presses 1.
8. Twilio joins the human to the conference.
9. If no answer, busy, failed, or declined, the caller returns to the AI to leave a message.

This is more work than a cold transfer, but it avoids a bad caller experience.

Webhook

A webhook is an HTTP callback from a service to your app.

Twilio uses webhooks heavily. Incoming call? Webhook. Human leg answered? Webhook. Gathered digit? Webhook. Transfer status changed? Webhook. Conference event? Webhook.

The important bit is that Twilio signs webhook requests using the public URL it called. If your app validates the wrong URL because it sees an internal Cloud Run URL or localhost, validation fails.

In our app, request validation builds candidate public URLs from PUBLIC_BASE_URL, PUBLIC_HOST, forwarded headers, host headers, and the ASGI URL.

Relevant docs: Twilio webhook security

WebSocket / WSS

WebSocket is the long-lived connection used for real-time media or event streams. WSS is WebSocket over TLS.

For Twilio voice AI, the URL must be public and secure. Localhost does not count unless it is behind a public tunnel. Cloud Run, ngrok, and load balancers can also make this confusing because the app may see an internal http://… URL while Twilio called the public https://… URL.

In the current app, /voice returns TwiML that points Twilio at /ws, and /ws validates a short-lived stream token before accepting the call media.

The post The ABCs of Voice AI Agents appeared first on ISbyR.

MentionVault’s Architecture

The front end of my app (the website) is a nextJS running on Vercel, the database is Supabase, batch (Python jobs) that populate the database are GCP cloud functions, and in one of them I’m using Google Vertex AI (for extractions of the mentions from the episodes metadata)…. hey look, I start to look like a proper enterprise with stuff deployed all over he place!

Observability Overview

Splunk O11y terminology is somewhat confusing, so here is what we will be deploying for each component:

I tried to stick to the default Splunk O11y Open Telemetry (OTEL) packages, but as you will see, that didn’t always work (for my use case).

First things first, get your hands on a 14-day Splunk O11y trial at https://www.splunk.com/en_us/download/o11y-cloud-free-trial.html

Once you log in…. and it’s a blank canvas (see note below), so let’s start painting.

Note: Don’t be alarmed if at the start (before you bring any data), the UI looks very bare and you kind of think to yourself, “where is all the shiny stuff?”. It’s by intention, the approach that Splunk O11y team took: “We will start showing you widgets once we have the data to power them!”.

GCP Infrastructure

In a nutshell, Splunk O11y will pull all the metrics from the GCP Monitoring API. To configure it, start the wizard from the UI by navigating to Data Management > Available Integrations > (search for “gcp”) > Google Cloud Platform.

By following the instructions in the wizard, you will provide information like the authentication method, the GCP project ID, and which data you want to collect, and in exchange, the wizard will tell you which commands you need to run in the GCP console shell or on your laptop (if you have gcloud CLI installed).

Remember that I told you that Splunk O11y will pull ALL the metrics from GCP Monitoring API?! It definitely will! If in the wizard, you are too lazy to pick and choose specific services and just ask for “the lot”, you might end up pulling and PAYING too much.

As you can see above, I did ask for “the lot”, and in a couple of late hours on the first day, Splunk O11y made about 3 times the number of metric calls compared to what it does now on a daily basis.

Anyway, after completing the wizard and manually triggering GCP Run functions (I didn’t want to wait for their next scheduled runs), the dashboards came to life.

As it is part of Splunk Infrastructure Monitoring, you will see all the “infrastructure” metrics, like the number of requests to these functions, CPU and Memory utilisation, etc.. You will not be able to peek “inside” the functions into the Python code to see where the time is being spent (that part we will do later during an APM deployment phase).

Real User Monitoring (RUM)

After having my infrastructure covered by the Splunk O11y Infrastructure Monitoring, I jumped to configure RUM for my front-end.

The way Splunk O11y (or most of the other vendors’) RUM works is by injecting a piece of JavaScript code into the web page so that when a page is loaded, this piece of code collects a bunch of data (like what you clicked, how long did it take for the page to load, etc.) and sends all that valuable information to the analytics platform (Splunk O11y in our case).

To configure RUM in Splunk O11y, you need to obtain a token from: Settings > Access Tokens > Create Token. Make sure to select “RUM token” in the wizard.

In the next step, if needed, you can adjust permission (as to who can view the token value) and finally set the token expiration date (default is 30 days, and the maximum is 18 years).

If the new token doesn’t appear on the Access Tokens page straight away, just refresh the page.

On this page, you can see all the tokens with their expiration date (which conveniently highlights if a token is about to expire

After the token is created, you can start the RUM onboarding wizard by navigating to Data Management > Available Integrations > (search for “rum”) > Browser Instrumentation.

The Wizard will ask you what RUM token to use, the name of your application and the deployed environment. It will then provide you with the deployment steps based on your deployment/architecture (CDN / self-hosted / NPM). NPM was my choice.

Note: You can also deploy the Session Replay functionality, but I’ve skipped it for the moment.

Running the suggested npm install @splunk/otel-web --save will install the required package(s), and will also update your package.json and package-lock.json.

As you can see, the suggested version of splunk-instrumentation.js had hardcoded values (that are either sensitive and/or expected to change from one deployment environment to another)

import SplunkOtelWeb from '@splunk/otel-web';
SplunkOtelWeb.init({
   realm: "au0",
   rumAccessToken: "Super_Secret_Token",
   applicationName: "MentionVault",
   deploymentEnvironment: "DEV"
});

I Codex (after my guidance), improved it by taking out the hardcoded values from the code into environment variables, so now it looks like that

import SplunkOtelWeb from '@splunk/otel-web';

const rumAccessToken = process.env.NEXT_PUBLIC_SPLUNK_RUM_ACCESS_TOKEN;
const deploymentEnvironment = process.env.NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT;

if (typeof window !== 'undefined') {
  if (!rumAccessToken) {
    console.warn('Splunk RUM access token is not set; skipping instrumentation.');
  }
  else {
    SplunkOtelWeb.init({
        realm: 'au0',
        rumAccessToken,
        applicationName: 'MentionVault',
        deploymentEnvironment,
    });
  }
}

To load it, a small component components/splunk-rum.tsx was created

'use client'

import '@/splunk-instrumentation'

export function SplunkRum() {
  return null
}

and it was then added at the top of the app/layout.tsx.

After updating the local environment values, restarting the local Next.js server and browsing the (local) website, the Digital Experience dashboards came to life

You can even see here some JavaScript errors that were happening while I was trying to convert hard-coded values into the env vars.

The sessions are also captured, including the waterfall of what was loaded and clicked on each page.

That’s cool, but wait! How do I deploy Splunk O11y RUM to my NextJS, Vercel-hosted environment(s)? Turns out, pretty easy!

Assuming you already had Vercel configured to build your site from the GitHub repo (and why wouldn’t you?), all that is needed to be done is to add the environment variables to Vercel, and then push your local code to one of the branches in GitHub that is “monitored” by Vercel pipelines.

Note: Make sure to specify different values for the NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT variable in each Vercel environment.

And like that, the Tag Spotlight dashboard started having a bit more colours, and it shows requests from my local environment as well as from the preview and production Vercel-hosted ones.

APM

While RUM provides insights into how real users experience your application, it doesn’t reveal how the (web) server spends its time serving each page request.

APM instrumentation augments either the execution of the code or the code itself.

The first approach is zero-code (A.K.A. automatic) instrumentation, where commonly used libraries (such as requests in Python) are replaced at runtime with instrumented versions. Although no code changes occur when your code calls these libraries, the instrumented versions collect and export telemetry data.

The second approach is code-based instrumentation, where developers use OpenTelemetry (in our case) or vendor-specific, language-specific libraries to instrument their code at key points to generate the required telemetry data.

My preference is to use the first approach, but let’s see how we go.

One more caveat: usually the APM instrumented applications will send their OTEL data to an OTEL collector (for filtering, enrichment, routing, etc.) that in turn will forward the data towards the analytics platform (like Splunk O11y Cloud), but since I am relying on managed services for my application (Vercel and GCP Cloud Run) I didn’t have any infrastructure to deploy the collector, so I am trying to send the data directly to Splunk O11y Cloud APM.

Front-End Instrumentation

Create a new Access token following steps similar to the ones described in the RUM section, but make sure to select INGEST as the token type. Then kick off the APM onboarding wizard by navigating to Data Management > Available Integrations > (search for “apm”) > Node.js (OpenTelemetry) v3.x.

When entering the details in the wizard, instead of the default OTEL collector running locally (on the same host as the instrumented app), I needed to provide the Splunk O11y Cloud endpoint. The endpoint is https://ingest.<realm>.signalfx.com/v2/trace, where realm is the “location” of your Splunk O11y deployment that you can get from the URL in the browser.

Side note: I guess signalfx is hard-coded somewhere very deep, if Splunk can’t change the URLs to (or add new) Splunk-branded ones 6 years after the acquisition of SignalFx.

In the next step, the wizard will suggest a set of steps to complete to instrument your app.

And here the “Fun” begins…

The first 2 are easy; you simply install the package and add some environment variables for the Splunk OTEL to pick up its configuration.

The 3rd one, however, stumbled me a little bit. Since I am not running a “pure” node application but a Next.js one, I didn’t know what I needed to run (instead of node -r @splunk/otel/instrument <your-app.js>) to start the local Next.js server with Splunk OTEL instrumentation. After a bit of Googling/ChatGPT-ing, I landed on updating my package.json start dev script (note the --require and not -r as well as escaping quotes).

 ...
 "scripts": {
    "build": "next build",
    "dev": "NODE_OPTIONS=\"--require @splunk/otel/instrument\" next dev",
 ...

Restarted the server, browsed my site locally, and…. nothing happens :-(.

Following the suggestion in Splunk docs, I enabled OTEL debugging by adding an OTEL_LOG_LEVEL variable to my start script (actually, I created a new dev-debug one)

...
 "scripts": {
    "build": "next build",
    "dev": "NODE_OPTIONS=\"--require @splunk/otel/instrument\" next dev",
    "dev-debug": "OTEL_LOG_LEVEL=debug NODE_OPTIONS=\"--require @splunk/otel/instrument\" next dev",
...

And of course , I realised that I forgot to add the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN to the environment variables.

Note: I probably missed something else, but if I was using an .env.local file to store the OTEL-related environment variables, they were not picked up (while other ones, like Supabase configuration, were), so I needed to pass the values either via the start script in package.json or via the OS (export SPLUNK_REALM=...).

Restarted Next.JS local server, browsed, and … oh Joy! APM dashboard came to life, I could see Traces, like the one below.

Since I already had Splunk O11y RUM configured, I could also drill down (or is it actually pan out?) to the RUM session that triggered this trace:

Now, after I validated that it is capturing traces, I decided to try and deploy it to Vercel, and here the REAL “Fun” begins…

I made sure to set all the necessary env vars in Vercel, but the deployment was failing. The deployment logs were showing this error:

23:13:45.367 node:internal/modules/cjs/loader:1215
23:13:45.368 throw err;
23:13:45.368 ^
23:13:45.368
23:13:45.368 Error: Cannot find module '@splunk/otel/instrument'
23:13:45.368 Require stack:
23:13:45.368 - internal/preload

But why? But how? @splunk/otel is declared in the package.json, so this module should be installed and available, shouldn’t it?

Turn out (according to ChatGPT):

“What’s happening – Vercel sets your NODE_OPTIONS for every Node process it spins up, including the ones it runs before npm install. At that point, node_modules doesn’t exist yet, so –require @splunk/otel/instrument throws MODULE_NOT_FOUND and the build aborts.

How to fix it – Don’t point NODE_OPTIONS directly at the package on Vercel. Instead …”

The “instead” part required a bit of trial and error, but eventually landed on the need to create instumentation.ts

export async function register() {
  if (process.env.NEXT_RUNTIME !== 'nodejs') {
    return;
  }

  try {
    const { start } = (eval('require') as NodeJS.Require)(
      '@splunk/otel',
    ) as typeof import('@splunk/otel');

    const logLevel =
      process.env.NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT === 'production'
        ? 'info'
        : 'debug';

    start({
      logLevel: logLevel,
    });
  } catch (error) {
    const err = error as NodeJS.ErrnoException;
    if (err?.code === 'MODULE_NOT_FOUND') {
      console.warn('Splunk OTel instrumentation not available yet, skipping preload.');
      return;
    }
    throw error;
  }
}

The deployment worked, but the instrumentation didn’t work. Setting the OTEL_LOG_LEVEL=debug in Vercel also didn’t enhance the Vercel Run logs by a bit.

Interestingly, somewhere along the way, my Traces from my local deployment also started showing calls to the local Supabase instance.

Without access to debug the deployment, I had to give up rethink my approach: what is Vercel’s recommended way of using OTEL?

While Vercel has prebuilt integrations for some APM vendors, Splunk O11y Cloud is not one of them. But fear not! There is a way forward; we can use Custom OTEL Exporters.

So, install Vercel’s OTEL wrapper npm i -E @vercel/otel@1.13.1 .

Note: Make sure to pin the @vercel/otelpackage to the latest 1.x version, as v2 has some dependency conflicts with @splunk/otel-web.

And now create/update instrumentation.ts

import { registerOTel, OTLPHttpProtoTraceExporter } from '@vercel/otel';

export function register() {
  registerOTel({
    serviceName: 'MentionVault',
    traceExporter: new OTLPHttpProtoTraceExporter({
      // Splunk O11y OTLP traces endpoint
      url: `https://ingest.${process.env.SPLUNK_REALM}.signalfx.com/v2/trace/otlp`,
      headers: {
        'X-SF-Token': process.env.SPLUNK_ACCESS_TOKEN!, // ingest token
      },
    }),
    attributes: {
      'deployment.environment': process.env.NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT ?? 'local',
    },
  });
}

Note: We are using OTLPHttpProtoTraceExporter and not the OTLPHttpJsonTraceExporter (as it appears in the example in Vercel docs) since Splunk O11y Cloud expects the OTLP data in the protobuf (and not JSON) format. After redeploying that to Vercel and browsing the hosted website, traces started streaming to the Splunk O11y deployment, with one caveat – the link between APM and RUM is gone . I’ll need to spend some time to see if we can bring it back, but that is another item to add to the TODO list.

GCP Cloud Run (Python) Functions Instrumentation

Details to be updated soon….

From a first glance, simply following the wizard works locally

But the fun part will probably be making sure it works in GCP deployment as well….

TO BE CONTINUED….

The post Splunk O11y Deployment appeared first on ISbyR.

The Issue

You can configure any of the pre-added models in the Splunk UI by going to the MLTK App and then hitting the “Connection Manager” tab.

When you select a service, you can see a list of pre-defined models. These are already somewhat outdated, for example, for Gemin, you don’t have any of the 2.5 models.

So, “how do we add new LLM models to Splunk MLTK?” you might ask.

The Solution

Easy-ish…

A bit of background

This configuration is managed in a Splunk KV Store collection (named mltk_ai_commander_collection), and in essence, it’s a big JSON that has all the providers and the models.

For example, here is the snippet for the Gemini Service and the first of its models

        "Gemini": {
            "Endpoint": {
                "value": "https://generativelanguage.googleapis.com/v1beta/models",
                "type": "string",
                "required": false,
                "description": "The API endpoint for sending chat completion requests to Google's Gemini language model."
            },
            "Access Token": {
                "value": "",
                "type": "string",
                "required": true,
                "hidden": true,
                "description": "The authentication token required to access the Gemini API."
            },
            "Request Timeout": {
                "value": 200,
                "type": "int",
                "required": false,
                "description": "The maximum duration (in seconds) before a request to the Gemini API times out."
            },
            "is_saved": {
                "value": true,
                "type": "boolean",
                "required": false,
                "description": "Is Provider details stored"
            },
            "models": {
                "gemini-pro": {
                    "Response Variability": {
                        "value": 0,
                        "type": "int",
                        "required": true,
                        "description": "Adjusts the response's randomness, impacting how varied or deterministic responses are."
                    },
                    "Maximum Result Rows": {
                        "value": 10,
                        "type": "int",
                        "required": false,
                        "description": "The maximum number of result entries to retrieve in a response."
                    },
                    "Max Tokens": {
                        "value": 2000,
                        "type": "int",
                        "required": false,
                        "description": "The limit on the number of tokens that can be generated in a response."
                    },
                    "Set as default": {
                        "value": false,
                        "type": "boolean",
                        "required": false
                    }
                },

So if we want to add a new model, all we need to do is add another element to the models array.

Now, here’s a big disclaimer: There might be a better/easier way to do that. This is probably not supported. Some models might not necessarily work with this approach, etc….

While there is a Loolup Editor app, it will only help you (to edit KV store collections) if there is a lookup configured for it. Which is not the case for the mltk_ai_commander_collection one.

High-level steps

Another way (and the one we will take) is to use Splunk REST API, and at a high level, it consists of the following steps:

1. Get the current configuration (and the _key of the collection item) in a JSON format
2. Update in a text editor the JSON payload
3. Update the KV collection with the new JSON

Detailed steps

I will provide examples using Postman, but you can use curl or any other method of your choice for interacting with the REST API.

Get the current configuration

Run a GET call to the collection/data endpoint

The actual URL is https://localhost:8089/servicesNS/nobody/Splunk_ML_Toolkit/storage/collections/data/mltk_ai_commander_collection

Copy the results and take a note of the _key at the end of the JSON.

Update the JSON

Paste the JSON in a text editor of your choice.

Go to the Provider for which you want to add a new Model (Gemini) in our case,

Duplicate the model object inside the Service object and change the model name.

For example, here I copied/pasted the gemini-2.0-flash to the end of the Gemini service object and renamed it to be gemini-2.0-flash.

NOTE: You must ensure that the model name you provide here is exactly the same as it would appear when calling the inference API for the LLM Service.

For example, for Gemini

Update the KV collection

Now we need to update the collection with the updated JSON payload.

Send a POST request to the collection/data endpoint

• replace the _key part of the URL with the value that you have in your JSON
• remove the square brackets ([]) that surround the JSON

The actual URL is something like that: https://localhost:8089/servicesNS/nobody/Splunk_ML_Toolkit/storage/collections/data/mltk_ai_commander_collection/68540d2d0d2a214efd0d3b61.

Now, refresh the Connection Management page and enjoy a fresh new model at your disposal

Simply use the new model in the | ai command

And here is a sneak peek into an LLM Telemetry dashboard I’m working on

I hope that helped you to understand how to add new LLM models to Splunk MLTK.

The post Add new LLM models to Splunk MLTK appeared first on ISbyR.

I didn’t have OpenAI subscription, but I do have an Azure subscription and Azure OpenAI deployment to play with, so I replaced the “standard” OpenAI node with the Azure OpenAI one.

But when I started the execution, the Azure OpenAI Chat Model node threw an exception, straight in my face: “The response was filtered due to the prompt triggering Azure OpenAI’s content management policy.”.

The Problem

The summary of the error was not too informative, to be honest.

But, if you expand error details, you can see where the actual problem is:

The thing with Azure OpenAI (or other AI models served by the Azure AI Foundry, for that matter) is that all the requests are going through Azure Guardrails, like Content Filters and Blocklists. And the default content filter decided that the prompt that the n8n Agent node was trying to run was too “fishy”. Look, TBH, I can’t blame it for that, as when you peek under the hood (of the prompt that is sent to the LLM), you can see it is “screaming” at it with commands like ----- IGNORE BELOW -----, which can easily be perceived as a jailbreak attempt.

The Solution

So, what do you do if something default doesn’t work?! You customise it! And Azure AI content filters are not an exception, and are very easy to customise:

• Go to Azure AI Foundry and make sure that you are in the right project of course.
• Click the Guardrails + Controls on the left side panel.
• Select the Content filters tab.

• Click the Create content filter button to start the custom content filter wizard.
  • Provide a name for your content filter on the Basic information page.
  • The Input filter page is the one where we need to make the changes. Find the Prompt shields for jalbreak attacks category and set the action to either Annotate only or Off. (Selecting Annotate only runs the respective model and returns annotations via API response, but it will not filter content).

• Next, Next to get to the Connection step.
• Here, you will select the deployment that you want to apply this content filter to.

• Hit Next and then Replace in the Replace existing content filter dialogue box.

And that’s it. Next time I executed this step in n8n, it ran successfully.

NOTE: Of course, guardrails in general, and content filters specifically, exist for a very good reason. So you should be very careful when tweaking them or turning them off. You should always consider who will have access to this inference endpoint and what data is accessible to it.

But, since I was playing with it in my personal environment, I didn’t mind making these tweaks to the content filter.

More posts related to my AI journey:

The post n8n – The response was filtered due to the prompt triggering Azure OpenAI’s content management policy appeared first on ISbyR.

If you try to follow the “Create a Custom Skill for Azure AI Search” lab that is part of the “Implement knowledge mining with Azure AI Search” course it fails (at least until Microsoft updates the lab files as per my PR).

There are 2 issues in the update-skillset.json file that is part of this lab

Issue and Error #1

The request is invalid. Details: The property ‘includeTypelessEntities’ does not exist on type ‘Microsoft.Skills.Text.V3.EntityRecognitionSkill’. Make sure to only use property names that are defined by the type.

To fix it remove the line that contains includeTypelessEntities in the JSON file mentioned above.

This is due to a depreciation of this parameter by Microsoft

Issue and Error #2

If you try to run the update-skillset script again, after fixing the first error, you will be greeted by error #2:

“One or more skills are invalid. Details: Error in skill ‘#1’: Outputs are not supported by skill: entities”

To fix it, a few lines below the line that you’ve just removed under the outputs section, replace

"name": "entities"

with

"name": "namedEntities"

Since that is the available output name of this API.

That’s it folks, enjoy

More posts related to my AI journey:

The post “Create a Custom Skill for Azure AI Search” lab fails appeared first on ISbyR.

A bit of the terminology,

I will not dive too deep here, but just enough to get started. The definitions below are my simplified understanding, and they are most likely not fully correct.

What is RAG

There are many places where you can learn about RAG, but for the context of this post, I’d say that RAG allows you to supplement the initial prompt for the LLM with a bit more (or a lot more, that’s up to you) context.

What is a Vector Database?

Vector Database is one of the mechanisms/data stores that will enable you to provide this additional context to the LLM. Unline “regular” databases, vector database doesn’t necessarily store the actual data (though it can), but it will store the embedding of the data you later wish to search to retrieve the above-mentioned context.

What are embeddings?

Embeddings are multi-dimensional numerical representations of a piece of data (text for example), The multi-dimensionality allows to “place” semantically similar terms close to each other. For example, if using semantic search we search for “dog” then “puppy” and “mutt” and mutt will be considered close terms, while if using a lexical search (one that looks at the literal text similarity), will probably consider “dogma” and “hot dog” as closer terms.

The ITSM Assistant

The problem

Now let’s say you want to open a ticket in your ITSM ticketing system that your Internet Access is not working properly. You could start by searching for a particular request type or a knowledge base article, but what if you are not a technically savvy person and all you care about is that you can’t get to Facebook?

The Solution

ITSM Assistant to the rescue!!! It’s a chat interface that will:

1. ask a user about the issue they are currently facing
2. will look in the vector database for semantically similar historical requests, and get their IDs
3. get the content of the tickets from the data store (simple CSV file int in this case)
4. will feed this context to an LLM
5. provide a user with the suggested request form and some of the fields that should be populated

As you can see in the screenshot below the user didn’t mention that they have problems with “internet access”, but just said, “I can’t get to Facebook”. Despite that ITSM Assistant was able to pull data that is related semantically to the user’s issue. LLM (after being fed all the context) suggested the correct Service Request type and some of the information the user should add to the ticket for it to be promptly resolved.

How it works under the hood?

The components

• Pinecone – vector database
• Streamlit – “…turns data scripts into shareable web apps in minutes.”, both front and back-end all in Python.
• Stramlit Community Cloud – for hosting the Stremalit app
• AzureOpenAI – the LLM
• all-MiniLM-L6-v2 – “…a sentence-transformers model: It maps sentences & paragraphs to a 384-dimensional dense vector space”

Step 0 – Load the data into Vector Database

I found an ITSM ticket dump on the internet.

Next, we need to get embedding for each ticket and insert it into the vector database (Pinecone in my case).

I had a Jupiter notebook that was doing this job.

# Importing the necessary libraries
import pandas as pd

# Importing the csv file
data = pd.read_csv('GMSCRFDump.csv', encoding = 'ISO-8859-1')

# removing duplicate tickets
ID_mins = data.groupby(['Title', 'Description', "CallClosure Description"]).ID.transform("min")
data_n = data.loc[data.ID == ID_mins]

# create a new array with a field that has both title and description each ticket
title_description = data_n['Title'] + " __ " + data_n['Description']
# create an arraid of ticket IDs
tid = data_n['ID']

# import a transformer that will be used to encode the ticket data
from sentence_transformers import SentenceTransformer
import torch

# Define the model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer('all-MiniLM-L6-v2', device=device)

# Setup Pinecone connection
import os
os.environ['PINECONE_API_KEY'] = '1b5da094-f784-4beb-8fc3-262712a667ae'
os.environ['PINECONE_ENVIRONMENT'] = 'gcp-starter'

from pinecone import Pinecone, PodSpec

# get api key from app.pinecone.io
api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'
# find your environment next to the api key in pinecone console
env = os.environ.get('PINECONE_ENVIRONMENT') or 'PINECONE_ENVIRONMENT'
pinecone = Pinecone(api_key=api_key)

# Create index

index_name = 'snow-data'
# only create index if it doesn't exist
if index_name not in pinecone.list_indexes().names():
    pinecone.create_index(
        name=index_name,
        dimension=model.get_sentence_embedding_dimension(),
        metric='cosine',
        spec=PodSpec(
            environment=env, 
            pod_type='s1.x1'
        )
    )

# now connect to the index
index = pinecone.Index(index_name)

# the following section, takes a batch of tickets, for each one of them makes and embeeding, "attaches" id and title+description as meta data , and upserts that into Pinecone index

from tqdm.auto import tqdm

batch_size = 120
vector_limit = 12000

title_description = title_description[:vector_limit]
title_description

for i in tqdm(range(0, len(title_description), batch_size)):
    # find end of batch
    i_end = min(i+batch_size, len(title_description))
    # create IDs batch
    ids = [str(x) for x in range(i, i_end)]
    # create metadata batch
    metadatas = [{'tid': t_id, 'text': t_desc} for t_id, t_desc in list(zip(tid,title_description))[i:i_end]]
    print(metadatas)
    # create embeddings
    xc = model.encode([t_desc for t_desc in title_description[i:i_end]])
    # create records list for upsert
    records = zip(ids, xc, metadatas)
    # upsert to Pinecone
    index.upsert(vectors=records)

Step 1 – The Streamlit App

Streamlist is a straightforward Python framework that allows you to build (simple) web apps. All without any HTML, JavaScript and CSS knowledge. You can run it locally or host it somewhere, for example using their Community Cloud.

You can find the code for the app in the ITSM Assistant repo here. So I’ll not provide much code from now on, but instead, talk to write about my any caveats.

To try it at home one will need to create secrets.toml file under the .streamlit folder and populate it with your Azure OpenAI and Pinecone credentials/configuration

AZURE_OPENAI_API_KEY = "xxxxxxxxxxxxx"
AZURE_OPENAI_ENDPOINT = "https://xxxxxxxxx.openai.azure.com/"

PINECONE_API_KEY = "xxx-xxx-xxx-xx"
PINECONE_INDEX = "snow-data"

Steps 2 and 3 – Searching for Historical Tickets

One caveat is depending on the amount of data one can decide to upsert into the vector db (in addition to the embeddings themselves) not only ticket ID (as metadata), but all the ticket fields (like Description, Resolution, etc.). This way your semantic search can return all the data you need and there is no need to have Step 4 (retrieval of data from data store)

For the sake of learning, I did not, so after we get ticket IDs from Pinecone, we use it to filter the data in the data store (fancy name for CSV) to get the ticket information that needs to be sent as context to the LLM.

Step 4 – Ask LLM for help

Now that we have the context (similar ticket data) we can send the request to LLM to help our struggling user and point them in the right direction.

Step 5 – Response to User

The only thing worth mentioning here is I had a bit of a struggle in printing the list of fields nicely.

LLM is coming back with a JSON response similar to below:

{
  "common_theme": "Server Reboot",
  "title": "Server Reboot Request",
  "suggested_fields": "SSO ID, Name, Email, Seat Number, Location, Cell Phone No"
}

Streamlit can use markdown for output, so to format the list of fields nicely I had to do something like this:

suggested_fields = llm_response['suggested_fields'].split(', ')
suggested_fields = "- " + "\n- ".join(suggested_fields)
 
nl = "  \n"
st.chat_message("ai").markdown(f"It looks that in order to help you, you will need to raise a new **\"{llm_response['title']}\"**.{nl}\
When raising this request please provide some of the required information like:{nl}{suggested_fields}")

P.S.

You can find the app here: https://app-itsm-assistant.streamlit.app/

More posts related to my AI journey:

The post Learning about RAG and Vector Databases appeared first on ISbyR.

Please see the notes inside as the code comments

# Import os to handle environment variables
import os
# Import sreamlit for the UI
import streamlit as st
# Import Azure OpenAI and LangChain
from langchain_openai import AzureChatOpenAI
from langchain_core.messages import HumanMessage
from langchain.callbacks import get_openai_callback


st.title(" ITSM Assistant App")

with st.sidebar:
    os.environ["AZURE_OPENAI_ENDPOINT"] = "https://aoai-itsm.openai.azure.com/"
    # get the Azure OpenAI API key from the input on the left sidebar
    openai_api_key = st.text_input("OpenAI API Key", type="password") 
    os.environ["AZURE_OPENAI_API_KEY"] = openai_api_key
    "[Get an Azure OpenAI API key from 'Keys and Endpoint' in Azure Portal](https://portal.azure.com/#blade/Microsoft_Azure_ProjectOxford/CognitiveServicesHub/OpenAI)"

def generate_response(input_text):

    model = AzureChatOpenAI(
        openai_api_version="2024-02-15-preview",
        azure_deployment="gpt35t-itsm",
    )
    message = HumanMessage(
        content=input_text
    )
    
    with get_openai_callback() as cb:
        st.info(model([message]).content) # chat model output
        st.info(cb) # callback output (like cost)

with st.form("my_form"):
    text = st.text_area("Enter text:", "What are 3 key advice for learning how to code?")
    submitted = st.form_submit_button("Submit")
    if not openai_api_key:
        st.info("Please add your OpenAI API key to continue.")
    elif submitted:
        generate_response(text)

Now. a few notes:

• Model initiation will need
  • AZURE_OPENAI_ENDPOINT – get if from Azure Portal > Azure OpenAI. Select your service > Keys and Endpoint
  • azure_deployment – Get it from the Azure OpenAI Portal > Deployments (the value under the Deployment Name column)
  • openai_api_version – the easiest way I found is to go to the Azure OpenAI Portal > Playground > Chat > View Code (in the middle top)

More posts related to my AI journey:

The post Streamlit Langchain Quickstart App with Azure OpenAI appeared first on ISbyR.

You know that annoying ... at the end of a field?!

So there is a magic display.max_colwidth option (and many other wonderful options).

From the official docs:

display.max_colwidth : int or None
    The maximum width in characters of a column in the repr of
    a pandas data structure. When the column overflows, a "..."
    placeholder is embedded in the output. A 'None' value means unlimited.
    [default: 50] [currently: 50]

Here are a couple of examples on how to use it:

`

More posts related to my AI journey:

The post Stop pandas truncating output width … appeared first on ISbyR.