LLM Archives - ISbyR https://isbyr.com/tag/llm/ Infrequent Smarts by Reshetnikov Sun, 05 Jul 2026 13:40:25 +0000 en-US hourly 1 https://wordpress.org/?v=7.1.4 The ABCs of Voice AI Agents https://isbyr.com/the-abcs-of-voice-ai-agents/ https://isbyr.com/the-abcs-of-voice-ai-agents/#comments Sun, 05 Jul 2026 13:19:39 +0000 https://isbyr.com/?p=1310 I have been building a voice AI receptionist, and there are a lot of terms and acronyms. Some of them are normal telephony acronyms that have been around forever, like DTMF and PSTN (and might need a refresher). Some are AI acronyms, like STT, TTS, LLM and RAG (that we are now sick of hearing … Continue reading The ABCs of Voice AI Agents →

The post The ABCs of Voice AI Agents appeared first on ISbyR.

]]>
I have been building a voice AI receptionist, and there are a lot of terms and acronyms. Some of them are normal telephony acronyms that have been around forever, like DTMF and PSTN (and might need a refresher). Some are AI acronyms, like STT, TTS, LLM and RAG (that we are now sick of hearing every day). Then there are the “glue” terms that only start to matter once you actually try to connect Telephony, Voice and LLMs.
This is my attempt to write down the useful ones, A.K.A The ABCs of Voice AI Agents.

It is not a full architecture doc. It is more of a map of the words that kept coming up while building the bot.

Call Flow

To give a bit of context, below is the high-level flow that depicts the moving parts of a person calling an AI Voice Agent:

  1. Inbound call — the caller dials a Twilio number over the PSTN. Twilio answers and holds the call.
  2. Webhook — Twilio sends an HTTP POST to the FastAPI /voice endpoint to ask what to do with the call.
  3. TwiML response — FastAPI replies with TwiML containing a <Connect><Stream> instruction, telling Twilio to open a media stream to the app.
  4. WebSocket + audio — Twilio opens a WebSocket to /ws and starts streaming the caller’s audio as mu-law frames. Pipecat takes ownership of the connection.
  5. Audio → STT — Pipecat passes the raw audio to the STT service (e.g. Cartesia Ink). VAD detects end-of-turn before the audio is sent.
  6. Transcript → LLM — STT returns a text transcript. Pipecat appends it to the conversation and sends the full context to the LLM (e.g. Gemini).
  7. Response text → TTS — the LLM streams its response back to Pipecat, which forwards the text to the TTS service (e.g. Cartesia Sonic).
  8. Audio → Pipecat — TTS converts the text to speech and returns audio frames to Pipecat.
  9. Audio stream → Twilio — Pipecat sends the synthesised audio back over the WebSocket as mu-law media messages.
  10. Audio → caller — Twilio plays the audio to the caller over the PSTN. The caller hears the agent’s voice and the loop repeats for their next utterance.

So, without further ado, I present to you…

The ABCs of AI Voice Agents

AgentCore

AWS Bedrock AgentCore is a fully managed, enterprise-grade platform that helps developers build, deploy, and scale AI agents, and it was considered as a possible runtime.

The problem, however, was that Twilio couldn’t (or I couldn’t make it) connect to it directly. Twilio Media Streams has a specific WebSocket handshake and message format. AgentCore has its own auth and protocol expectations.

Relevant docs: AWS AgentCore

AI – Artificial Intelligence

For a Voice Agent, AI really means a few different things: speech recognition, text generation, text-to-speech, routing logic, and more. The LLM is only one part of the system.

ASR / STT – Automatic Speech Recognition and Speech-to-Text

These are often used interchangeably. It is the part that turns the caller’s audio into text that the LLM can understand.

In this project, the pipeline uses Cartesia STT (Ink model). Twilio ConversationRelay can also do ASR for you if you use that product instead of raw Media Streams. Other common vendors are Deepgram, Google Cloud STT, AssemblyAI, and OpenAI Whisper-style models.

The practical thing to remember: STT quality is not only about model accuracy. End-of-turn timing, background noise, phone audio, accents, and whether the user interrupts all matter.

Barge-in

Barge-in is when the caller interrupts the agent while the agent is speaking.

This is normal human behaviour. The system has to decide whether to stop speaking, listen, or update the conversation state.

CallSid / StreamSid

These are Twilio identifiers.

CallSid identifies the call. StreamSid identifies the media stream.

Relevant docs: Twilio CallSid and StreamSid

Cartesia

Cartesia is the speech and voice agent platform in this project.

In this project, Cartesia provides the speech components: STT for converting caller audio into text and TTS for converting the model’s response back into audio.

Cartesia can also handle more of the voice application layer: call orchestration, interruptions, deployment, logs, and call analytics (see Cartesia Line).

Cartesia Line

Cartesia Line is a code-first voice agent platform.

Cartesia Line can handle voice/call orchestration, Ink STT, Sonic TTS, interruptions, deployment, logs, and call analytics.

Line also has useful built-in concepts like transfer_call, send_dtmf, and handoffs (if you manage your telephony through them).

Note: If you are using your own Twilio number (which you can easily import with/connect to the Cartesia platform) and stream path, you still need access to the live Twilio REST API to manage the call controls.

Relevant docs: Cartesia Line

Conference

A conference is a Twilio call room with participants.

For warm transfer, the caller can be parked in a conference with hold music or a wait message while the human leg is being dialled. When the human agent accepts the transfer, they join the same conference.

The details matter:  startConferenceOnEnter,  endConferenceOnExit,  waitUrl, status callbacks, and what happens if a participant leaves.

Relevant docs: TwiML <Dial><Conference>.

ConversationRelay

Twilio ConversationRelay is a higher-level Twilio voice AI product (compared to Media Streams).

Instead of sending raw phone audio to your app, Twilio can handle STT and TTS and send structured WebSocket (text) messages to your application. Your app receives prompts/transcripts and sends text tokens back.

That can simplify the audio side. The tradeoff is that you are using Twilio’s ConversationRelay workflow rather than building your own Pipecat audio pipeline.

Relevant docs: Twilio ConversationRelay

DTMF – Dual-Tone Multi-Frequency.

This is the phone keypad signal: 0-9, star, and hash.

In this project, DTMF matters for warm transfer. The human agent answers, hears a brief, and presses 1 to accept the transfer. That is much safer than just dumping the caller onto whatever number was dialled.

Twilio <Gather> can collect DTMF digits.
Cartesia Line has a send_dtmf tool for pressing buttons when the voice agent itself needs to navigate a phone menu.

Relevant docs: Twilio <Gather>  and Cartesia Line  send_dtmf 

E.164

E.164 is the international phone number format, like +614….

It matters because APIs expect phone numbers to be normalised. Transfer tools should store and compare numbers in a predictable format.

HMAC

Hash-based Message Authentication Code is a cryptographic method used to simultaneously verify a message’s integrity and authenticity.

HMAC is used in the current app for the stream token. The /voice webhook validates Twilio’s signature, creates a call state, then mints a short-lived token tied to the CallSid. Twilio echoes that token in the WebSocket start.customParameters, and /ws validates it.

Why not just rely on Twilio’s webhook signature? Because the WebSocket upgrade does not carry the same signed form payload as the /voice POST.

IVR – Interactive Voice Response.

This is the classic “press 1 for…” phone menu. A voice AI receptionist can replace some IVR flows.

In our case, the human agent acceptance step is basically a tiny IVR: “press 1 to accept”.

Latency

Latency is a delay.

In a voice bot, latency is made up of many small delays:

  • Twilio media to app.
  • STT.
  • VAD/end-of-turn.
  • LLM first token.
  • Tool calls, like Notion lookup.
  • TTS first audio.
  • Audio buffering back to Twilio.

The Notion (or other data store) lookup work is a good example. A live scan across Notion databases during the greeting made the call feel slow. The better approach was a quick caller lookup first, then background refresh/enrichment, with caching for slower data.

LLM – Large Language Model.

This is the reasoning layer. It reads the conversation, decides what to say, and calls tools when needed. In this project, the model has been Gemini.

The model should not be allowed to invent call control. It needs tools with clear rules: identify the caller, find the routing recipient, create a message, list assignments, cancel an assignment, and transfer a call.

Mark / Clear

mark is how the app asks Twilio to tell it when previously sent audio has finished playing. clear clears buffered audio.

These are Twilio Media Streams messages.

For transfer, mark was the safer way to avoid cutting off the “please hold” sentence when a caller is being transferred. Without it, the app can redirect the call before the caller hears “I will transfer you now”.

Relevant docs: Twilio mark and clear messages

Media Streams

Twilio Media Streams lets you stream call audio to your WebSocket server.

There are unidirectional streams and bidirectional streams. For the voice bot, bidirectional is the useful one because the app needs to receive the caller audio and send the agent audio back.

Twilio sends JSON messages like connected, start, media, mark, and stop. The audio payload is base64. When you send audio back to Twilio, you send media messages. When you want to know that buffered audio has finished playing, you send a mark and wait for Twilio to echo it.

Relevant docs: Twilio Media Streams

Notion

Notion is used as the operational data source in this project.

The bot looks up callers, contractors, clients, assignments, HR directory entries, routing recipients, and message logs from Notion.

Observability

Observability is how you understand what happened during a call.

For voice AI, logs alone are not enough. You want spans and metrics around:

  • inbound webhook.
  • WebSocket start/stop.
  • STT latency.
  • VAD/end-of-turn timing.
  • LLM latency and tool calls.
  • TTS latency.
  • transfer lifecycle.
  • call cleanup.

Relevant docs: Pipecat metrics

OAuth / JWT

OAuth and JWT are common auth patterns.

They matter in voice AI because every real-time endpoint is exposed to the internet. A public wss:// endpoint that accepts audio should not accept random traffic just because it looks like Twilio.

In this project, Twilio webhook signatures plus HMAC stream tokens were enough for the current Media Streams path. If using another runtime, hosted agent, or browser client, OAuth/JWT-style auth might be more relevant.

PCMU / mu-law / 8 kHz

PCMU, also called G.711 mu-law, is a common phone audio codec.

Twilio Media Streams sends and receives base64-encoded audio/x-mulaw at 8000 Hz for media frames. If you send the wrong audio format back to Twilio, the caller does not get useful audio.

This was one reason a direct “Twilio to some random WebSocket agent runtime” idea was not enough. The server must speak Twilio’s media message protocol and handle the expected audio format.

Pipecat

Pipecat is a framework for building real-time voice and multimodal agents.

In this build it provides the pipeline:

transport input -> STT -> user aggregator / VAD -> LLM -> TTS -> transport output -> assistant aggregator

The useful thing about Pipecat is that it gives names and structure to the pieces. The dangerous thing is assuming the transport protocol is magic. Twilio Media Streams still has its own JSON messages, audio format, CallSid, StreamSid, marks, and stop events.

Relevant docs: Pipecat

PII

Personally Identifiable Information.

Voice AI systems touch a lot of it: names, phone numbers, call recordings, transcripts, assignments, clients, and messages.

In this receptionist project, the agent should not volunteer surnames, phone numbers, or private details. It can use retrieved data to route the call, but it should speak only the minimum needed. For example, first names are usually enough for a transfer offer.

This is not just a prompt issue. Logs, traces, tool results, and evaluation datasets need the same discipline.

PSTN

Public Switched Telephone Network.

This is the normal phone network. When a caller dials a Twilio number from a mobile or landline, they are coming from the PSTN.

The annoying bit is that the PSTN is narrowband and phone-oriented. You are usually dealing with 8 kHz phone audio somewhere in the path, not clean studio audio.

Relevant vendors: Twilio, Telnyx, Vonage

RAG

Retrieval-Augmented Generation.

RAG means retrieving relevant knowledge and giving it to the model before it answers. In this project, Notion lookup is not exactly a generic vector RAG flow, but the idea is similar: fetch caller, contractor, client, assignment, HR directory, and routing context so the model can make the right decision.

The important bit is privacy. Retrieved data should be scoped. The agent should not volunteer surnames, phone numbers, or private records. If a contractor calls, they can get information relevant to their own assignments, not everyone else’s.

RTP

Real-time Transport Protocol.

RTP is how media packets often move in VoIP systems. You may not touch it directly if you are using Twilio Media Streams or ConversationRelay, but it is underneath a lot of telephony.

When people talk about codecs, jitter, packet loss, and media paths, RTP is usually close by.

SigV4

AWS Signature Version 4.

This came up when looking at AWS AgentCore. Direct Twilio-to-AgentCore looked tempting, but it did not fit.

The problems were:

  • Twilio <Stream url> cannot practically sign the WebSocket request with AWS SigV4 headers.
  • Twilio <Parameter> values arrive inside the later start.customParameters message, not in the initial WebSocket handshake where AgentCore auth would need them.
  • Even if auth worked, the protocols did not match. Twilio sends Twilio Media Streams JSON with phone audio. Agent runtimes usually expect their own WebSocket protocol.

The conclusion was simple: you need a bridge or a single service that speaks Twilio on one side and your agent runtime on the other.

Relevant vendors/frameworks: AWS AgentCore, Twilio Media Streams, custom relay service.

SIP

Session Initiation Protocol.

SIP is a signalling protocol used to set up and control voice calls. If you are connecting phone systems, PBXs, trunks, and carriers, SIP eventually appears.

For this receptionist project, I mostly stayed at the Twilio Voice API/TwiML layer rather than managing SIP directly. That was enough for inbound calls, Media Streams, call redirects, and conferences.

State store

State is what the app remembers about the active call.

For a single container, in-memory state can work during local testing. In production, Redis or another shared store becomes important if there are multiple instances or restarts.

In this project state includes the caller, public base URL, allowed transfer numbers, active transfer state, stream token, and whether the call has already been redirected or ended.

Stop event

Twilio sends a stop event when the stream stops or the call ends.

This matters for cleanup. If the caller hangs up and the app does not cancel the pipeline, the model/STT/TTS work can keep running for no reason. Hangups need to cancel the active pipeline, clear state, and avoid trying to speak into a dead call.

Telnyx

Telnyx is another telephony provider.

We looked at Telnyx as an alternative to Twilio. The conclusion was similar for direct-to-AgentCore: Telnyx can stream audio and gives strong call control APIs, but stream_url and stream auth are still about connecting Telnyx to your server. They do not magically make a third-party agent runtime speak Telnyx’s protocol.

So Telnyx can be a good provider, but you still need a relay/bridge unless the voice agent framework directly supports its stream format.

Relevant docs: Telnyx Call Control

Tool calling

Tool calling is when the LLM asks the app to perform an action.

For this receptionist, tools are not optional decoration. They are how the agent does real work:

  • identify the caller by phone number.
  • find a routing recipient.
  • list assignments.
  • cancel an assignment.
  • create a message.
  • start a warm transfer.

The rules around tools matter. For example, transfer_call should only work for previously selected, call-capable recipients. The model should not be allowed to transfer to a random number it heard in the conversation.

Relevant frameworks: Gemini tool declarations, Pipecat tool handlers, Cartesia Line tools, OpenAI/Anthropic tool calling.

TTFB / TTFA

Time To First Byte and Time To First Audio.

For voice, TTFA is often the more useful mental model. The caller does not care when the server got a byte. They care when they hear something useful.

This is why streaming matters. Stream the LLM response if possible. Start TTS early if possible. Do not block the first greeting on slow CRM/database enrichment unless that data is needed to say hello.

TTS

Text-to-Speech.

This is the part that turns the agent’s text back into audio. In the current Pipecat setup, Cartesia TTS does this. Cartesia calls its TTS model family Sonic.

Twilio ConversationRelay can use TTS providers like Google, Amazon, Deepgram, and ElevenLabs depending on configuration and access.

The practical thing to remember: TTS is where latency becomes very obvious. A text chatbot can pause for a second and nobody really notices. A phone caller notices.

Turn-taking

Turn-taking is the question of when the caller’s turn ends and the agent’s turn starts.

Text chat mostly avoids this. Voice does not. A person can pause mid-sentence, say “um”, talk over the agent, press a key, or hang up while the model is thinking.

Good turn-taking needs VAD, STT partial/final transcripts, interruption handling, and prompts that tell the model what it has already said.

Twilio

Twilio is the telephony platform in this project.

It owns the public phone number, receives the inbound PSTN call, calls the /voice webhook, opens the Media Streams WebSocket, sends caller audio to the app, receives generated audio back, and performs the warm transfer call control.

The important thing I had to learn is that Twilio is not just a dumb audio pipe. Twilio has its own call state, identifiers, webhook signatures, TwiML instructions, media stream protocol, status callbacks, conferences, and REST APIs. If you want to move a live caller from an AI stream into a human warm transfer, you need to use Twilio call control deliberately.

Relevant products/frameworks: Twilio Programmable Voice, Twilio Media Streams, TwiML, Twilio REST API, Twilio Conferences, ConversationRelay.

TwiML

Twilio Markup Language.

TwiML is XML that tells Twilio what to do with a call. The webhook gets an inbound call, returns TwiML, and Twilio executes it.

Examples:

<Response> <Connect> <Stream url="wss://example.com/ws" /> </Connect> </Response>

or:

<Response> <Gather numDigits="1"> <Say>Press 1 to accept the call.</Say> </Gather> </Response>

The important bit: TwiML is not just a response format. It is call control.

Relevant docs: Twilio TwiML for Programmable Voice.

VAD

Voice Activity Detection.

VAD decides whether someone is speaking or not. That sounds small, but it is one of the things that makes a voice bot feel natural or painful.

If VAD ends the turn too early, the bot interrupts. If it waits too long, the call feels slow. If it treats background noise as speech, the agent can get confused.

In the Pipecat build, Silero VAD is used in the LLM user aggregator path. Cartesia Line handles turn-taking and interruption detection at the platform layer.

Relevant frameworks: Silero VAD, Pipecat VAD,

Warm transfer

Warm transfer means the caller is not blindly transferred.

The agent first gathers enough context, then contacts the human, gives them a short brief, and only connects the caller if the human accepts.

In this project, the warm transfer flow uses Twilio call control:

  1. The model calls the transfer tool.
  2. The app prepares transfer state.
  3. The bot tells the caller to hold.
  4. The app waits for a Twilio media mark confirming the hold prompt played.
  5. The app redirects the live caller call into a conference.
  6. The app calls the human agent.
  7. The human hears a brief and presses 1.
  8. Twilio joins the human to the conference.
  9. If no answer, busy, failed, or declined, the caller returns to the AI to leave a message.

This is more work than a cold transfer, but it avoids a bad caller experience.

Webhook

A webhook is an HTTP callback from a service to your app.

Twilio uses webhooks heavily. Incoming call? Webhook. Human leg answered? Webhook. Gathered digit? Webhook. Transfer status changed? Webhook. Conference event? Webhook.

The important bit is that Twilio signs webhook requests using the public URL it called. If your app validates the wrong URL because it sees an internal Cloud Run URL or localhost, validation fails.

In our app, request validation builds candidate public URLs from PUBLIC_BASE_URL, PUBLIC_HOST, forwarded headers, host headers, and the ASGI URL.

Relevant docs: Twilio webhook security

WebSocket / WSS

WebSocket is the long-lived connection used for real-time media or event streams. WSS is WebSocket over TLS.

For Twilio voice AI, the URL must be public and secure. Localhost does not count unless it is behind a public tunnel. Cloud Run, ngrok, and load balancers can also make this confusing because the app may see an internal http://… URL while Twilio called the public https://… URL.

In the current app, /voice returns TwiML that points Twilio at /ws, and /ws validates a short-lived stream token before accepting the call media.

The post The ABCs of Voice AI Agents appeared first on ISbyR.

]]>
https://isbyr.com/the-abcs-of-voice-ai-agents/feed/ 2
Add new LLM models to Splunk MLTK https://isbyr.com/add-new-llm-models-to-splunk-mltk/ https://isbyr.com/add-new-llm-models-to-splunk-mltk/#comments Tue, 24 Jun 2025 14:57:26 +0000 https://isbyr.com/?p=1234 Splunk MLTK 5.6.0+ allows you to configure LLM inference endpoints, but the list is somewhat limited. Below, I’ll explain how you can add new LLM models to Splunk MLTK. The Issue You can configure any of the pre-added models in the Splunk UI by going to the MLTK App and then hitting the “Connection Manager” … Continue reading Add new LLM models to Splunk MLTK →

The post Add new LLM models to Splunk MLTK appeared first on ISbyR.

]]>
Splunk MLTK 5.6.0+ allows you to configure LLM inference endpoints, but the list is somewhat limited. Below, I’ll explain how you can add new LLM models to Splunk MLTK.

The Issue

You can configure any of the pre-added models in the Splunk UI by going to the MLTK App and then hitting the “Connection Manager” tab.

When you select a service, you can see a list of pre-defined models. These are already somewhat outdated, for example, for Gemin, you don’t have any of the 2.5 models.

So, “how do we add new LLM models to Splunk MLTK?” you might ask.

The Solution

Easy-ish…

A bit of background

This configuration is managed in a Splunk KV Store collection (named mltk_ai_commander_collection), and in essence, it’s a big JSON that has all the providers and the models.

For example, here is the snippet for the Gemini Service and the first of its models

        "Gemini": {
            "Endpoint": {
                "value": "https://generativelanguage.googleapis.com/v1beta/models",
                "type": "string",
                "required": false,
                "description": "The API endpoint for sending chat completion requests to Google's Gemini language model."
            },
            "Access Token": {
                "value": "",
                "type": "string",
                "required": true,
                "hidden": true,
                "description": "The authentication token required to access the Gemini API."
            },
            "Request Timeout": {
                "value": 200,
                "type": "int",
                "required": false,
                "description": "The maximum duration (in seconds) before a request to the Gemini API times out."
            },
            "is_saved": {
                "value": true,
                "type": "boolean",
                "required": false,
                "description": "Is Provider details stored"
            },
            "models": {
                "gemini-pro": {
                    "Response Variability": {
                        "value": 0,
                        "type": "int",
                        "required": true,
                        "description": "Adjusts the response's randomness, impacting how varied or deterministic responses are."
                    },
                    "Maximum Result Rows": {
                        "value": 10,
                        "type": "int",
                        "required": false,
                        "description": "The maximum number of result entries to retrieve in a response."
                    },
                    "Max Tokens": {
                        "value": 2000,
                        "type": "int",
                        "required": false,
                        "description": "The limit on the number of tokens that can be generated in a response."
                    },
                    "Set as default": {
                        "value": false,
                        "type": "boolean",
                        "required": false
                    }
                },

So if we want to add a new model, all we need to do is add another element to the models array.

While there is a Loolup Editor app, it will only help you (to edit KV store collections) if there is a lookup configured for it. Which is not the case for the mltk_ai_commander_collection one.

High-level steps

Another way (and the one we will take) is to use Splunk REST API, and at a high level, it consists of the following steps:

  1. Get the current configuration (and the _key of the collection item) in a JSON format
  2. Update in a text editor the JSON payload
  3. Update the KV collection with the new JSON

Detailed steps

I will provide examples using Postman, but you can use curl or any other method of your choice for interacting with the REST API.

Get the current configuration

Run a GET call to the collection/data endpoint

The actual URL is https://localhost:8089/servicesNS/nobody/Splunk_ML_Toolkit/storage/collections/data/mltk_ai_commander_collection

Copy the results and take a note of the _key at the end of the JSON.

Update the JSON

Paste the JSON in a text editor of your choice.

Go to the Provider for which you want to add a new Model (Gemini) in our case,

Duplicate the model object inside the Service object and change the model name.

For example, here I copied/pasted the gemini-2.0-flash to the end of the Gemini service object and renamed it to be gemini-2.0-flash.

NOTE: You must ensure that the model name you provide here is exactly the same as it would appear when calling the inference API for the LLM Service.

For example, for Gemini

Update the KV collection

Now we need to update the collection with the updated JSON payload.

Send a POST request to the collection/data endpoint

  • replace the _key part of the URL with the value that you have in your JSON
  • remove the square brackets ([]) that surround the JSON

The actual URL is something like that: https://localhost:8089/servicesNS/nobody/Splunk_ML_Toolkit/storage/collections/data/mltk_ai_commander_collection/68540d2d0d2a214efd0d3b61.

Now, refresh the Connection Management page and enjoy a fresh new model at your disposal

Simply use the new model in the | ai command

And here is a sneak peek into an LLM Telemetry dashboard I’m working on

I hope that helped you to understand how to add new LLM models to Splunk MLTK.

The post Add new LLM models to Splunk MLTK appeared first on ISbyR.

]]>
https://isbyr.com/add-new-llm-models-to-splunk-mltk/feed/ 2
Learning about RAG and Vector Databases https://isbyr.com/learning-about-rag-and-vector-databases/ https://isbyr.com/learning-about-rag-and-vector-databases/#respond Thu, 21 Mar 2024 14:19:25 +0000 https://isbyr.com/?p=1174 I am learning about different concepts and architectures used in the LLM/AI space and one of them is Retrieval-Augmented Generation. As always I prefer learning concepts by tinkering with them and here is my first attempt at learning about RAG and Vector Databases. A bit of the terminology, I will not dive too deep here, … Continue reading Learning about RAG and Vector Databases →

The post Learning about RAG and Vector Databases appeared first on ISbyR.

]]>
I am learning about different concepts and architectures used in the LLM/AI space and one of them is Retrieval-Augmented Generation. As always I prefer learning concepts by tinkering with them and here is my first attempt at learning about RAG and Vector Databases.

A bit of the terminology,

I will not dive too deep here, but just enough to get started. The definitions below are my simplified understanding, and they are most likely not fully correct.

What is RAG

There are many places where you can learn about RAG, but for the context of this post, I’d say that RAG allows you to supplement the initial prompt for the LLM with a bit more (or a lot more, that’s up to you) context.

What is a Vector Database?

Vector Database is one of the mechanisms/data stores that will enable you to provide this additional context to the LLM. Unline “regular” databases, vector database doesn’t necessarily store the actual data (though it can), but it will store the embedding of the data you later wish to search to retrieve the above-mentioned context.

What are embeddings?

Embeddings are multi-dimensional numerical representations of a piece of data (text for example), The multi-dimensionality allows to “place” semantically similar terms close to each other. For example, if using semantic search we search for “dog” then “puppy” and “mutt” and mutt will be considered close terms, while if using a lexical search (one that looks at the literal text similarity), will probably consider “dogma” and “hot dog” as closer terms.

The ITSM Assistant

The problem

Now let’s say you want to open a ticket in your ITSM ticketing system that your Internet Access is not working properly. You could start by searching for a particular request type or a knowledge base article, but what if you are not a technically savvy person and all you care about is that you can’t get to Facebook?

The Solution

ITSM Assistant to the rescue!!! It’s a chat interface that will:

  1. ask a user about the issue they are currently facing
  2. will look in the vector database for semantically similar historical requests, and get their IDs
  3. get the content of the tickets from the data store (simple CSV file int in this case)
  4. will feed this context to an LLM
  5. provide a user with the suggested request form and some of the fields that should be populated

As you can see in the screenshot below the user didn’t mention that they have problems with “internet access”, but just said, “I can’t get to Facebook”. Despite that ITSM Assistant was able to pull data that is related semantically to the user’s issue. LLM (after being fed all the context) suggested the correct Service Request type and some of the information the user should add to the ticket for it to be promptly resolved.

How it works under the hood?

ITSM Assistant using RAG and Vector Databases Solution Diagram

The components

  • Pinecone – vector database
  • Streamlit – “…turns data scripts into shareable web apps in minutes.”, both front and back-end all in Python.
  • Stramlit Community Cloud – for hosting the Stremalit app
  • AzureOpenAI – the LLM
  • all-MiniLM-L6-v2 – “…a sentence-transformers model: It maps sentences & paragraphs to a 384-dimensional dense vector space”

Step 0 – Load the data into Vector Database

I found an ITSM ticket dump on the internet.

Next, we need to get embedding for each ticket and insert it into the vector database (Pinecone in my case).

I had a Jupiter notebook that was doing this job.

# Importing the necessary libraries
import pandas as pd

# Importing the csv file
data = pd.read_csv('GMSCRFDump.csv', encoding = 'ISO-8859-1')

# removing duplicate tickets
ID_mins = data.groupby(['Title', 'Description', "CallClosure Description"]).ID.transform("min")
data_n = data.loc[data.ID == ID_mins]

# create a new array with a field that has both title and description each ticket
title_description = data_n['Title'] + " __ " + data_n['Description']
# create an arraid of ticket IDs
tid = data_n['ID']

# import a transformer that will be used to encode the ticket data
from sentence_transformers import SentenceTransformer
import torch

# Define the model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer('all-MiniLM-L6-v2', device=device)

# Setup Pinecone connection
import os
os.environ['PINECONE_API_KEY'] = '1b5da094-f784-4beb-8fc3-262712a667ae'
os.environ['PINECONE_ENVIRONMENT'] = 'gcp-starter'

from pinecone import Pinecone, PodSpec

# get api key from app.pinecone.io
api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'
# find your environment next to the api key in pinecone console
env = os.environ.get('PINECONE_ENVIRONMENT') or 'PINECONE_ENVIRONMENT'
pinecone = Pinecone(api_key=api_key)

# Create index

index_name = 'snow-data'
# only create index if it doesn't exist
if index_name not in pinecone.list_indexes().names():
    pinecone.create_index(
        name=index_name,
        dimension=model.get_sentence_embedding_dimension(),
        metric='cosine',
        spec=PodSpec(
            environment=env, 
            pod_type='s1.x1'
        )
    )

# now connect to the index
index = pinecone.Index(index_name)

# the following section, takes a batch of tickets, for each one of them makes and embeeding, "attaches" id and title+description as meta data , and upserts that into Pinecone index

from tqdm.auto import tqdm

batch_size = 120
vector_limit = 12000

title_description = title_description[:vector_limit]
title_description

for i in tqdm(range(0, len(title_description), batch_size)):
    # find end of batch
    i_end = min(i+batch_size, len(title_description))
    # create IDs batch
    ids = [str(x) for x in range(i, i_end)]
    # create metadata batch
    metadatas = [{'tid': t_id, 'text': t_desc} for t_id, t_desc in list(zip(tid,title_description))[i:i_end]]
    print(metadatas)
    # create embeddings
    xc = model.encode([t_desc for t_desc in title_description[i:i_end]])
    # create records list for upsert
    records = zip(ids, xc, metadatas)
    # upsert to Pinecone
    index.upsert(vectors=records)

Step 1 – The Streamlit App

Streamlist is a straightforward Python framework that allows you to build (simple) web apps. All without any HTML, JavaScript and CSS knowledge. You can run it locally or host it somewhere, for example using their Community Cloud.

You can find the code for the app in the ITSM Assistant repo here. So I’ll not provide much code from now on, but instead, talk to write about my any caveats.

To try it at home one will need to create secrets.toml file under the .streamlit folder and populate it with your Azure OpenAI and Pinecone credentials/configuration

AZURE_OPENAI_API_KEY = "xxxxxxxxxxxxx"
AZURE_OPENAI_ENDPOINT = "https://xxxxxxxxx.openai.azure.com/"

PINECONE_API_KEY = "xxx-xxx-xxx-xx"
PINECONE_INDEX = "snow-data"

Steps 2 and 3 – Searching for Historical Tickets

One caveat is depending on the amount of data one can decide to upsert into the vector db (in addition to the embeddings themselves) not only ticket ID (as metadata), but all the ticket fields (like Description, Resolution, etc.). This way your semantic search can return all the data you need and there is no need to have Step 4 (retrieval of data from data store)

For the sake of learning, I did not, so after we get ticket IDs from Pinecone, we use it to filter the data in the data store (fancy name for CSV) to get the ticket information that needs to be sent as context to the LLM.

Step 4 – Ask LLM for help

Now that we have the context (similar ticket data) we can send the request to LLM to help our struggling user and point them in the right direction.

Step 5 – Response to User

The only thing worth mentioning here is I had a bit of a struggle in printing the list of fields nicely.

LLM is coming back with a JSON response similar to below:

{
  "common_theme": "Server Reboot",
  "title": "Server Reboot Request",
  "suggested_fields": "SSO ID, Name, Email, Seat Number, Location, Cell Phone No"
}

Streamlit can use markdown for output, so to format the list of fields nicely I had to do something like this:

suggested_fields = llm_response['suggested_fields'].split(', ')
suggested_fields = "- " + "\n- ".join(suggested_fields)
 
nl = "  \n"
st.chat_message("ai").markdown(f"It looks that in order to help you, you will need to raise a new **\"{llm_response['title']}\"**.{nl}\
When raising this request please provide some of the required information like:{nl}{suggested_fields}")

P.S.

You can find the app here: https://app-itsm-assistant.streamlit.app/

More posts related to my AI journey:

The post Learning about RAG and Vector Databases appeared first on ISbyR.

]]>
https://isbyr.com/learning-about-rag-and-vector-databases/feed/ 0