RAG for Behavior, Not Answers: Lessons from an AI Roleplay Trainer

Here is the first version of my retrieval layer, in its entirety:

// The mistake, in one line:
var embedding = await embeddingGenerator
    .GenerateAsync(manualContent);

// One vector. Whole document. Silently useless.

One embedding. For the entire training manual.

I think everyone makes this mistake exactly once, and the reason it's such a good teacher is that it doesn't crash. There's no exception, no red console output, no failing test. You get a vector back. You store it. You query it. It returns your one chunk every single time, with what looks like a perfectly reasonable score, and the model happily generates something that sounds plausible.

It fails silently, which is the worst way for anything to fail.

That was the beginning of a proof of concept I spent a while on and ultimately shelved. It never went to production. But almost everything I learned from it was about the parts nobody puts in the tutorials, so it's worth writing down.

What I Was Actually Building

The goal was a training tool for sales agents who handle inbound phone calls at automotive dealerships.

There's a methodology these agents are taught — a call-script manual covering how to open, how to qualify, how to handle the objections you hear a hundred times a week, how to close. The usual way you practice it is roleplay with a manager, which is expensive, awkward, and hard to schedule.

So: let the model be the customer.

The agent picks a scenario (a caller asking about a specific vehicle they saw online, or a non-specific "what do you have in my price range" call), picks a customer personality, and picks how many objections get thrown at them. Then the phone "rings."

Because it's an inbound call, the human always speaks first. The model answers as the customer, stays in character for the whole conversation, drops the objections in where they'd naturally land, and doesn't break. When the call ends, it switches roles entirely — from customer to evaluator — and produces a score, written feedback, and a short list of improvement points, graded against whether the agent actually followed the methodology.

That last part is what made this interesting, and it's where the RAG stopped resembling anything I'd read about.

This Isn't the RAG You've Read About

Nearly every RAG article you'll find describes the same shape: you have documents, a user asks a question, you retrieve the passages that answer it, you paste them into the prompt, the model summarizes them back. Retrieval-augmented question answering. The retrieved text is the source of the answer.

That's not what was happening here.

I was retrieving behavioral instructions for a model that had to act. The chunks weren't material to be summarized — they were the rules of a performance. And they were being used in two completely different modes:

• During the call, the model needed to know how a customer following this methodology's psychology would respond — and critically, it needed to never recite the script at the person being trained. Leaking the retrieved content would be a total failure. The agent is supposed to be practicing the script; if the customer starts quoting it, the exercise is over.
• After the call, the same corpus became the grading rubric. Now the model did need the literal text, because "you skipped the qualification step" is a claim you have to be able to ground in something.

Same documents. Two opposite relationships to them. One has to be invisible, the other has to be quotable.

I don't have a tidy framework for this. But if you're building something where the model does rather than explains, be aware that most of the advice you'll find silently assumes the explaining case.

The Document Fought Back

The corpus was a single semi-structured text file, and it was hostile to every naive chunking strategy I tried.

It had:

• A four-level hierarchy, expressed with dash prefixes — -, --, ---, ----. Depth carried meaning.
• ID-based cross-references scattered throughout. A section would say, in effect, "if the customer pushes back here, jump to #OBJECTION-PRICE, then come back and resume at #CLOSE."
• Branching conditional flows. Not linear prose. "If they say A, do this. If they say B, do that instead."
• Objection-response pairs, each mapped to the situations where it applies.
• Customer personality profiles — a four-color model where each color implies a different pace, tone, and set of buying triggers.

Now consider what a standard recursive token splitter does to that. It counts tokens. It finds a separator near the boundary. It cuts.

Which means it will happily:

1. Sever a cross-reference from its target. The chunk that says "jump to #OBJECTION-PRICE" gets retrieved. #OBJECTION-PRICE itself does not. The model now knows a door exists and nothing about what's behind it.
2. Cut a conditional flow mid-branch. This one is genuinely dangerous. The chunk contains "if the customer says they need to talk to their spouse…" and stops. The model retrieves half a decision tree — and a good model does not say "I don't have enough information." A good model completes the pattern. Fluently. Confidently. With something that sounds exactly like the methodology and isn't.

That second failure mode is the one that made me rewrite the whole ingestion pipeline. A visibly broken chunk is a bug you fix in ten minutes. A chunk that's been amputated at a branch point produces confident, well-written, methodologically wrong training — and the entire purpose of the tool was to teach the methodology correctly.

Chunking was never a size problem. It was a structural problem. The question is never "how many tokens fit," it's "what is the smallest piece of this document that is still true on its own?"

Multi-Granularity Chunking

The fix that finally worked was to stop looking for the right chunk size, because there isn't one.

Consider two questions this system had to serve:

• "What is this methodology, broadly?"
• "What exactly do I say, right now, in this moment?"

Those are different retrieval problems. They want wildly different amounts of text. And they are asked over the exact same source content.

So I indexed the same content three times, at three granularities:

Granularity	Size	The question it answers
Overview	100–500 chars	"What is this section for? When does it apply?"
Complete script	800–1,500 chars	"Walk me through this entire call segment."
Individual step	50–150 chars	"What is the actual line I say next?"

Redundant storage, deliberately. Embeddings are cheap; retrieving a 1,500-character script when the model needed one sentence is not.

The ingestion was a small Python converter that parsed the dash hierarchy and emitted JSONL — 82 semantically-bounded sections, 15 metadata fields per chunk, and all 85 cross-references preserved rather than flattened away. Roughly this shape:

{
  "id": "objection-price-03",
  "granularity": "complete_script",
  "hierarchy": ["Objection Handling", "Price", "Too Expensive"],
  "references": ["#CLOSE", "#QUALIFY"],
  "applies_to_profile": ["RED", "BLUE"],
  "stage": "mid_call",
  "text": "..."
}

Two things mattered more than the chunking itself:

Cross-references survived as data. Because references is a field, a retrieved chunk can pull in its targets — the conditional branch arrives whole instead of amputated. This alone fixed the confident-improvisation problem.

Metadata carried the conditions. applies_to_profile and stage mean retrieval can be filtered by where you are in the call and who you're talking to, not just by what words are semantically nearby. Vector similarity has no idea that a closing line is wrong in the first thirty seconds of a call. A metadata filter does.

If I had to compress this whole post into one sentence: the metadata design mattered more than the embedding model. I changed models during the project and barely noticed. I changed the metadata schema and everything got better.

Retrieval I Didn't Own

Here's the architectural decision I still go back and forth on.

I didn't build a retrieval layer. There's no vector database in this system that I run, no similarity search I wrote, no reranker I tuned. I uploaded the chunked corpus to a hosted vector store, attached it to an assistant with a file-search tool enabled, and called the API.

Retrieval happens server-side, inside someone else's infrastructure. I send a message. Chunks get selected by machinery I can't see. The model responds.

I never see a similarity score. Not once. There is no number anywhere in my application telling me how confident the retrieval was.

A note on shelf life, since I'm publishing this weeks before the specific API I used gets switched off for good: the assistant-and-thread endpoints I built against are being removed, and anyone doing this today would wire it up through the newer responses-style API instead. But hosted vector stores and a server-side file-search tool didn't go anywhere — they moved and kept growing. The plumbing changed names. The tradeoff below did not, and it's the tradeoff that matters.

I want to be honest about both sides of that, because it's a real position and not just laziness:

What you get: no infrastructure. No vector database to provision, secure, back up, or pay for at idle. No embedding pipeline to keep running as the corpus changes. No dimension mismatches, no index rebuilds, no "which distance metric" debate. For a proof of concept whose entire job was to answer "does this idea work at all," that's an enormous amount of undifferentiated work I got to skip. The PoC existed because this was cheap.

What you give up: observability, and with it, the ability to debug. When the model said something subtly off-methodology, I could not answer the first question I wanted to ask — did it retrieve the wrong chunk, or retrieve the right chunk and reason badly? Those have completely different fixes. One is an ingestion problem, the other is a prompt problem. Without scores, without even a reliable list of what got retrieved, I was reduced to changing something and re-running the conversation to see if it felt better.

That's not engineering. That's superstition with a build step.

My honest read: hosted retrieval is the right call for proving an idea, and the wrong call the moment quality becomes the thing you're working on. The crossover point arrives earlier than you'd like — roughly the first time someone asks "why did it do that?" and you don't have an answer.

Temperature 0.8, On Purpose

Standard RAG advice is to run the temperature low. Near zero. You're grounding answers in retrieved facts, so you want determinism and you want to suppress invention.

I ran this at 0.8, and it wasn't a leftover default.

A customer who responds identically every time is worthless as a training partner. The whole point is that the agent has to handle a person — someone who might be curt, might ramble, might raise the price objection in a way they haven't heard before. If the agent can memorize the customer's lines, they're not practicing a phone call, they're practicing a script recital. Predictability is the failure mode.

So the variance that RAG normally treats as a defect was, here, the product.

The tension is obvious: the same setting that makes the customer feel alive also makes the model more willing to improvise the methodology it's supposed to be following. Which loops right back to the chunking. High temperature is only survivable if your retrieved context is structurally complete. A model at 0.8 handed half a conditional branch will fill in the other half beautifully and wrongly. The same model at 0.8 handed a whole branch, with its cross-referenced targets attached, stays inside the lines and varies only its delivery.

The grading pass, of course, ran cold. Different job, different setting.

The Road Not Taken

I sketched a second version on AWS and never built it out. Recording it here because the comparison clarifies what the first version actually chose:

	Version A (built)	Version B (sketched)
Embeddings	Hosted, opaque	Titan Embeddings V2, 1,536 dimensions, cosine
Store	Hosted vector store	Vector buckets on S3, wired to a Bedrock Knowledge Base
Retrieval	File-search tool, server-side	RetrieveAndGenerate, or bare Retrieve
Similarity scores	Never visible	Yours
Idle cost	Effectively none	Non-zero
Time to first working demo	An afternoon	Considerably more than an afternoon

The interesting choice in Version B is Retrieve versus RetrieveAndGenerate. The combined call is the convenient one — it retrieves and generates in a single round trip. But this application's prompt was doing a lot of work: hold a persona, inject objections on a schedule, never leak the source text, then switch roles and grade. That's not a prompt you want managed for you.

So Version B would have used bare Retrieve and owned the generation call outright. Which is a decent rule generally: the moment your prompt is a real asset, stop using the convenience wrapper that owns it.

Why It's On The Shelf

The proof of concept worked. It proved what it was supposed to prove — that a model given a properly structured methodology can hold a customer persona through a full call and then produce grounded, specific feedback about whether the script was followed. That question got answered, and the answer was yes.

There was no budget to productionize it. That's the whole ending. Not a technical wall, not a failure — the PoC did its job and the next phase wasn't funded.

The one honest technical caveat I'd carry into any second attempt is the conversation loop. The API's polling model was fine. Latency was acceptable, the pauses weren't painful, and it never broke the exercise. But it wasn't a phone call. Anyone who's used a real-time voice mode knows the difference immediately — the turn-taking, the ability to interrupt, the absence of that small dead beat before every reply. For a tool whose entire premise is "this should feel like an inbound call," fine is a meaningful gap. A production version would need to be built on a real-time voice pipeline from the start, not a text loop with speech bolted on.

What I'd Keep

Stripped of the specifics, here's what survives:

1. One embedding for a whole document is a silent failure. It returns a vector. It returns results. It is completely useless, and nothing tells you.
2. Chunking is structural, not dimensional. Stop asking how many tokens fit and start asking what the smallest self-sufficient unit is. For documents with branches and cross-references, a token splitter is actively destructive.
3. Index the same content at multiple granularities. "What is this?" and "what do I say now?" are different retrieval problems over identical text. Storage is cheaper than the wrong-sized answer.
4. Preserve cross-references as metadata, not prose. A reference that survives as a field can be followed. A reference that survives as text is just a string the model can't act on.
5. Metadata design beats embedding model choice. Swapping models was barely noticeable. Adding the right filter fields changed everything.
6. Hosted retrieval trades observability for speed. Take that trade to prove an idea. Reconsider it the first time you need to explain a bad output.
7. Match temperature to the job, not to the pattern. RAG's usual "keep it near zero" assumes you're answering questions. If your model is performing rather than reporting, variance is the feature — but only if the retrieved context is complete enough to survive it.
8. The unusual RAG shape is worth naming. Retrieving behavior is different from retrieving facts, and most published advice quietly assumes the latter.

If you're curious about the smaller, more self-contained end of local AI work, I wrote about building an image caption generator in C# with a local model a while back — a much simpler project, and a good place to start if this one sounded like a lot.

---

Shelved isn't the same as wasted. The PoC answered its question, and the chunking lessons transferred to every retrieval problem I've touched since.

---

Happy coding! ⚡