Why RAG Is Becoming Essential for Banking and Compliance

I recently built a custom Retrieval-Augmented Generation (RAG) pipeline to move past the limitations of keyword matching. By processing a regulatory document, chunking the text strategically, and embedding it into ChromaDB, I wanted to test how semantic search handles complex, domain-specific text like Banking and Compliance.

The results pointed to a clear conclusion: RAG is becoming essential infrastructure in highly regulated environments.

The Basics: What Actually Is RAG?

Standard AI models generate answers based on their pre-trained memory. If you ask them about your company's private, highly specific regulatory data, they either don't know the answer or, worse, hallucinate one.

RAG fixes this by breaking the process into two distinct steps:

Retrieval: First, the system searches your private, secure database to find the exact paragraphs relevant to your question.
Generation: Second, it gives those paragraphs to the LLM and instructs it: "Answer the user's question using only this verified text."

It essentially gives the LLM an open-book test, ensuring answers are grounded in your actual, private data (while keeping it private) rather than the model's unpredictable memory.

Vector Databases vs. Traditional Databases: Searching by Concept

To make the "Retrieval" step work, you need a Vector Database. Traditional databases store information in rigid rows and columns. When you search them, they look for exact keyword matches. If you search for "automobile," a traditional database will completely miss a document that only uses the word "car."

Vector databases, like ChromaDB, are completely different. They take unstructured text and turn it into a high dimensional array of numbers called a vector. These vectors represent the actual semantic meaning of the text. Instead of searching by exact words, they search by concept.

To find a match, the database measures the geometric relationship between your query's vector and the document vectors. In practice, vector databases offer three standard ways to do this: Euclidean (L2) distance, which measures the straight-line gap between two points; the raw dot product, which rewards both alignment and magnitude; and Cosine Similarity, which measures only the angle between the vectors. For text embeddings, Cosine Similarity is the most popular choice: two sentences can carry the same meaning at very different lengths and intensities, and cosine ignores all of that by comparing direction alone. In embedding space, direction is meaning.

The Mathematics of Meaning: Why We Use Cosine Similarity

When you embed text into a vector database like ChromaDB, sentences are transformed into dimensional vectors. These vectors represent the meaning of the text, like what the text is really saying. To find the context for a user's query, such as a question, the database must compute the directional alignment between the query vector and the document vectors.

We rely heavily on Cosine Similarity for this validation because it measures how closely two vectors point in the same direction. The mathematical formula is defined as:

Cosine(θ) = (A · B) / (||A|| × ||B||)

How the math plays out:

Cosine measures alignment: If two pieces of text share the same semantic meaning, their vectors point in the same direction. If the angle between them is 0, since cos(0) = 1, we get a close match. If two pieces of text are unrelated, their vectors sit near 90 degrees, and cos(90°) = 0. If they point in opposite directions, cos(180°) = −1. That gives us a clean, rankable scale: +1 (same meaning), 0 (unrelated), −1 (opposite).

What if you actually used Sine?

This is the question every student asks, because we learn sine and cosine as a pair. Here is the catch: sin(0°) = 0, but sin(180°) = 0 as well. Sine rises to a peak at 90 degrees and then falls straight back down. A sine-based system would therefore give two sentences with identical meaning and two sentences with completely opposite meaning the exact same score: zero. Sine only measures how perpendicular two vectors are. It cannot rank similarity, and ranking similarity is the entire job.

There is also a structural detail. In high-dimensional spaces where standard cross-products do not exist, the Sine of the angle between two vectors has to be derived from cosine anyway, using the Pythagorean identity:

sin(θ) = √(1 − cos²(θ))

So any "sine-based" search engine would secretly be a cosine engine with extra steps. And if you genuinely want a dissimilarity score, you do not need sine at all. You simply use Cosine Distance, defined as 1 − cos(θ), which rises smoothly from 0 (identical) to 2 (opposite) without ever losing the ability to rank.

So why is Cosine the industry standard?

Here is the part most people miss: the computer never calculates a single trigonometric function. Look at the formula again. Cosine Similarity is just the dot product of two vectors divided by their lengths — pure multiplication and addition, which is exactly the operation modern GPUs are optimized to execute across thousands of dimensions in parallel. Many embedding models even normalize their vectors to a length of 1 before storage, at which point Cosine Similarity collapses into a plain dot product. The "cosine" in the name is geometry, not a trig calculation.

The Architecture: Moving Past Ctrl+F

Standard keyword search relies on token matching. If a compliance auditor searches for "Liquidity Coverage Ratio issues", a traditional system may completely miss sections that discuss a specific phrase, such as "term high-quality liquid asset shortfalls."

This separation of concerns allows institutions to leverage foundation models as reasoning engines while keeping proprietary data completely secure. That is why RAG is one of the most valuable tools for banks and financial institutions.