# Embeddings vs LLMs: Why Does RAG Need Both?

While working with RAG, I wanted to understand why we need an embedding model and an LLM separately.

If the LLM can understand and answer questions, what exactly is the embedding model doing?

So I tried a small experiment where I kept retrieval and answer generation separate.

That made the difference between the two much clearer.

## A simple support example

I created a small knowledge base with a few support cases around login issues, billing, file uploads, and account security.

Then I asked:

**“I entered the wrong password several times and now I can't sign in.”**

The knowledge base didn't contain that exact sentence. The relevant information was written differently:

**“Accounts are temporarily locked after five failed login attempts. The account unlocks automatically after 30 minutes.”**

This was useful because the retrieval couldn't depend only on matching the same words.

## What the embedding model did

I used `nomic-embed-text` to convert the question and each support case into embeddings.

I then compared them using cosine similarity.

The account-lockout case came out as the closest match with a similarity score of **0.649**.

The embedding model didn't answer the question. It simply helped find the information that was most related to what the user was asking.

## What the LLM did

Once I had the relevant support case, I passed it along with the user's question to `llama3.2:3b`.

The LLM returned:

**“Your account is now temporarily locked due to the five failed login attempts. Don't worry, it will unlock automatically after 30 minutes.”**

So the LLM wasn't doing the retrieval here. It received the information that had already been found and turned it into a clear answer.

## What if retrieval gets it wrong?

I also tried giving the LLM an unrelated support case while keeping the same login question.

This is where the separation became more obvious.

The LLM can only work with the context we give it. If retrieval brings back the wrong information, the LLM doesn't suddenly have access to the correct information from our knowledge base.

So getting the retrieval right matters just as much as generating a good answer.

## What about using the LLM alone?

I also asked the same question directly to the LLM without giving it any retrieved context.

It could still give a reasonable answer because login and account-lockout problems are common and the model already has general knowledge about them.

But that's different from answering using our own knowledge base.

If our system had a specific rule, policy, or piece of information that the LLM didn't already know, we would need to retrieve that information and give it to the model.

## What I took from this

The embedding model and the LLM are solving two different parts of the problem.

**Embeddings help find the right context. The LLM uses that context to generate the answer.**

A strong LLM doesn't remove the need for good retrieval. If the wrong context is retrieved, the LLM has the wrong information to work with.

That's why both parts matter in a RAG system.

## Experiment

I’ve kept the complete experiment and code on GitHub:

**GitHub:** <https://github.com/VASHNIROY/Churn-prediction/tree/main/experiments/embeddings-vs-llm>