Reading this was very valuable. I really appreciate the Vespa mention and introduction to their Multi-Vector HNSW Indexing - I’ve recently thought a lot about how difficult chunking is and this seems like a promising avenue.

This is one of the things we learned recently about building production workflows with LLMs. Happy to answer any questions/feedback here <3

For me it's nice to see the re-use of existing infra - big query. Personally, I was rooting for Postgres, but your logic of why not makes sense! Great post.

I don't know what a RAG is (and apparently it's forbidden to explain). And at this point, I'm afraid to ask.

Taken from the same blog:

"Roughly, RAG is runtime prompt engineering where you build a system to dynamically add relevant things to your prompt before you ask the agent for an answer."

Have you tried asking Google LLM RAG or ChatGPT what RAG in the context of LLMs is?

> “Retrieval-Augmented Generation” is nothing more than a fancy way of saying “including helpful information in your LLM prompt.”

Here to learn more about this. Important for a startup I know.

RAG is as valuable as the data you can retrieve.

If the amount of data is small you don't need the flexibility of RAG. And if it is irrelevant it will stay irrelevant after found.

For sure, it's only worth doing if you actually have so much relevant data that it doesn't fit in the context! This is definitely the case for us for this problem, but it's not universal.

There's hype and FOMO for sure and you're right that there's lots to learn from information retrieval work. But why be dismissive of the whole thing? People learning from past research and applying it in new contexts seems like a good thing and there's people (like https://x.com/bclavie and https://x.com/jobergum), who are legit experts in information retrieval and are showing how to apply it to RAG properly.

There are knowledgeable people out there with excellent use-cases for RAG, to find data that is relevant. And they come from years of using the previous state of the art techniques in information retrieval. They have all my support and I am looking forward to read more about it.

But the majority of talk about RAG is coming from hype driven individuals that produce low merit and uninformed opinions.