Maru

It was a fucking great presentation and so cool. But what was most interesting was an experiment he conducted where he tested how accurately LLMs can answer questions when given curated knowledge bases to retrieve information from (aka "Retrieval Augmented Generation"):

In IDEAL CIRCUMSTANCES, where the LLM was artificially given the exact right piece of info to answer the question, it answered correctly only 75% of the time. That was the best case scenario! Only achievable in lab settings!

In "normal" circumstances, so using the standard algorithms to allow the LLM to retrieve the information from its knowledge base by itself, it only answered correctly roughly 60% of the time.

WITHOUT A CURATED KNOWLEDGE BASE IT ANSWERED CORRECTLY 7% Of THE TIME.

LLMs are so fucking stuuuuupid. Because it is a LANGUAGE ALGORITHM, it's good at stringing together coherent sentences based on an input. That's what it was built for. To make words. It is REALLY, REALLY not a Answer Questions or Complete Tasks algorithm.

goma-ramen　"After the rain"