How I utilize AI LLM prompts of questions and answers

Programmer: Stan Switaj

I ask my local AI models a lot of questions. Some of them I ask again a week later, worded a little differently, and every time the model works the answer out from scratch, as if we had never talked. So I gave my AI a memory: a PostgreSQL knowledge base of questions and answers. Now, when I ask a question I've asked before, the AI first shows me what it answered before, sums those answers up, and then gives me a fresh one.

Here is how it works and what it looks like.

The idea in one picture

Every question goes through four steps:

1. I ask a question, in whatever words I like: "what color is a fire truck?", "a firetruck is what color?", "fire engine color?".
2. PostgreSQL is searched, by meaning and by words, in about half a second, before the AI even starts.
3. The AI decides whether one of the saved questions asks the same thing as mine.
4. The answer is saved back, so the next time the history is one answer longer.

What the answer looks like

When the question has been asked before, the AI answers in three parts.

1. Saved responses

First come the answers it gave before, straight from PostgreSQL, newest first, each with the date and time it was saved. How many it shows is a setting: I keep it at the last 5.

2. Summary of saved responses

Then a short summary of those saved answers: what they agree on, and how the answer has changed over time.

3. Fresh answer

Finally the AI's own answer, worked out now as it normally would. This one is saved too and becomes the newest entry in the history.

Here is the whole reply as it appeared in OpenCode, with GLM-5.3-Flash running on my own server:

The question here was "fire engines - what colours are they painted?". It was matched to the saved question "what color is a fire truck?", even though the words are different.

Same question, or just the same topic?

This turned out to be the interesting part. To find "the same question in other words", the knowledge base turns each question into an embedding, a list of numbers that captures its meaning, and compares them. I expected a simple rule like "a match above 0.85 is the same question" to work. It doesn't.

I measured eight pairs of questions that mean the same thing and eight pairs that are on the same topic but ask something different, on four different embedding models:

• "what color is a fire truck?" vs "fire engines are usually which colour?" (same question) scored as low as 0.81
• "how do I restart the server?" vs "how do I stop the server?" (different questions) scored as high as 0.93

No threshold can separate those two groups, and bigger models didn't fix it. So the database does what it is good at, finding the closest candidates quickly, and the AI decides whether one of them really asks the same thing, which is what language models are good at. "Restart" and "stop" are different questions; "fire engine colour" and "fire truck color" are the same one.

That's why a question like "what color is a passenger bus?" gets its own entry instead of being mixed in with fire trucks. The summary only ever covers answers to the same question.

Common words: the grass and sky problem

A day after the first version, I asked "What color is grass?" and the knowledge base offered "What color is the daytime sky?" as a possible match. The AI correctly decided they were different questions, but the sky should never have been offered at all. The two questions share only common words: what, color, is.

My suggestion was simple: question words and words like is and are should carry less weight, because they are common words. That is exactly what the fix does, in two layers:

1. A list of common words (what, which, how, why, is, are, do, the, a, of, and so on) that carry no weight at all when two questions are compared word by word.
2. Words that are common in the knowledge base itself carry little weight too. I have asked several "what color is…?" questions, so color has become a common word in my knowledge base, while grass, sky and fire are rare and distinctive. This is the classic search-engine idea of IDF (inverse document frequency): the rarer a word is, the more it tells you.

The rest of each question is cleaned up a little (colour = color, engines = engine, and fire truck = firetruck). Then the knowledge base measures how much of the distinctive wording two questions share, and folds that into the match:

match = meaning × (0.85 + 0.15 × shared distinctive words)

Sharing only common words lowers a match by up to 15%. Sharing the words that matter keeps it where it was.

Anything below 0.70 is not shown to the AI, so the grass question now comes back clean and gets its own entry.

Interestingly, the obvious version of the idea, deleting the common words before computing the meaning, did not work: grass vs. sky actually went up, from 0.72 to 0.76. Embedding models already mostly ignore words like is and what. What was missing was a check on which words the two questions share, and that is where the common-word list earns its keep. I tested it on 12 reworded questions (all still match) and 11 pairs of different questions: the ones that share only common words now drop out, and the genuinely tricky ones ("restart the server" vs "stop the server") still go to the AI to decide.

Under the hood

• PostgreSQL holds three tables: the questions, every answer ever given to each question (the response history), and a raw log of every prompt and reply.
• Search by meaning uses a small embedding model (bge-small-en-v1.5) that runs on the CPU in a few milliseconds. Search by words uses PostgreSQL's own full-text search plus fuzzy (trigram) matching for typos and near-spellings. The two are merged, with meaning counting double. Common words are discounted, so two questions must share distinctive words, not just what and is, to count as a strong match.
• A small command-line tool does all the database work: search, save, show the history, list.
• An OpenCode plugin connects it to my AI coding assistant. The moment I press Enter it runs the search and attaches the matching saved answers to my prompt, so the model has them right away without an extra step. When the answer is finished, it files the fresh answer under the right question, or as a new question.
• A one-line setting in a config file in the project folder, provide_responses = 5pyjscsshtmlphpadbshccppjavaktgorsluaplsqlmdtxt, decides how many saved answers to show.
• Only questions are stored. Coding tasks ("write a program", "fix this file") are not, and anything that looks like a password or key is refused.

It all runs on my own machines with open models (GLM-5.3-Flash and Qwen3.8). Nothing goes to a cloud service.

Why I like it

• Answers I can compare. I see what the AI said before next to what it says now, and the summary shows me at a glance whether the answer is stable or has changed.
• Faster to the point. The saved answers are there in half a second, before the model has written a word.
• It gets better with use. Every question I ask makes the knowledge base a little more useful.
• It catches drift. If a fresh answer disagrees with the saved ones, the AI says so, and I know to look closer.

What's next

The same approach could group questions by topic too, for example "related questions you asked about vehicle colours". For now, one clean history per question is exactly what I wanted.

Search Engine Implementation

The search engine of BestInternetSearch.com I implemented this way of providing prompt question and answers, perhaps try it out with a question or two.