One of the nicest things about building AskTrip in public is that good feedback does not just help us explain the product better. Sometimes it directly changes what we build next.

That has happened again.

After Luis Carlos’s thoughtful comments last week, which I wrote about in Negative feedback is the best, we received another very helpful nudge from Helen-Ann. Different issue, same pattern: a user points out something important, and it opens up a better way forward.

Helen-Ann asked a question that returned only three references in Trip. What was interesting was not just the low number, but what happened next. Or rather, what did not happen next. Beyond Trip did not trigger.

Beyond Trip is AskTrip’s fallback mechanism for questions where Trip itself finds too little. At the moment, it searches Google Scholar and OpenAlex, but only towards the end of the answering process. That means the system has already done most of the work before it realises it needs to go further. At that point it has to begin again, which can almost double the response time.

In this case, Beyond Trip did not trigger because the system believed it had found enough to work with. It identified six relevant papers, which was above the threshold for invoking the fallback search. Only three of those were eventually used in the answer, but the system does not make the Beyond Trip decision based on the final number selected. It makes that decision earlier, based on how many papers appear relevant at that stage. That is the key weakness this feedback exposed.

Helen-Ann kindly sent me the papers she had found herself. All of them were in PubMed.

That mattered because Trip currently includes only around 20% of PubMed’s content. These papers were in the other 80%. So the obvious thought was: if we had searched all of PubMed, we would probably have found them. The less obvious part is that this is not a simple fix. PubMed is huge, and pulling all of it directly into Trip would come with real costs and complications.

But feedback often helps you see that the choice is not between doing nothing and doing everything. Sometimes there is a third option.

What we are now planning is this: at the start of the Q&A process, when AskTrip turns a question into search queries for Trip, it will also generate queries designed for PubMed. Both searches will run from the beginning. We will collect results from both, but only use the PubMed set if Trip itself turns up too few relevant papers, or too few papers make it through into the final answer.

That may sound like a technical change, but it should make a practical difference. The current version of Beyond Trip only starts late, after the main process has already run. The new approach prepares for that possibility much earlier. So if we do need to go beyond Trip, we can do it far more quickly.

We still plan to keep Google Scholar and OpenAlex as further fallback options. But they will sit one step later in the chain, only being used if a full PubMed search still leaves us short of useful evidence.

So once again, a piece of user feedback has not just highlighted a weakness. It has helped shape a better system.

That is one of the reasons feedback matters so much. Not because it is always flattering, but because it often shows you exactly where the next improvement needs to be.