Amazingly, in around a fortnight, we have created a rudimentary ACQA system and the initial results are both good and bad! We’ve run through a number of questions and some have been pretty good and others pretty bad. An example of a good ACQA was Is brexpiprazole effective in schizophrenia? with our system generating the following answer:

“Brexpiprazole has shown effectiveness in treating schizophrenia based on a post hoc analysis of various short- and long-term studies. These studies demonstrated that brexpiprazole led to clinically relevant improvements in functioning for patients with schizophrenia in both short and long durations. Patients on brexpiprazole experienced greater improvements in Personal and Social Performance (PSP) scores and all PSP domains compared to those on placebo. The Global Assessment of Functioning (GAF) scale also indicated better functional remission for brexpiprazole-treated patients. These improvements were noticeable during the treatment of acute schizophrenia and generally persisted over a one-year period. However, low-dose brexpiprazole might not be as effective as standard-dose brexpiprazole in improving efficacy and safety for acute schizophrenia, potentially carrying a risk of weight gain. Additionally, a study highlighted that brexpiprazole treatment reduced impulsive behavior and associated brain activation in patients with schizophrenia. Overall, brexpiprazole appears to be effective in managing symptoms and improving functioning in schizophrenia.“

I’m not saying it’s perfect but pretty good for a first attempt. This answer was all referenced and mostly created from secondary evidence. However, a worse answer was generated for Is zolmitriptan effective in migraines?, which generated this answer:

“Zolmitriptan, a medication commonly used for migraines, belongs to the triptan class of drugs. It works by constricting blood vessels in the brain and reducing inflammation. Studies have shown its effectiveness in relieving migraine symptoms such as headache, nausea, and sensitivity to light and sound. It can provide relief within two hours of taking it. However, its effectiveness may vary and it may have side effects. Consultation with a healthcare professional is important. The provided text does not directly address the effectiveness of zolmitriptan in migraines.“

But understanding the process and why some answers are so poor have given us a load of learning, some easy to respond to but others less so. Currently, the biggest struggle is transforming the question into appropriate search terms. We’ll continue to use LLMs to try to solve this but an easier route might be to include the users more and make it semi-automatic. In many way I favour this, it gives users more understanding of the process and it’s less ‘black box’. It might add an extra step but the extra reassurance might be worth it.

Early days and all to play for!