Introduced over ten years ago, the PICO search is very popular with our users. However, it is frequently misunderstood. Yesterday we received an email from a user:
“We are finding different results between advanced search and PICO search. We are using the same terms. Is this possible? Is there any way to solve it?“
So, what’s going on? PICO search is not designed to be an exhaustive search, it is designed to find a small number of highly relevant results. Or, put it another way, it’s a very specific search not a sensitive one.
At the heart of the PICO search is something called contingency searching, which helps us deliver a specific search. After the user enters their search term our first search is for all the PICO elements as title only searches. If there are too few results we then make the final search term entered a ‘title and text’ search and repeat the search and if that too has too few results we make the penultimate term a ‘title and text’ and we repeat that until we get a manageable number of results. All these repeated searches are done in the background; from a user’s perspective it’s a single search.
So, if you want lots of results use the default or advanced search. However, if you want a more focussed set of results use PICO (although the default search is pretty good as well)!
As part of the improvements to Trip we have had a special focus on systematic reviews (SRs), as these are a key element of EBM (and many recent posts on this blog relate to SRs). As it stands we are just shy of 500,000 (this figure includes a number of Health Technology Assessments). In addition to published SRs we also link to over 150,000 registered SRs – those planned and ongoing.
However, an important consideration of ours is that users may see a systematic review (at the top of the EBM pyramid) and suspend scepticism. So, we’re still working hard on a quality score for SRs. This is going well and I’d like to think it’ll make an appearance before the end of the year.
So, size is important but size with quality is even better.
One of Trip’s virtues is being easy to use. Behind this ease is a hugely complex website, one that took over two years to re-code. As well as re-coding the website there were other problems that affected the quality of the site and we’re addressing these in a systematic manner. As we tick these off the ‘to do’ list, the site get stronger. Some recent work includes:
Systematic reviews (SRs)
We automatically grab new SRs from PubMed (and we recently improved the filter for identifying new SRs). However, we also obtain SRs from other sources and we have finished automating this process. So, now, every week we grab a whole batch of new SRs. Previously the ‘other sources’ was a manual process and was not regularly undertaken.
Ongoing clinical trials
Another system that was previously semi-manual and not undertaken regularly, was the import of ongoing clinical trials from clinicaltrials.gov. This is now automated with new trials added weekly.
The user interface – to alert us to a broken link – works well but the Trip process to fix these links was sub-optimal. We have now reworked that and it works really well. Less broken links = increased quality.
Our ‘to do’ list has been full of things like this – not major individual pieces of work but important aspects that might affect the performance of Trip. Each one ticked off means the less likely users are to have a poor experience – surely a good marker of quality.
As mentioned previously we are trying to identify as many systematic reviews as possible to include in Trip. One method is to use third party services that capture academic publications from a variety of sources. We, in turn, try to identify systematic reviews from these service and add them to Trip. This works well – generally – but two issues have arisen that we had not anticipated.
Predatory journals: I received an email yesterday which started “I noticed on my most recent search that a predatory journal made it into into your search result“. It transpired that the article in question was a systematic review. In other words the 3rd party scraped the article from the web and we grabbed it ‘blindly’. This is clearly problematic and we’ll take steps to stop this happening in the future.
Pre-prints: This is less clear cut – so would welcome input. In the new system I’m seeing systematic reviews from places such as medRxiv. On one hand these are potentially problematic as they haven’t been through peer review. But on the other hand they are clearly labelled as not having gone through peer review and, also, they may well be good quality and contain valuable information that might not be seen for months (due to the slow peer review process).
It’d be interesting and useful to hear your thoughts on the above! So, please leave a comment or email me directly email@example.com
This post is an attempt to ‘think aloud’ about connected/related articles…. By that I mean, if you find an article you like how can you quickly find others that are similar. We know that searching is imprecise and a user might find articles that match their intention at say result #2, #7, #12 and then they may lose interest and miss ones at #54 or #97.
At Trip we have had something called SmartSearch for years. This mines the Trip weblogs to highlight articles that have been co-clicked in the same search session. So, if a user clicks on articles #2, #7 and #12 we infer a connection. We have successfully mapped these connections and it reveals a structure in the data. In the example below it’s a small sample of connections taken from searches for urinary tract infections:
Each blue square represents a document and the lines/edges are connections made by co-clicking the documents within the same search session. You can see from the annotation that these form clusters around topics. However, co-clicking is not perfect!
Fortunately, there are other types of connections that I think we can use – semantic similarity and citations.
Semantic similarity: I’m thinking principally of PubMed’s related articles. This uses statistical methods to find articles with similar textual information e.g.
At the top is the document of interest and below are the articles deemed semantically similar. So, these articles are all related and one could make connections between them.
Citations: Articles typically list a bunch of references – so the article is citing these. And any article can itself be cited. So, you have forward and backward citations. Again, these have been shown as connections and mapped, e.g (source):
So, three types of connections: co-click, semantic similarity and citations. In isolation all have their issues but combined it could be something incredibly powerful. Well, that’s the theory….
While I believe SmartSearch is brilliant, I don’t think we’ve implemented it particularly well. The main issue I have is that a user needs to ‘call’ the results. On one hand that’s not a big deal but it looks like this:
I’ve highlighted it in red so you don’t miss it (an important issue in itself) but also it’s not really telling the user why they should click. In other words, it has a weak ‘call to action’. In part this is because it’s not ‘real time’ – a user clicks a button and the system calculates the related articles. I’m thinking if we told users that there were, say, 25 closely connected articles and 7 very closely connected articles, possibly teasing what these were, it would be much more compelling.
Another consideration, the notion of connected articles can work on two levels: the individual article and a collection of articles.
Individual articles: Each article within Trip could feature other connected articles be it co-clicks, semantic similarity or citations. It could be that we create a badge (thinking of the Altmetric Donut) that helps indicate to users how many connections there might be.
Collection of articles: If a user clicks on more than one article we, in effect, add up the information from the individual article data. This allows for some clever weightings to be brought in to highlight particularly important/closely connected articles.
But what information is important/useful to the user? I’m seeing two types of display:
List: A list of articles, arranged by some weighting to reflect ‘closeness’ – so those at the top are deemed closer to the article(s) chosen. We could enhance that by indication which are systematic reviews, guidelines etc
Chronological list: As above but arranged by date. The article(s) chosen would be shown and then a user could easily see more recent connected papers and also more historical papers. The former being particularly useful for updating reviews!
Right, those are my thoughts, for now. They seem doable and coherent but am I missing something? Could this approach be made more useful? If you have any thoughts please let me know either in the comments or via email: firstname.lastname@example.org
One excellent bit of feedback is to add connections between clinical trial registries and subsequent studies. This should be feasible. Similarly, link PROSPERO records (register of ongoing systematic reviews).
We’re keen to help users use the best quality evidence to inform their decisions. While we use the pyramid to help express the hierarchy of evidence there is a danger of that being too simplistic. For instance, not all systematic reviews are high-quality and some are, frankly, terrible.
We have been working on quality scores for RCTs and guidelines for some time and these should both be released by early 2023. However, of equal importance, is scoring systematic reviews. Given Trip covers hundreds of thousands of systematic reviews, any tool we introduce needs to be automated. Well, we’ve taken the first tentative steps…
We have devised a scoring system, capable of automation, and trialled this on a sample of 32 systematic reviews. We knew the assessments of the 32 prior to starting and the scoring was done by a 3rd party and was freely available data on the web (using ROBIS the scores were low, high or unclear risk of bias). We then correlated our scores against the ROBIS scores and this is what the graph looks like:
The Y-axis is our score (range from -3 to 8) and the x-axis is simply the number of the systematic review (so 15 were graded as low risk of bias, 9 as high risk of bias and 8 as unclear).
For a first attempt the results are impressive and shows the validity of the approach. The average score per risk of bias category is as follows:
Low – 5.3
Unclear – 3.75
High – 0.78
We clearly need to spend more time on this trying to understand why, for instance, the 3rd ‘low risk of bias’ systematic review scored so low in our system. But there’s time for that, time to adjust weighting, possibly add or remove scoring elements.
Bottom line: we’re well on the way to rolling out an automated systematic review scoring system that can help Trip users make better use of the evidence we cover
After our recent post on the subject I thought I’d explore the new systematic reviews added to Trip. So, for the last week we uploaded 829 new systematic reviews from PubMed. To give a flavour of the coverage, here are the sample of the most recent:
The role of noninvasive scoring systems for predicting cardiovascular disease risk in patients with nonalcoholic fatty liver disease: a systematic review and meta-analysis.
A systematic review on microplastic pollution in water, sediments, and organisms from 50 coastal lagoons across the globe.
The effects of exposure to environmentally relevant PFAS concentrations for aquatic organisms at different consumer trophic levels: Systematic review and meta-analyses.
Provisional Versus Dual Stenting of Left Main Coronary Artery Bifurcation Lesions (from a Comprehensive Meta-Analysis).
The Impact of Cognitive Impairment on Clinical Outcomes After Transcatheter Aortic Valve Implantation (from a Systematic Review and Meta-Analysis).
A meta-analysis of the genetic contribution estimates to major indicators for ketosis in dairy cows.
Heterojunction photocatalysts for the removal of nitrophenol: A systematic review.
The effect of rhythmic movement on physical and cognitive functions among cognitively healthy older adults: A systematic review and meta-analysis.
Effectiveness of multicomponent training on physical performance in older adults: A systematic review and meta-analysis.
Molecular mechanism of the anti-inflammatory effects of plant essential oils: A systematic review.
An interesting mix, that’s for sure, and we should possibly explore removing non-human studies!
The above is a sample from PubMed, we also get systematic reviews from other sources:
Grey literature, which we explore on a manual and monthly basis – this includes a host of Health Technology Assessments
The latter is not yet automated, but will be shortly. So, it wouldn’t surprise me if we don’t add 1,000+ systematic reviews to Trip every week!
There’s an awful lot of systematic reviews being carried out!
The move to a new, stable system, has allowed us to start really improving the quality of Trip. Trip is a hugely valuable tool, but it isn’t perfect and the old system was creaking.
One immediate area for attention has been the way we grab systematic reviews. We have three main ways of adding systematic reviews to Trip:
A number of publishers are considered producers of systematic reviews and their content is not routinely added to PubMed – so we manually grab those records.
PubMed – we use a filter to identify systematic reviews
Others – we try to identify systematic reviews from a small number of third-party sources
The middle one, PubMed filter, is a complex area to navigate given the tension between sensitivity and specificity. Too sensitive (to identify ALL systematic reviews) and you bring in a load of false positives. Too specific (to only identify TRUE systematic reviews) and you miss a load of systematic reviews – false negatives.
So, we’ve been carrying out a lot of tests on PubMed and have plumped for this filter:
(systematic review[sb] OR meta analy*[TI] OR metaanaly*[TI] OR “Meta-Analysis”[PT] OR “Systematic Review”[PT] OR “Systematic Reviews as Topic”[MeSH] OR “systematic review” [TI] OR “health technology assessment” [TI] OR “Technology Assessment, Biomedical”[Mesh])
At the time of writing the above search identifies 372,212 results (click here to try it yourself). We estimate the other sources contribute an additional 80-100,000 systematic reviews. So, we’re on our way to half a million!
The new PubMed filter will also be checked much more regularly than previously and the third option (third-party sources) are next – again improved filter and more regular checking.
Systematic reviews are hugely important in the EBM world and therefore we’re delighted with progress and we hope our users will be too.
Our new index was seemingly all fine, but, we now know that we have a problem!! The problem lies with how we categorise articles as systematic reviews. So, you will find that many articles are being incorrectly labelled as systematic reviews. Apologies for this and we’re racing to fix this.