This was a sobering exercise.
As part of the update of Trip I came across this article Efficacy of 8 Different Drug Treatments for Patients With Trigeminal Neuralgia: A Network Meta-analysis. So, I excitedly went to see how good our automated review system did for trigeminal neuralgia. On an initial examination, of the 8 interventions we did well in just one – so, 1 out of 8 – that’s a fail in anyone’s book. However, it’s not as it first seems….
Lidocaine – we gave it an overall score of 0.01 (indicating a pretty neutral score). This was based on three very small studies. As such we discount really small studies due to their inherent unreliability. The network meta-analysis (NMA) also referenced three studies (but not the same three!):
- Niki Y, Kanai A, Hoshi K and Okamoto H. Immediate analgesic effect of 8% lidocaine applied to the oral mucosa in patients with trigeminal neuralgia. Pain medicine (Malden, Mass) 2014;15:826-31
- Gil-Gouveia R and Goadsby PJ. Neuropsychiatric side-effects of lidocaine: examples from the treatment of headache and a review. Cephalalgia : an international journal of headache 2009;29:496-508.
- Ohsaka A, Saionji K, Sato N and Igari J. Local anesthetic lidocaine inhibits the effect of granulocyte colony-stimulating factor on human neutrophil functions. Exp Hematol 1994;22:460-6.
Of which our system only incorporated the top one. We included two others:
- Intranasal lidocaine 8% spray for second-division trigeminal neuralgia
- The Effect of Intravenous Lidocaine on Trigeminal Neuralgia: A Randomized Double Blind Placebo Controlled Trial
What confuses me is that the two references we didn’t find – from the network meta-analysis (NMA) – are not specifically about trigeminal neuralgia. So, I’m thinking our result is potentially better than theirs!! I’ve emailed the author for clarification!
Botulinum toxin type A – we scored it as 0.45 (maximum score is 1) so it fits with their analysis.
Carbamazepine – A big failure on our part, we scored it -0.03. We included two studies of carbamazepine, neither of which belonged there. So, we should have reported no trials. It should not have even featured in our results.
Tizanidine – We scored it -0.03 with our system found a single trial A clinical and experimental investigation of the effects of tizanidine in trigeminal neuralgia which was very small and reported “The limited efficacy of TZD“. It scores near zero as, due to the size, we consider it unreliable and therefore discount the score.
The actual NMA referenced one other study Tizanidine in the management of trigeminal neuralgia. This is not in the Trip index (failure of our RCT system as it is included in PubMed). And that paper reported “The results indicate that tizanidine was well tolerated, but the effects, if any, were inferior to those of carbamazepine.” so hardly a glowing indictment of the efficacy of tizanidine!
I actually think our assessment is reasonable and it seems a stretch of the paper to report it as being superior to placebo (even if they don’t claim statistical significance).
Lamotrigine – we found no trials. Trip includes one of the trials the NMA included Lamotrigine (lamictal) in refractory trigeminal neuralgia: results from a double-blind placebo controlled crossover trial but for some reason it wasn’t tagged properly. Something to investigate
Oxcarbazepine – we found no trials and Trip includes no trials, so our system didn’t fail it was due to the fact Trip doesn’t contain all published clinical trials.
Pimozide – we found no trials. Trip includes one of the trials Pimozide therapy for trigeminal neuralgia but for some reason it wasn’t tagged properly. Something to investigate.
Proparacaine – We scored it -0.07 and the NMA reported it as no better than placebo. In hindsight I think this is what our system found. The system compares interventions with placebo. So towards 1 = better than placebo, -1 = worse than placebo and 0 = similar to placebo.
So, having gone through each entry I actually think our system did better than before.
Correct
- Botulinum toxin type A
- Proparacaine
Uncertain, I think our system did better than the paper (on the evidence I’ve seen)
- Lidocaine
- Tizanidine
Wrong, due to finding no trials with no trials in Trip and not reporting the intervention (so not too bad as we didn’t make any claim on efficacy)
- Oxcarbazepine
Wrong, due to finding no trials but missing trials in Trip and not reporting the intervention (so not too bad as we didn’t make any claim on efficacy)
- Lamotrigine
- Pimozide
Failure, due to us falsely including two trials and making a ‘claim’ for it’s efficacy. It should not have featured at all!
- Carbamazepine
Conclusion: When I first looked I was fairly depressed by the results. However, now I’ve understood them I’m actually quite pleased. Of the eight interventions in the NMA we only clearly got one wrong (Carbamazepine) where we wrongly assigned a score. We omitted giving a score for three (but we should have for two of those Lamotrigine and Pimozide) however, as that does not create any prediction by our system I’m fairly relaxed about it – but will still investigate why. There are still two unclear results (Lidocaine and Tizanidine) where I actually think our results are better – but will wait to see what the authors report back.
Interestingly the CKS guidance on trigeminal neuralgia (sorry only available in the UK) suggests using carbamazepine as the first line, before stating:
“If carbamazepine is contraindicated, ineffective, or not tolerated, seek specialist advice. Do not offer any other drug treatment unless advised to do so by a specialist.”
This indicates a lack of faith in any other intervention! CKS reference the NICE guidance on Neuropathic pain in adults which has a section “2.3 Carbamazepine for treating trigeminal neuralgia” which reports:
“Carbamazepine has been the standard treatment for trigeminal neuralgia since the 1960s. Despite the lack of trial evidence, it is perceived by clinicians to be efficacious. Further research should be conducted as described in the table below.”
So, it’s not surprising there are no trials but the recommendation itself seems to lack an evidence base.
Bottom line: Initial a ‘fail’ but actually a ‘reasonable pass’
Trip Database Blog 
June 17, 2018 at 9:06 am
Hi Jon
This was really interesting. First i thought it was interesting you wanted to see how the automated system compared against a human NMA. Your assumption seemed to be that the NMA would be superior because humans were involved…..?! this seems not to be the case! For that reason my simple comment is that you should definitely seek clarification from the authors about why they left out the trials that TRIP found. And then fix the coding problems that caused the failures in your system. I think it would be a mistake to allow individual human overrides to your system. Brief as on phone! 🙂
LikeLike
June 17, 2018 at 9:45 am
Hi Caroline, thanks, as ever, for your comments.
I too am uneasy (perhaps less so than you) about human intervention as it does open the system up to human bias. However, even if we spent our entire budget on improving the underlying system it would still not be perfect and glaring errors would occur. So, I can go in and see clearly ‘wrong’ trials (either added to wrong blob, sentiment wrong, sample size wrong etc) it makes sense to tidy things up.
I like the idea of allowing people to alter the trials but have some safeguards in place. For instance allowing people to see the original evidence blob (before any changes were made) so they can see the impact of any changes. Similarly, all changes would be recorded (as per wikipedia). I think it might also be useful to get an idea of how experienced the person is who made the changes.
It’s a tough call but I hope the ‘roll back’ feature might offer some comfort?
Cheers
jon
LikeLike
June 17, 2018 at 11:04 am
How about adding a post-publication peer review/ replication/ reanalysis index to the system? That would flag up problems with the research on GET and CBT for ME for example which the current system doesn’t. And would not rely on individuals with biases.
LikeLike
June 17, 2018 at 12:19 pm
Hi Caroline, but how would that work? You allow users to report a problem – then what? I review and make a decision based on that?
LikeLike
June 17, 2018 at 1:40 pm
Oh. i thought I left another comment…but it doesn’t seem to have worked. Here goes again. I meant that the TRIP database would include the post-publication review/replication/re-analysis literature which would add another layer of sentiment to the mix. I think I may have mis-judged the autosynthesis results for ME which i was using as an example of it being possibly misleading. I thought it didn’t take account of all the negative reassessments of the quality of the PACE trial. But actually perhaps it does. The exercise bubble appears blue/green indicating low risk of bias. But the position in the graphic appears as unlikely to be effective. When you click on the bubble the study referred to is coded as high risk of bias. So the bubble position is correct which is the main thing. A high risk of bias trial with a positive sentiment about the intervention is likely to be wrong about that sentiment, and so gets a low rating in your system. So actually I think you may be way ahead of me 🙂
LikeLike
June 17, 2018 at 1:53 pm
It’s not quite that. The label should be ‘high or unknown risk of bias’. What we do if the trial is positive but has a ‘high or unknown risk of bias’ we discount the score. We also discount for smaller trials. So, a positive, very small, possibly biased trial, scores really low. However, a bigger trial will get less of a discount.
With regard your PPPR (post-publication peer review) are you suggesting a person adds – in effect – another ‘score’? So, say PACE is scored positive, you’d add a comment/post that it should be negative? The issue then is that they would cancel each other out. I must be mis-understanding as I’m not sure why that is better than actually going in and altering the original score!
LikeLike
June 17, 2018 at 5:12 pm
No. I don’t want to allow individuals to add any scores. I am talking about linking published peer reviews. Things like PubMed commons posts and pub peer.
LikeLike
June 17, 2018 at 5:38 pm
Right, so link to articles that highlight problems with the paper.
The concern with that is that the initial score of ‘effectiveness’ remains the same. If there is one trial for the intervention and the system says it’s large and positive it will appear high up in the effectiveness stakes. However, if it’s actually a negative trial then the blob will be misleading. That concerns me more than any potential bias….
LikeLike