The development of a new risk of bias tool for network meta-analysis


Network meta-analysis (NMA) is an extension of pairwise meta-analysis which aims to simultaneously synthesise evidence (of effects) from multiple studies on healthcare interventions of interest.  Our aim is to develop a tool to assess the degree to which the methods lead to a risk of bias in the review conclusions. Our specific objectives are to: (i) conduct a methodological review to generate a list of potential items for inclusion in such a tool; (ii) based on the findings of the methodological review, decide on the structure of the tool; (iii) conduct a Delphi process to refine the tool; and (iv) pilot test the tool, and conduct training.


A steering group of experts in tool development, bias and NMAs was convened. We followed the methods proposed by Whiting (2013) to develop the tool. For the methodological review, we included tools, scientific papers and editorial standards that present items related to bias, reporting, or methodological quality, or articles that assess the methodological quality of reviews with NMA. We searched MEDLINE, the Cochrane library, and difficult to locate/unpublished literature. Once all items were extracted, we combined conceptually similar items, classifying them as referring to bias or to other aspects of quality (e.g. reporting). When relevant, items related to reporting were re-worded into items related to bias in NMA review conclusions, and then re-worded as signalling questions. The steering group reviewed and refined the list of items. Feedback from a larger expert group was obtained via a Delphi survey. Participants were asked to rate whether items should be included. All agreed-upon items, additional or aggregated items, were included in a second round of the Delphi survey. An explanation and elaboration guidance statement was drafted for each item included in the final tool. The tool will be piloted in virtual workshops by training participants in its use. 


For the methodological review of items, we found 3599 citations, and of these, 59 articles were included. Of the 59 included articles, 14 were tools, checklists or standards, 13 were guidance documents for NMAs, 26 were articles related to bias or methods, and six were papers assessing the methodological quality (or risk of bias) of reviews with NMA. These 59 articles yielded 99 items, of which 22 items related to bias in NMAs were included. We recruited 21 participants for the Delphi survey. Delphi experts stated that 18 out of the 22 items should be included in the tool. 


The methodological review of items study provided groundwork for the creation of a bias assessment tool for NMAs. The items were entered into a Delphi process to solicit expert feedback on which items would be included in the tool. Being able to critically appraise the findings of NMAs is central to informed decision-making in patient care. Patients, healthcare providers and policy makers need the highest quality evidence to make decisions about which treatments should be used in healthcare practice.

Carole Lunny poster RoB NMA Tool


Leave a Reply

Your email address will not be published. Required fields are marked *