AI Tools in Evidence Synthesis: Evaluation Tracker
A living registry of evaluations of AI tools used in evidence synthesis. Use it to find tools, compare evaluations, and contribute ongoing research.This project is led by Dr Ciara Keenan
Please direct questions to [email protected]
Find tools
Discover AI tools being used across evidence synthesis workflows.
Compare evaluations
See what evidence exists, who conducted it, and how much we should trust it.
Contribute new research
Share any published or planned evaluations for inclusion in the registry
Why do we need a registry?
Artificial intelligence tools are increasingly being used across evidence synthesis workflows, from literature screening to data extraction and synthesis drafting. However, evaluations of these tools are often difficult to find and inconsistent in quality.
This project was created to bring greater transparency and structure to that landscape by developing a living registry of current and upcoming evaluations of AI tools used in evidence synthesis. The aim is to make evaluations easier to discover, while also supporting a high-level critical appraisal of the evidence behind these tools, including consideration of methodological quality, transparency, and potential commercial or proprietary influences.
Ultimately, the goal is to help everyone who is interested make more informed decisions about the evidence supporting the use of AI in evidence synthesis.
Contribute to the registry
Due to the rapid advancements in the field, this project is intended to be collaborative and community-driven. If you have completed, published, are currently conducting, or are planning an evaluation of an AI tool used in evidence synthesis, I would be very grateful if you could get in touch with a few details.
With your permission, I would like to cite and link to relevant evaluations, protocols, preprints, conference abstracts, and ongoing work so that others can more easily discover and learn from research taking place across the field.
Contributions from researchers, students, practitioners, and organisations are all welcome.
Support MetaEvidenceAI
MetaEvidenceAI is a free community resource that I maintain voluntarily in my spare time for the public good. I’m not looking to make money from it, and my time on the project is unpaid. There are, however, some unavoidable running costs, including the domain name, website hosting, and Airtable database costs. These currently come to around $727 per year.
If you’ve found MetaEvidenceAI useful and would like to leave a small tip, it would be very much appreciated. Any contributions would go directly towards covering the running costs and helping keep the resource available to the community.Thank you for supporting the project.
Purpose
This registry is being developed as a human-led and commercially independent project to identify and summarise evaluations of AI tools used in evidence synthesis. The aim is to make these evaluations easier to find and compare while being transparent about the strength and limitations of the available evidence.
Conflict of interest statement
I recieve no form of funding in relation to this project. I do not have any commercial interest in the AI tools listed in this registry. I have not been involved in conducting any of the evaluations included here, and I have never received funding from the evaluators or tool developers*.*The only exception in regard to tool developers is sysrev, who provided me with a small payment in 2020 to support work gathering evidence on COVID. This payment was made before this repository was needed or established and is declared here for transparency reasons only.
Scope of project
This work is being conducted outside of my normal working hours. As a busy mother of two, I have made some pragmatic decisions about the scope I can handle so that the registry can be built in a useful and sustainable way. These decisions are listed below so that users can understand how the registry has been developed and where its current limits are.Scope of included evaluations
The initial focus is on evaluations published from 2023 onwards. This decision was made because the field is moving quickly, and recent evaluations are likely to be most relevant to current AI tools, models, and evidence synthesis workflows.This date limit is also pragmatic. Earlier evaluations may still be important, particularly for long-standing automation tools used in evidence synthesis. Once the registry has caught up with the current literature, it may be valuable to extend the search backwards, especially if colleagues in the evidence synthesis community are willing to contribute or help verify older evaluations with me.Human-led extraction and appraisal
Key parts of the registry are being completed in a human-led way. This includes identifying evaluations, checking eligibility, extracting core details, and carrying out quality appraisal. In particular, the quality appraisal is led by a human reviewer because it requires judgement about methods, transparency, conflicts of interest, commercial or proprietary involvement, and the reliability of the evaluation.AI is used in some parts of the registry to support efficiency. For example, AI may be used to generate short summaries from human-written notes or structured fields. Where a column has been AI-generated, this is made clear in the column description. AI-generated summaries should therefore be understood as secondary summaries of human-led extraction and appraisal, rather than as independent assessments.Transparency and limitations
This registry is still being compiled. Some fields, including performance metrics, findings summaries, and quality assessment summaries, may remain blank until the underlying evaluations have been reviewed. Where available, users can still click through to the original evaluation, publication, preprint, or project page.The registry is intended to be useful, and iterative rather than complete from the outset. It will be updated as further evaluations are identified, checked, and summarised.
© metaevidence 2026