I have a major concern about their applicability given the extreme heterogeneity reported.
The I² values for the three analyses, over 96% for CT reading times, 99% for colonoscopy, and 84% for turnaround times, are exceptionally high. In plain terms, this isn’t just a little variation; it tells us the studies being combined are fundamentally different from each other in their design, execution, or context. They are comparing apples, oranges, and a few wrenches.
The core idea of a meta-analysis is to pool similar studies to get a clearer answer. When heterogeneity is this extreme, that pooled result becomes statistically questionable and, frankly, difficult to trust or interpret. The conclusion that “no significant effects were found” might not be a true reflection of AI’s impact, but rather an artifact of mashing together incomparable data.
This issue is compounded by the fact that most of the underlying studies were judged to have a serious or critical risk of bias. So, we are not only mixing very different studies, but many of them have significant methodological weaknesses to begin with.
Given this context, performing a meta-analysis might be considered a critical misstep. Presenting a single, pooled number under these conditions can be misleading. A more honest and rigorous approach would have been to acknowledge the fragmentation upfront and rely on a detailed narrative synthesis to describe the landscape of findings, rather than forcing a statistical combination that the data simply doesn’t support. This concern, unfortunately, casts significant doubt on the paper’s main quantitative conclusions about AI’s effect on efficiency.