SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing

frontiers in Research Metrics and Analytics

SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing

Research output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these situations. However, they are protracted processes that require expertise to execute. These are problematic characteristics in a constantly changing environment. To solve these challenges, we introduce an innovative systematic review methodology: SYMBALS. SYMBALS blends the traditional method of backward snowballing with the machine learning method of active learning. We applied our methodology in a case study, demonstrating its ability to swiftly yield broad research coverage. We proved the validity of our method using a replication study, where SYMBALS was shown to accelerate title and abstract screening by a factor of six. Additionally, four benchmarking experiments demonstrated the ability of our methodology to outperform the state-of-the-art systematic review methodology FAST two.

One INTRODUCTION

Both the number of publishing scientists and the number of publications are constantly growing. The natural scientific tool to provide clarity in these situations is the systematic review, which has spread from its origins in medicine to become prevalent in a wide number of research areas. Systematic reviews offer a structured and clear path to work from a body of research to an understanding of its findings and implications. Systematic reviews are ubiquitous in today's research. A search in the Scopus abstract database for the phrase "systematic review" yields more than forty-five thousand results for the year twenty twenty alone.

Nevertheless, systematic reviews have shortcomings. They are particularly protracted processes, that often require an impractical level of expertise to execute. These issues have been recognised for decades, but not solved. This hampers our ability as researchers to apply this potent tool in times where change is ceaseless and sweeping.

However, with recent advances in machine learning and active learning, new avenues for systematic review methodologies have appeared. This is not to say that these techniques make traditional systematic review techniques obsolete. Methodologies employing automation techniques based on machine learning are often found to omit around five percent of relevant papers. Additionally, usability and accessibility of automation tools is a common issue and many researchers do not trust machine learning methods enough to fully rely on them for systematic reviews.

Therefore, in this paper, we argue for the combination of the proven method of backward snowballing with novel additions based on machine learning techniques. This yields SYMBALS: a SYstematic review Methodology Blending Active Learning and Snowballing. The challenges faced by systematic review methodologies motivate the research question of our paper:

How can active learning and snowballing be combined to create an accessible and swift systematic review methodology?

The remainder of this paper is structured as follows. In Section Two, we cover related work on systematic review methodologies and active learning techniques for systematic reviews. In Section Three, we introduce SYMBALS, our innovative systematic review methodology. We explain each step of the methodology in detail. Section Four evaluates and demonstrates the effectiveness of our methodology using two case studies: a full application of SYMBALS Four point one and a benchmarking study Four point two. In Section Five, we discuss the implications of the case studies and the limitations of our research. Finally, we conclude and present ideas for future research in Section Six.