Sort and Assess Your Search Results

STAK Activity

Sort and Assess Your Search Results

This activity helps students screen and sort sources in a literature search and to document and defend their choices.

INTRODUCTION

When conducting a literature search, students need to be able to scan, systematically sort and include/exclude a range of source. In addition, they must be able to explain how they performed their search, the sorting/selection process they undertook and defend their choices. This activity takes students through a literature search process that requires them to systematic sort sources and shows them how to arrange their findings and document their approach using a step-by-step flowchart.

LEARNING OUTCOMES

On completion of this activity, students will:

Know how to systematically sort literature from a larger search result, using a two-phase process
Be able to defend the sorting and selection decisions taken when surveying literature and findings for a given assignment
Be familiar with digital tools to support the screening/sorting process and to explain this process as an appendix to their assignment.

HOW TO DELIVER THIS ACTIVITY

The activity reviews how students can arrange their findings systematically using a step-by-step screening model. The activity refers to digital tools that can ensure transparency in the selection process and support collaboration in the screening process, and flow charts that can be used to depict / document the sorting process.

BEFORE THE ACTIVITY

Set students an assignment question and ask them to complete a thorough literature search, including searches of the relevant subject’s databases.
Ask students to come to the class with a large volume of possible sources, in the form of a RIS file, if possible, which they have sorted with a view to selecting core documents for their assignment.
Students should define the criteria they have used to decide whether to include or exclude sources for their current assignment. Ask students the question: which parts of the literature do they want to include and exclude, and why?

THE ACTIVITY: STEP BY STEP

Step 1

Using this flowchart as a template, ask students to count and record the number of sources from database searching and from other sources, and provide details of the sources/databases searched.

Students can make use of software to support this process, e.g., Rayyan, which is an open-source program. Records can be imported into Rayyan (e.g., in RIS format) from databases or reference management software.
If several databases have been searched and/or documents have been collected from other sources, unique documents are counted by identifying duplicates and indicating their number in the flowchart.
Rayyan and other reference management software can manage the identification and removal of duplicates.

Step 2

Ask students to screen the sources for relevance to the assignment by reading titles and abstracts/summaries and making exclusions. Students should record the number of sources remaining and excluded and give arguments for the exclusions.

When using Rayyan, students can mark documents as “maybe” so they can discuss them with peers and colleagues and decide whether to include or exclude them later.
At this point, you can specify the concepts that you would like to highlight in the title/abstract for faster sorting. Once Rayyan has 50 manually screened records, it can make a machine rating of the remaining documents which can support (or replace) manual screening.

Step 3

Ask students to review the full text of the remaining sources, e.g., by skimming the method sections and conclusion of the articles (for books reports use the introduction and conclusion). Ask students to complete the flowchart with the number of sources included and excluded and give arguments for exclusions.

If Rayyan is used, this screening round can also be documented by inputting arguments for exclusion in the program.

After the activity

Ask students to submit the flowchart as an appendix to their assignment. Students should also refer to the process they have completed in the method section of their assignment to create transparency around their research methods.

EDUCATOR TO-DO

Familiarise yourself with a digital tool for screening literature, such as Rayyan, and take a look at the flowchart.
Make sure you can export files from different databases and programs.
Set the students the work listed above (under BEFORE THE ACTIVITY) and check that they have completed it before attending class.
Ask students to consider the following question ‘Which parts of the literature do they want to include and exclude?’

MATERIALS REQUIRED

Flowchart with indication of searched sources/databases.
The Rayyan programme.
Reference management software, such as Zotero or Mendeley.
Assessment tools for specific study designs, e.g., qualitative studies. See example.