Module 3 - Using Research to Develop Guidelines

Module 3: Using research to develop guidelines

What is “research” and what is “evidence”?

Since our guidelines are based on medical and scientific research, we thought it might be helpful to explain the basic principles of research. Having an understanding of the research process and the different types of research may make it easier to contribute to the guideline development process.

Research is about looking for new information that could help us make better decisions about treatments or policies. Researchers use different methods like surveys, experiments or interviews to collect and analyse data. They ask questions, gather information, study it and then share their findings.

There are two main types of research:

Qualitative research: This type of research tries to understand why people think or behave in certain ways. It doesn't focus on numbers but explores people's experiences and beliefs. Qualitative researchers might use methods like interviews or focus groups.

Quantitative research: This research collects data in numbers. It might ask questions like how many people get sick each year or whether a new treatment works better than an old one. Quantitative researchers often use methods like surveys or clinical trials.

The results of both types of research are written up and published in scientific journals.

There are four main types of study we use in our guidelines – systematic reviews, clinical trials, observational studies, diagnostic studies and health economic studies.

Systematic reviews

Systematic reviews bring together the results of all the studies that have been carried out around the world in a particular time frame (for example, 2018-2023). These studies will look at a particular research question. The researchers combine the results to give a more complete picture of what the evidence says. Systematic reviews can also tell us about the quality of all the research that has been done.

The vital parts of a systematic review include:

• identifying research papers using clearly defined search methods
• choosing research papers using clearly defined reason for including and excluding information, for example, including studies which only look at people over the age of 18 or excluding studies which look at people with learning disabilities
• assessing research papers against methodological standards

You may hear the term “meta-analysis” when you discuss research papers at SIGN. A meta-analysis is a special type of systematic review that uses statistical methods to combine the results of two or more studies that considered the same research questions in the same way.

Clinical trials

Clinical trials are like tests for new treatments. They involve groups of patients who might get a new treatment, an old treatment or sometimes just a fake treatment called a placebo. This helps scientists see if the new treatment works better than nothing or better than what's already out there.

These trials are used to check if medicines or other healthcare methods are safe and effective. There are different types of clinical trials:

• Randomised controlled trial (RCT): This kind of trial compares two groups of people. One group gets the new treatment, and the other gets either the usual treatment or a fake treatment. The decision about who gets what is random, and the people who run the study don't know who gets what. This helps make sure the results are fair and not influenced by personal beliefs.
• Observational studies: In these studies, researchers don't do anything to the patients. Instead, they watch what happens naturally. Patients might be grouped by things like whether they smoke or if they have a certain condition. There are different types of observational studies:
• Cohort studies: Patients are grouped based on their exposure to something, like smoking and followed for a while.
• Case-control studies: Patients are grouped based on whether they have a certain outcome, like cancer and researchers try to find out what might have caused it.
• Diagnostic studies: These try to find the best way to diagnose a condition. They might compare new tests to ones already in use.
• Health economic studies: These look at how much treatments cost compared to how much they help patients. They help us understand if a treatment is worth the money.

 

How are research studies identified?

There are many international databases of scientific and medical research results. These databases help researchers to search for and bring together studies that may be published in different or unexpected journals. The most widely used medical and scientific databases

• MEDLINE: This is run by the National Library of Medicine in the USA and has a huge list of journals
• Cochrane library: This is managed by the Cochrane Collaboration, a global group that makes systematic reviews of healthcare and looks for evidence from clinical trials and other studies
• Embase: This focuses on drugs and clinical medicine, and has better European coverage than MEDLINE
• CINAHL: This is a nursing and health-related database that covers all aspects of nursing, health education, occupational therapy, social services and other related disciplines
• PsychINFO: This is produced by the American Psychological Association and covers psychology, psychiatry and related subjects

There are two ways to search for research papers in a database:

• Using key words, for example, words in the title or abstract (summary of the paper), authors’ names or where the research was published
• Using medical subject headings; This means looking for papers based on specific topics, like heart disease.

 

How are relevant research papers identified for guidelines?

The guideline development group decides what aspects of the condition the guideline will cover (called the ‘remit’). The group then produces a set of ‘key questions’ about how to manage the condition.

The guideline development group have to be realistic about the number of key questions that can be answered in a single guideline. If the guideline development group set too many key questions, their workload can become too difficult to manage. It is important to limit the guideline scope to those topic areas where there is genuine uncertainty and where implementation of evidence-based recommendations will improve care and reduce variation in practice.

Key questions guide the guideline and are accurately phrased to direct the search and get precise answers.  Key questions are broken down onto the structure below to form the basis of search strategies developed by Information Scientists to identify relevant literature.

 

Population	Intervention	Comparator(s)	Outcomes(s)	Setting(s)
People to which the question applies, for example age group, sex and whether people are at risk of particular conditions.	Intervention being considered, for example treatments involving medications, medical devices, or diagnostic tests	Alternatives being considered, for example current treatment or standard care	What you want to accomplish, accurate diagnosis or relieve or improve symptoms	Care setting such as as primary care, community, acute or emergency settings should be described

 

Our Evidence and Information Scientists use the databases we mentioned earlier to search for papers that are relevant to the guideline. They use the key questions to develop search strategies to search for relevant research papers.

A typical search strategy will identify between 400 and 500 papers. These are presented in the form of abstracts that summarise the paper. 

Before we start to critically appraise the research papers, our Evidence and Information Scientist will take out any papers that are clearly not relevant.  This is known as the first stage of the study selection process (also known as sifting). 

At the second stage of sifting, study abstracts are used to assess if studies are likely to be a potential source of evidence.  At the final selection stage, inclusion and exclusion criteria are applied to the full study reports. These criteria are developed from the key question and allow studies to be excluded based on specified factors such as geographical or healthcare context, study methodologies or numbers of participants. The guideline development group provide expert input to the study selection process and reject any that do not meet the conditions the guideline development group agreed.

 

Reviewing the research papers

Once papers have been chosen as possible sources of evidence, the Evidence and Information Scientists will assess the study methods to see how well the study has been done. This assessment is based on a number of questions in a checklist. The questions vary between the different types of study, and we have designed checklists for each type. These checklists bring a degree of consistency to the appraisal process. The questions focus on the parts of the study’s design that are known to have a significant influence on whether the results and conclusions are valid.

 

What was the research question and why was the study needed?

The introduction to a research paper should give the background to the research and why the research is being done. The research question is the broad question that the research is trying to answer. If you cannot find the research question in the paper, it tends to suggest that the authors did not have a clear aim and that they may not have designed the study very well.

 

How was the study done and was the design appropriate to the question?

Some studies follow patients up over a period of time – these are known as ‘prospective studies.’ Others trace what happened to people in the past and are known as ‘retrospective studies.’ What type of study should have been used depends on the research question. Below are some examples with the research question in bold.

• How many breast cancer patients die each year? This question is best answered by a survey as we are interested in numbers of patients.
• Is cigarette smoking dangerous? This question is best answered by a cohort study where two or more groups are chosen based on how exposed they are to cigarette smoke, and are followed up over a period of time to see what the outcome is.
• Does hormone replacement therapy (HRT) improve bone density? The question we are asking is does it work? This is best answered by a RCT where patients are randomly given either HRT or a placebo. Patients in both groups are followed up for a period of time and specific outcomes are measured such as an improvement in bone density.
• Does living under a power line increase your chance of developing cancer? This is best answered by a case-control study where people with a particular disease or condition are identified and ‘matched’ with controls (patients who live in an area free from power lines). In this case, data would be collected on how exposed people have been to possible causes of cancer in the past.

 

Assessing the quality of the study

• Can we trust all published studies? It is important to remember that just because a research paper has been published in a journal it doesn’t mean that we can trust it. Published studies may still have a number of flaws. This is why it is important that all studies used in our guidelines are critically appraised first.

• Who is the study about? We make sure that the study has included the groups of people we are interested in by asking the following questions:

• How were the people who took part in the study recruited? If you wanted to find out patients’ preferences for a treatment, you could put an advert in the local paper. However, this would introduce selection bias as only the people who were motivated to take part and read papers would do so. It would be better to issue a questionnaire to every service user who visited their GP that day.

• Who was included and excluded in the study? Some trials in the UK exclude patients for example who already have an illness or who do not speak English. This can introduce selection bias. The results of a trial of medicine done in young healthy males may not apply to older females.

• Are the patients in study groups similar? To help limit bias, in all types of studies, (RCT, cohort study or a case-control study), the groups being compared should be as like one another as possible. This can include age, gender, stage of disease and social background as well as other features. Bias is anything which influences the conclusions of a study and affects how the groups in the study are compared.

• Was the assessment blind? Blinding means that the people involved in the study do not know who is getting which treatment.
• if patients knew, they might overestimate how much better they feel
• if investigators knew, they might overestimate the effect of the medicine
• did the study look at statistical questions first?

Understanding statistics is a challenge for most guideline group members. It may help to consider the following two areas when you look at the research:

1. The size of the sample – the trial should be big enough to have a high chance of detecting any statistically worthwhile effect and be sure that no benefit really exists if it is not shown in the trial.
2. How long the study will follow up the people who took part – a study must take place for a long enough period of time for the effect of the treatment to be reflected in the outcomes. If researchers were looking at the effects of a new painkiller used after operations, they may only need a follow up period of 48 hours. If they were looking at how nutritional supplements taken by preschool children affected their final height as adults, the researchers would need to follow up the people who took part for a number of decades.

Once we have asked these questions, we should be able to tell:

• what sort of study it was
• how many people were involved in the study
• where people came from
• what type of treatment was offered
• how long the follow up period was
• what methods were used to measure the outcomes of the study.

 

The Evidence and Information Scientists will rate each study as high quality (with a very low risk of bias), acceptable (with a low risk of bias) or low quality (with a high risk of bias). We can then decide if the paper can be used in the guideline or whether it is not good enough and we should reject it.  The results of this assessment will decide how much evidence is relevant.

 

What if someone has already written a guideline in the same area?

Sometimes good-quality guidelines will have already been written by other agencies. SIGN makes use of other guidelines produced elsewhere for use in NHSScotland. Guidelines that are produced using this approach will refer to these existing guidelines and will try not to repeat work that has already been done. However, before we refer to any existing guidelines, we will make sure they have been developed using acceptable methods. Sometimes existing guidelines may not be directly relevant to patients in Scotland or may have been developed using poor methods.

 

Do the quiz

To complete this module, take a few minutes to do the end-of-module quiz.  It’s a quick way to make sure you’ve learned everything you need.