The basic evidence based research steps are:
- Begin with a problem or issue stated as a question. 1 2
- Plan your search for the levels of evidence you need. 3
- Search for secondary or primary sources and primary evidence. 4 5
- Evaluate your results. 6 7
- Implement your findings. 8 9
- Publish your findings to build the evidence base. 10
About the tool
After study examples are located - are they what you need?
1. Identify one or more studies that resemble your problem/issue.
2. Collect data from the relevant studies - use a form to organize the data collected from the studies. Information needed: description of study, how it matches your criteria, quality & results (5)
3. Analyze & critique the study design, context, results and relevance to your need.
Questions to critique qualitative or quantitative research: (6)
- Are findings valid: Is the research question clear & substantiated? Is the design appropriate? Is the sample method, size & reasoning appropriate? Was data collected & analyzed systematically?
- Are the findings thoroughly described?
- Are findings applicable to your question? Relevant? Useful for your context? Enhance your knowledge? Transferable?
Critical appraisal checklist to evaluate educational and training interventions (7)
Validity of study design
- Objective clearly stated?
- Description of participants detailed?
- Are groups similar in characteristics treated equally?
- Is research method clear? Appropriate? Avoiding bias (random, blind)?
- Are outcomes measured short-term, intermediate or long-term?
- Is research instrument described in detail? Validated? What questions were asked? What level of learning does the study address?
Education context
- What is learning environment type? (university, college, school, etc.)
- What teaching method is used? Is philosophy or instructional theory outlined?
- What is mode of delivery?
- Is what was taught clearly described?
- What outcomes were measured? (cognitive, affective, etc.)
Reliability of results
- Clearly explained?
- Address original question?
- Is data presented accurate and clear?
- Were statistical significance tests reported?
- Was outcome positive or negative for the intervention?
- Does data support the author's conclusions?
- Are potential problems with the research design presented?
Relevance of best available study
- Is the study population similar to mine?
- What competencies does the study address? Are they the same as mine?
- Are the practice implications reported?
- Can results be directly transferred to my situation? What aspects can inform my practice?
Articles on critically assessing information by Trisha Greenhalgh
(BMJ series from the book: How to read a paper: the basics of evidence based medicine)
How to read a paper: The Medline database.
Key points:
- Search Medline by subject, title of article, author, journal, etc.
- Do a focused search by combining keywords or phrases
- Refine the search by using limiters
- Scan titles for the most relevant
- Use other library databases if you don't find what you need
Click on pdf file for complete article - BMJ 1997; 315(7101): 180-3.
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127107&blobtype=pdf
Getting your bearings (deciding what the paper is about).
Key points:
- What question does the paper address?
- Critically appraise the methods used
- Primary study reports experiments, clinical trials and surveys
- Secondary research is overview (systematic or meta-analysis), guidelines, decision analyses and economic analyses
- Randomized controlled trials answer drug treatment or medical interventions and have advantages/disadvantages
- Prognosis or causation questions require cohort or case-control studies
- Case reports, though week evidence, can quickly alert practitioners on treatments or conditions
Click on pdf file for complete article - BMJ 1997; 315(7102): 243-246. http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127173&blobtype=pdf
Assessing the methodological quality of published papers.
Key points:
- Does this research add to the literature?
- Who was included / excluded?
- What intervention was compared to what and how was outcome measured?
- Did study design avoid bias in selection/exclusion performance and detection?
- What is the sample size?
- How long was the study?
Click on pdf file for complete article - BMJ 1997; 315(7103): 305-308. [see page 305]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127212&blobtype=pdf
Statistics for the non-statistician: Different types of data need different statistical tests.
Key points:
- Were the appropriate standard statistical tests used according to the study's protocol?
- Were groups analyzed for their baseline comparability?
- Does the test reflect the type of data analyzed (paired/unpaired, parametric/non-parametric)?
- If an intervention's effect may be negative was a two-tailed test used?
Click on pdf file for complete article - BMJ 1997; 315(7104): 364-366. [see page 364]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127256&blobtype=pdf
Statistics for the non-statistician. II: "Significant" relations and their pitfalls.
Key points:
- Were correlation and regression used correctly?
- Tests for causation on association - p.423
- Calculating the "bottom line effects on an intervention - p.424
- 10 ways to cheat on statistical tests when writing up results - p.425
Click on pdf file for complete article - BMJ 1997; 315(7105): 422-425. [see page 422]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127270&blobtype=pdf
Papers that report drug trials.
Key points:
- Is the value of a drug's safety, tolerability, efficacy and price expressed?
- Drug efficacy should be measured in clinical end points relevant to patients
- Features of the ideal surrogate end point - p.481
- Checklist for evaluating information provided by a drug company - p.482
Click on pdf file for complete article - BMJ 1997; 315(7106): 480-483. [see page 480]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127321&blobtype=pdf
Papers that report diagnostic or screening test.
Key points:
- Is the test validated by comparison against a gold standard in an appropriate spectrum of objects?
- A test is valid if it detects most people with the target disorder (high sensitivity) and excludes most people without the disorder (high specificity) and if a positive test usually indicates that the disorder is present (high positive predictive value).
- The likelihood ratio is the best measure of the usefulness of a test.
Click on pdf file for complete article - BMJ 1997; 315(7107): 540-543.
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127365&blobtype=pdf
Papers that tell you what things cost (economic analyses).
Key points:
- In addition to checking the math of an economic analysis, have direct/indirect and intangible costs been included?
- Most instruments for measuring health state and quality of life are not objective.
- 10 questions to ask about an economic analysis - p.597-599
Click on pdf file for complete article - BMJ 1997; 315(7108): 596-599. [see page 596]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127419&blobtype=pdf
Papers that summarize other papers (systematic reviews and meta-analyses).
Key points:
- Methodology for a systematic review of randomized controlled trials - chart p.672
- Advantages of systematic reviews - p.673
- Was question precisely defined and important sources thoroughly searched?
- Each trial is evaluated by bias of method, precision and applicable validity or relevance of the trials included.
- Meta-analysis plots numerous primary studies checking the precision and validity of their methods.
Click on pdf file for complete article - BMJ 1997; 315(7109): 672-675. [see page 672]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127461&blobtype=pdf
Papers that go beyond numbers (qualitative research).
Key points:
- A good qualitative study addresses a problem with a clear questions and uses more than one research method.
- Qualitative research methods include documents, observation (passive/participant), interviews and focus groups.
- What methods were used to collect and analyze the data?
- Is the conclusion explicit and reproducible?
- What quality control measures were used?
Click on pdf file for complete article - BMJ 1997; 315(7110): 740-743. [see page 740]
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=2127518&blobtype=pdf
Always be alert for bias.
Use critical appraisal to set aside your own personal opinion and look for bias in others.
How to recognize bias?
Cochrane Bias Methods www.ohri.ca/bmg/types_bias.html#definitions