Cookbook Medicine: Poor Quality of Evidence Behind Guidelines

You're only as strong as your weakest link.  In the world of evidence-based medicine, there are a number of ways to evaluate the strength of evidence.  Since May 2007, the United States Preventive Services Task Force (USPSTF), the most conservative body regarding the promulgation of evidence-based guidelines, has used levels of certainty to evaluate & grade the evidence behind their guidelines.  

For instance, recommendations for which they declare a high level of certainty are those such that "[T]he available evidence usually includes consistent results from well-designed, well-conducted studies in representative primary care populations. These studies assess the effects of the preventive service on health outcomes. This conclusion is therefore unlikely to be strongly affected by the results of future studies."  

Those recommendations for which USPSTF declares moderate level of certainty are those such that "[T]he available evidence is sufficient to determine the effects of the preventive service on health outcomes, but confidence in the estimate is constrained by such factors as the number, size, or quality of individual studies; inconsistency of findings across individual studies; limited generalizability of findings to routine primary care practice; and/or lack of coherence in the chain of evidence."

The American Academy of Family Physicians (AAFP) uses the ABC rating scale, where Level A (randomized controlled trial/meta-analysis) includes high-quality randomized controlled trials (RCT) that consider all important outcomes or high-quality meta-analyses (quantitative systematic review) using comprehensive search strategies.  Level B (other evidence) includes well-designed, nonrandomized clinical trials; nonquantitative systematic reviews with appropriate search strategies and well-substantiated conclusions; lower quality RCTs, clinical cohort studies and case-controlled studies with nonbiased selection of study participants and consistent findings; and other evidence, such as high-quality, historical, uncontrolled studies, or well-designed epidemiological studies with compelling findings.  Level C (consensus/expert opinion) is consensus viewpoint or expert opinion.

 Strength of Recommendation Taxonomy (SORT) grades the evidence as Level 1 (consistent, good quality patient-oriented evidence), Level 2 (inconsistent or limited quality patient-oriented evidence), or Level 3 (other evidence).  Note the focus on patient-oriented evidence.  For instance, patients don't really care that a medicine lowers their blood pressure, they just want to be sure they don't have a stroke.

I mention the above because when one considers the multitude of guidelines foisted anonymously upon us especially in our electronic health records (EHR), it would be nice know that the highest quality of evidence supports the recommendations.  However, as the authors in a study published this month noted, over half the recommended guidelines promulgated by the Infectious Disease Society of American were only Level III, based upon expert opinion only as opposed to the hoped for Level I based upon >1 randomized controlled trial.

The problem with all these guidelines is a temptation by many to view the practice of medicine as akin to following a cookbook, when indeed there is quite a bit of art even to interpreting & applying the science of medicine on an individual basis.  So lest we fall lockstep into performing every recommended guideline in our EHR, we still need to read the latest literature ourselves rather than have someone pre-digest & regurgitate it for us.  In fact, we should consider the source of said guidelines when choosing our federally-mandated EHR.

Disclosure:  I'm scheduled to give a presentation entitled "The EHR Challenge: How to Choose the Best Fit" tomorrow at our Nevada Academy of Family Physicians' 43rd Annual Winter Conference here in lovely South Lake Tahoe where the weather has been sunny and the skiing awesome.