Home Posts tagged "Examine Research Quarterly"

How to Read a Study

Today's guest post comes from the bright minds at Examine.com. I love their stuff, and if you want unbiased nutrition research you can trust, I’m sure you will too.

Because they are research experts I trust, I asked their team if they could help educate everyone on how to become more adept at reading and discerning published research. -EC

If you have ever had the pleasure (displeasure?) of reading through a scientific study, your eyes may have been attacked with confusing jargon such as “confidence interval”, “P-value”, and “subgroup analysis”.

Confused yet? In this post, we will give you the 101 on how to approach, question, and interpret a scientific study.

Why should I learn to read a study?

To avoid wasting money on ineffective products (like some supplements) or interventions (such as a particular training method), you need to be able to assess different aspects of a study, such as its credibility, its applicability, and the clinical relevance of the effects reported.

To understand a study, as well as how it relates to other available research on the topic, you need to read more than just the abstract. Context is critically important when discussing new research, which is why abstracts are often misleading.

A paper is divided into sections. Those sections vary between papers, but they usually include the following.

  • Abstract
  • Introduction
  • Methods
  • Results
  • Discussion
  • Conflicts of Interest

We’re going to walk you through each of these sections and give you pointers on what to look out for.

Abstract

The abstract is a brief summary that covers the main points of a study. Since there’s a lot of information to pack into a few paragraphs, an abstract can be unintentionally misleading.

Because it does not provide context, an abstract does not often make clear the limitations of an experiment or how applicable the results are to the real world. Before citing a study as evidence in a discussion, make sure to read the whole paper, because it might turn out to be weak evidence.

Introduction

The introduction sets the stage. It should clearly identify the research question the authors hope to answer with their study. Here, the authors usually summarize previous related research and explain why they decided to investigate further.

For example, the non-caloric sweetener stevia showed promise as a way to help improve blood sugar control, particularly in diabetics. So researchers set out to conduct larger, more rigorous trials to determine if stevia could be an effective treatment for diabetes. Introductions are often a great place to find additional reading material since the authors will frequently reference previous, relevant, published studies.

Methods

A paper’s “Methods” (or “Materials and Methods”) section provides information on the study’s design and participants. Ideally, it should be so clear and detailed that other researchers can repeat the study without needing to contact the authors. You will need to examine this section to determine the study’s strengths and limitations, which both affect how the study’s results should be interpreted.

A methods section will contain a few key pieces of information that you should pay attention to.

Demographics: information on the participants, such as age, sex, lifestyle, health status, and method of recruitment. This information will help you decide how relevant the study is to you, your loved ones, or your clients.

Confounders: the demographic information will usually mention if people were excluded from the study, and if so, for what reason. Most often, the reason is the existence of a confounder — a variable that would confound the results (i.e., it would really mess them up).

Design: Design variants include single-blind trials, in which only the participants don’t know if they’re receiving a placebo; observational studies, in which researchers only observe a demographic and take measurements; and many more. This is where you will learn about the length of the study, intervention used (supplement, exercise routine, etc.), the testing methods, and so on.

Endpoints: The “Methods” section can also make clear the endpoints the researchers will be looking at. For instance, a study on the effects of a resistance training program could use muscle mass as its primary endpoint (its main criterion to judge the outcome of the study) and fat mass, strength performance, and testosterone levels as secondary endpoints.

Statistics: Finally, the methods section usually concludes with a hearty statistics discussion. Determining whether an appropriate statistical analysis was used for a given trial is an entire field of study, so we suggest you don’t sweat the details; try to focus on the big picture.

Statistics: The Big Picture

First, let’s clear up two common misunderstandings. You may have read that an effect was significant, only to later discover that it was very small. Similarly, you may have read that no effect was found, yet when you read the paper you found that the intervention group had lost more weight than the placebo group. What gives?

The problem is simple: those quirky scientists don’t speak like normal people do.

For scientists, significant doesn’t mean important — it means statistically significant. An effect is significant if the data collected over the course of the trial would be unlikely if there really was no effect.

Therefore, an effect can be significant (yet very small) — 0.2 kg (0.5 lb) of weight loss over a year, for instance. More to the point, an effect can be significant yet not clinically relevant (meaning that it has no discernible effect on your health).

Relatedly, for scientists, no effect usually means no statistically significant effect. That’s why you may review the measurements collected over the course of a trial and notice an increase or a decrease yet read in the conclusion that no changes (or no effects) were found.

There were changes, but they weren’t significant. In other words, there were changes, but so small that they may be due to random fluctuations (they may also be due to an actual effect; we can’t know for sure).

P-Values

Understanding how to interpret P-values correctly can be tricky, even for specialists, but here’s an intuitive way to think about them.

Think about a coin toss. Flip a coin 100 times and you will get roughly a 50/50 split of heads and tails. Not terribly surprising. But what if you flip this coin 100 times and get heads every time? Now that’s surprising!

You can think of P-values in terms of getting all heads when flipping a coin.

A P-value of 5% (p = 0.05) is no more surprising than getting all heads on 4 coin tosses.
A P-value of 0.5% (p = 0.005) is no more surprising than getting all heads on 8 coin tosses.
A P-value of 0.05% (p = 0.0005) is no more surprising than getting all heads on 11 coin tosses.

A result is said to be “statistically significant” if the value is under the threshold of significance, typically ≤ 0.05.

Results

To conclude, the researchers discuss the primary outcome, or what they were most interested in investigating, in a section commonly called “Results” or “Results and Discussion”. Skipping right to this section after reading the abstract might be tempting, but that often leads to misinterpretation and the spread of misinformation.

Never read the results without first reading the “Methods” section; knowing how researchers arrived at a conclusion is as important as the conclusion itself.

One of the first things to look for in the “Results” section is a comparison of characteristics between the tested groups. Big differences in baseline characteristics after randomization may mean the two groups are not truly comparable. These differences could be a result of chance or of the randomization method being applied incorrectly.

Researchers also have to report dropout and compliance rates. Life frequently gets in the way of science, so almost every trial has its share of participants that didn’t finish the trial or failed to follow the instructions. This is especially true of trials that are long or constraining (diet trials, for instance). Still, too great a proportion of dropouts or noncompliant participants should raise an eyebrow, especially if one group has a much higher dropout rate than the other(s).

Scientists use questionnaires, blood panels, and other methods of gathering data, all of which can be displayed through charts and graphs. Be sure to check on the vertical axis (y-axis) the scale the results are represented on; what may at first look like a large change could in fact be very minor.

The “Results” section can also include a secondary analysis, such as a subgroup analysis. A subgroup analysis is when the researchers run another statistical test but only on a subset of the participants. For instance, if your trial included both males and females of all ages, you could perform your analysis only on the “female” data or only one the “over 65” data, to see if you get a different result.

Discussion

Sometimes, the conclusion is split between “Results” and “Discussion”.

In the “Discussion” section, the authors expound the value of their work. They may also clarify their interpretation of the results or hypothesize a mechanism of action (i.e., the biochemistry underlying the effect).

Often, they will compare their study to previous ones and suggest new experiments that could be conducted based on their study’s results. It is critically important to remember that a single study is just one piece of an overall puzzle. Where does this one fit within the body of evidence on this topic?

The authors should lay out what the strengths and weaknesses of their study were. Examine these critically. Did the authors do a good job of covering both? Did they leave out a critical limitation? You needn’t take their reporting at face value — analyze it.

Like the introduction, the conclusion provides valuable context and insight. If it sounds like the researchers are extrapolating to demographics beyond the scope of their study, or are overstating the results, don’t be afraid to read the study again (especially the “Methods” section).

Conflicts of Interest

Conflicts of interest (COIs), if they exist, are usually disclosed after the conclusion. COIs can occur when the people who design, conduct, or analyze research have a motive to find certain results. The most obvious source of a COI is financial — when the study has been sponsored by a company, for instance, or when one of the authors works for a company that would gain from the study backing a certain effect.

Sadly, one study suggested that nondisclosure of COIs is somewhat common. Additionally, what is considered a COI by one journal may not be by another, and some journals can themselves have COIs, yet they don’t have to disclose them. A journal from a country that exports a lot of a certain herb, for instance, may have hidden incentives to publish studies that back the benefits of that herb — so it isn’t because a study is about an herb in general and not a specific product that you can assume there is no COI.

COIs must be evaluated carefully. Don’t automatically assume that they don’t exist just because they’re not disclosed, but also don’t assume that they necessarily influence the results if they do exist.

Beware The Clickbait Headline

Never assume the media have read the entire study. A survey assessing the quality of the evidence for dietary advice given in UK national newspapers found that between 69% and 72% of health claims were based on deficient or insufficient evidence. To meet deadlines, overworked journalists frequently rely on study press releases, which often fail to accurately summarize the studies’ findings.

There’s no substitute for appraising the study yourself, so when in doubt, re-read its “Methods” section to better assess its strengths and potential limitations.

One study is just one piece of the puzzle

Reading several studies on a given topic will provide you with more information — more data — even if you don’t know how to run a meta-analysis. For instance, if you read only one study that looked at the effect of creatine on testosterone and it found an increase, then 100% of your data says that creatine increases testosterone.

But if you read ten (well-conducted) studies that looked at the effect of creatine on testosterone and only one found an increase, then you have a more complete picture of the evidence, which indicates creatine does not increase testosterone.

Going over and assessing just one paper can be a lot of work. Hours, in fact. Knowing the basics of study assessment is important, but we also understand that people have lives to lead. No single person has the time to read all the new studies coming out, and certain studies can benefit from being read by professionals with different areas of expertise.

Note from EC: As I’m busy, I try to rely on sources I can trust to help me carve out time (and sanity). That’s why whenever people ask me how to stay on top of nutrition research, I always refer them to Examine. 

At the end of the day, we’re busy individuals, and Examine keeps me on top of the cutting edge of research in 1/20th the time it would take me to do it myself. Instead of stressing out about screening, curating, reading, and summarizing research, Examine does it for me. Their membership is a great investment.

Sign-up Today for our FREE Newsletter and receive a four-part video series on how to deadlift!

Name
Email
Read more

Can You Trust the Research You’re Reading?

Today's guest post comes from the bright minds at Examine.com, who just released their new continuing education resource, Examine Research Digest. I love their stuff, and I'm sure you will, too. -EC

The internet is one of the last true democracies.

It’s a place where anybody with the necessary tools (a computer and an internet connection) can actively shape the perception of information...even if they have no qualification to do so.

Though the democratization of information is a good thing, one would assume that certain topics like scientific research would remain steeped in their foundations, because...well...that's how they remain reliable.

Unfortunately, in efforts to keep up with the demands for new, sexy content, many writers have taken to regurgitating information with little to no understanding of its context or how it affects you: the end reader. This is one of the many ways information gets skewed.

It’s often said that misinformation is a symptom of misinterpretation. The very definition of words can mean different things to different people.

questi8-n

One example of this is during research when a conclusion is reported as "significant." When scientists use this term, it implies "statistical significance." What this means is that the probability of the observations being due to the intervention is much greater than simply by chance.

This is very different than the general understanding of “significant.” Think of it this way: if your deadlift goes up from 405 to 410, that could be considered statistically significant in science. Would you say "my deadlift went up significantly," though? Probably not!

Now imagine how this simple misunderstanding of a term can impact the interpretation of a study. Something that may mean very little to a researcher is taken out of context by a well meaning blogger, eventually ending up as a eye-catching headline in your Facebook timeline.

A second way that information becomes misinformation is through the process of simplification.

When scientific studies are written, they are done so to most effectively relay their findings to other scientists, facilitating future studies and discoveries on the topic in question. If you’ve ever read a research study, you know that this approach to writing hinges on the use of precise terminology and complex verbiage so that nothing gets misinterpreted.

Unfortunately, this approach is less than ideal for relaying important findings to the people who can apply it. This leaves a few options:

1. "Dumb down" the content, hoping nothing gets lost in translation.

2. Keep as-is, with the understanding that it won't be able to reach as many people as intended.

3. In the most egregious option, data gets turned into "sound bites" that are easily transmitted by traditional media outlets.

Once one or more of these things happen, all traces of relevance to the original source get lost and misinformation starts to get spread. Moreover, another equally insidious way misinformation gets spread is by shifting focus onto one study (cherry-picking) rather than the entire body of evidence.

The internet has rapidly increased the speed of the news cycle. Information that once had time to be verified has taken a backseat to "as-it-happens" tidbits on Twitter. For the media to keep up, more factually inaccurate information gets disseminated in far less time.

Now, appreciate the fact that a news organization only has so much air time or so many words to talk about a new publication, and you can see how there isn't enough time to allow an adequate in-depth analysis of past studies or how the new study fits into the overall body of evidence.

Remember the media screaming “a high-protein diet is as bad as smoking?” Or that “fish oil caused prostate cancer?” These are perfect examples of two well-intentioned studies blown way out of proportion.

Pills

This leads to the fourth and final way misinformation gets spread: the reliance on controversy to gain an audience.

Earlier this year a blog post theorizing the connection between creatine consumption and cancer took social media by storm. The writers were savvy enough to understand that a title proclaiming creatine to be harmful had far more appeal than yet another post confirming its athletic performance benefits.

This sort of thing isn’t a new occurrence, but for some strange reason, audiences never tire of it. Once an controversial article starts getting shared, a case of broken telephone comes into play, transforming once-quality research into misinformation. As an industry, this is a problem we need to address.

"Epilogue" from EC

In spite of all this misinformation, there are people still fighting the good fight - and that's why I’m a big fan of Examine.com. They wrote our most popular guest post ever (on the science of sleep). And, whenever people ask me about supplementation, I refer them to Examine.com.

To that end, for those who want to be on the cutting edge of research, and want something that counters the overwhelming amount of misinformation, I'd recommend Examine.com's fantastic new resource, the Examine Research Digest (ERD).

ERD-intro-images

Before a study is presented in ERD, it's analyzed and reviewed by the researchers, then all references and claims are double-checked by a panel of editors. Subsequently, a final pass is done by a review panel of industry and academic leaders with decades of experience. Because you have a panel from different backgrounds, you know that you’re getting the complete picture, not the analysis of a single person.

Needless to say, I'm excited to take advantage of this resource personally to stay on up-to-date on some of the latest nutrition and supplementation research - and its practical applications for my clients and readers. I'd strongly encourage you to do the same, especially since it's available at a 20% off introductory price this week only. You can learn more HERE.

Sign-up Today for our FREE Newsletter and receive a four-part video series on how to deadlift!

Name
Email
Read more

5 Overlooked Resources for Making Your Strength and Conditioning Programs More Effective

I know there are a lot of fitness professionals who look to EricCressey.com as a continuing education resource. With that in mind, I wanted to discuss a few resources that have been tremendously valuable to me; hopefully you'll benefit from them (if you aren't already) as much as I have.

1. Video - Video is a powerful tool for coaching and monitoring progress in clients, and it's also very accessible nowadays, thanks to smart phones and digital cameras. Still, I'm always amazed at how few fitness professionals utilize it to help coach. I use it quite a bit in my evaluation process, especially with tough cases where I want to be able to monitor progress in movement quality. It's just as valuable on the training floor to back up coaching cues that you're giving.

Additionally, having access to the high-speed camera technology in our facility has been tremendously valuable in not only breaking down inefficient mechanics, but also demonstrating the powerful effects a good baseball strength and conditioning program can have on a pitcher's body control and power on the mound.

collins

2. Related Professionals - A fresh sets of eyes and a new perspective can have a huge influence on your strength and conditioning programs and how you coach. We've learned a ton from rehabilitation specialists, other fitness professionals, sport coaches, business consultants, and folks from a host of other professions.

As an interesting aside to this discussion, have you ever noticed how doctors - who have a minimum of eight years of higher education - refer patients out all the time to other doctors for second opinions? Yet, how often do you see personal trainers - who are a profession with an absurdly low barrier to entry - ask for another perspective from an unbiased third party? Food for thought.

3. Your Clients - I'm sure you'd love to think that you know your clients' bodies better than anyone else, but the truth is that those clients know themselves and how they're feeling much better than you ever could! I made the mistake early in my career of assuming too much and asking too few questions; I was talking 70% of the time and listening for the other 30%. Nowadays, I'm listening 70% of the time (at the very least) and I am a much better coach as a result.

As an example, now is a quiet time of year with all of our baseball guys in-season, so I'm using it as an opportunity to follow up with all our clients from this past off-season. I want to know how they felt during spring training, and how the transition to the start of the season went. All the feedback I get is valuable for not only next off-season, but helping them to tinker with things as needed right now.

4. New Training Equipment - Variety in a strength and conditioning program isn't just important to ensure optimal progress, but also to make sure that clients remain interested. Do you need to go out and buy all new equipment every other month? Of course not! However, adding some new training implements - or even just new uses for old equipment - can provide some variety. And, it's an opportunity for you to teach your client, as they're sure to ask: "What is this and what does it do?"

5. Business Partners/Assistants - When I first got started in Boston, I was doing all the scheduling and billing. While swiping credit cards and watching your schedule fill up is fulfilling at first, it eventually becomes a huge drain on your time, energy, and productivity. I'm a much better coach than I am a business logistics guy - and that's why the first person I contacted to help me start Cressey Sports Performance was my buddy, Pete Dupuis.

peted

Pete's become a fantastic business director (and vice-president) at CSP, and we've had double digit growth every year since we opened in 2007. He's managed my schedule, handled phone calls, done all our billing/invoicing, and become a liaison between coaches and clients when the clients aren't in the gym. In short, his efforts have made me more efficient so that I can evaluate, program for, and coach clients; review research; interact with other coaches; and do more staff/intern education.

Additionally, business partners, staff, and interns are great for asking the challenging questions that make you rethink the way you're doing things - and often provide suggestions and solutions that help make things more efficient and effective.

These are only five resources to get the ball rolling, but there are certainly many more available to fitness professionals in their quest to deliver a great client experience. I'd love to hear your thoughts in the comments section below on what resources have helped to make you better at what you do. 

And, while we're on the topic of continuing education, I want to give you a heads-up that many of our products are on sale for 25% off as part of Cyber Monday right now. The sale ends tonight, though, so don't delay in checking it out HERE.

Sign-up Today for our FREE Newsletter and receive a four-part video series on how to deadlift!

Name
Email
Read more
Page
LEARN HOW TO DEADLIFT
  • Avoid the most common deadlifting mistakes
  • 9 - minute instructional video
  • 3 part follow up series