Blogs

6 things to know about confidence levels in statistical significance testing

Nov 28, 2023
Mar 19, 2025
 • 
 min read

How confident can you be in your market research results?

Statistical significance testing is an essential tool in gathering insights and making data-driven decisions. It helps researchers determine if observed results have occurred by chance or if they are something worth paying attention to. It’s also critical to take sampling errors into account when interpreting results from statistical tests. This is done by calculating confidence levels, which indicate how sure we can be in our results.

But what confidence level should you use when you research? Is it better to use one that is higher? And when can you get away with using a lower confidence level? In this blog, let’s go over confidence level basics, including when to use certain levels and what to use for your research goals.

1. What is a confidence level?

Confidence level is a measure of how confident the researcher is that the results they observe are not due to chance.

Deciding whether your data is statistically significant is usually a researcher’s prerogative. Typically the confidence level is set at 95%, which is the industry standard. This means that when statistical significance is calculated, there is a 95% probability that the observed difference is not due to chance. With that confidence level, researchers can be confident that the results are within 5% of the true population value, which is enough accuracy for many surveys.

2. Statistical power vs. confidence level

Statistical power and confidence level are related concepts, but they represent different aspects of hypothesis testing. Statistical power refers to the probability of correctly rejecting a false null hypothesis, indicating a true effect. It is influenced by factors like sample size, effect size, and significance level.

Confidence level, on the other hand, represents the degree of certainty associated with a statistical estimate, often expressed as a percentage.

As the power of a test increases, the confidence level usually decreases, and vice versa. Researchers must find a suitable balance between power and confidence level based on their study goals.

3. When do researchers use a higher confidence level?

When the stakes are higher, you want a higher confidence level so that you can be more "sure" that what you are seeing is not just random chance. The higher the confidence level, the lower the risk of incorrectly rejecting a null hypothesis and drawing an incorrect conclusion.

4. Can you ever achieve a 100% confidence level?

It’s impossible to attain a confidence level of 100% because there will always be a certain amount of sampling error or bias that could lead to incorrect results. It is also impossible to eliminate the risk of sampling errors or bias. So, unfortunately, you cannot be 100% confident in your results. But you can still get pretty close.

5. When do researchers use a lower confidence level?

In some circumstances, using a lower confidence level is perfectly fine. For example, if there is no significance observed at 95%, the researcher may accept a 90% or even an 85% or 80% confidence level, depending on the situation.

6. How do you choose a confidence level?

Confidence levels are a researcher's choice! Choosing the right confidence level comes down to finding the right balance of rigor for your study. Too much rigor (too high of a confidence level / too discriminating), and you might not observe any statistically significant findings in your dataset. Too little rigor (too low of a confidence level / too relaxed), and there is lower confidence or statistical power behind any statistically significant findings in your dataset.

Statistical significance testing and confidence levels at Suzy

Before delving into the details, it's important to understand that statistical significance testing in the Suzy platform is an automated process at a 90% to 95% confidence level. But in Data Explorer, you can explore 80% to 85% confidence levels.

Data Explorer allows two confidence levels to be used at once. Uppercase letters represent a higher confidence level and lowercase letters represent a lower confidence level.

At Suzy, you have the flexibility to choose the confidence level you need to meet your project goals. To see how statistical testing and confidence levels work at Suzy, book a demo with our team to see our product in action.

See Suzy in Action
Book a Demo today
See Suzy in action. Learn how Suzy can boost your business.

Receive the most up-to-date resources and strategies

Thank you for subscribing!
Oops! Something went wrong while submitting the form.

Related resources

Product
How to fight back against research panel bots and protect your consumer insights

Learn how to fight back against bots and fraud in consumer research with smarter panels, AI defenses, and Suzy’s Biotic system.

Consumer Insights
Market Research
Product
AI
CPG
Apr 24, 2025
Apr 24, 2025
 • 
 min read
Blogs
From awareness to action: How platform trust accelerates the buyer journey

Discover how trusted platforms impact ad success and learn why the context of your message matters more than ever in driving consumer action.

Market Research
Consumer Insights
Media & Entertainment
Apr 23, 2025
Apr 23, 2025
 • 
 min read
Blogs
Streaming into the future: How Gen-Z and Gen-Alpha are reshaping brand engagement

Gen Z & Alpha are reshaping brand engagement through streaming. To win them over, brands must embed in creator culture and lead with authenticity.

Market Research
Consumer Insights
Media & Entertainment
Apr 16, 2025
Apr 16, 2025
 • 
 min read
View all