When conducting linear regression analysis on your research data, you naturally hope that some independent variables significantly affect the dependent variable. Achieving this indicates that you’ve successfully selected independent variables that are presumed to influence the dependent variable.
Did you know that variance values in regression analysis are critical? The variance in question here is residual variance. The magnitude of residual variance greatly influences the p-value obtained.
We know that the smaller of p-value, the higher of chance to reject the null hypothesis. Successfully rejecting the null hypothesis means that the alternative hypothesis is accepted. Typically, in regression analysis, research hypotheses are placed in the alternative hypothesis.
There’s a statement that says the smaller of residual variance, the better the model. Moreover, smaller residual variance leads to smaller p-values. Conversely, larger residual variance results in larger p-values. Is this statement true? Let’s discuss this further in this article.
Understanding Residual Variance
In research, you’re likely familiar with the term variance. Variance is vital in describing the data collected from your research. But what exactly is variance?
In statistics, variance is defined as a measure of how much data within a variable deviate from its mean value. Here, I refer specifically to residual variance. So, what is residual variance?
Before understanding residual variance, you need to know what residuals are. In regression analysis, residuals are the differences between the observed values and the predicted values of those observations.
To find the predicted values, you first need to perform linear regression estimation to determine the intercept and regression coefficients. Using these coefficients, you can calculate the predicted values of the observed variable. The difference between the actual observed value (Y actual) and the predicted value (Y predicted) is called the residual.
Residual variance is crucial in evaluating how well your model performs. It measures the accuracy of the model. Residual variance reflects how far the model’s predictions are from the actual values. The smaller the residual variance, the better the model. But why is this the case? Let’s examine this from the perspective of the formulas used.
Formula-Based Approach
To answer the question posed in the title regarding the impact of residual variance on p-value, we can analyze the formulas involved. First, we need to understand the relationship between variance, standard error, and t-statistics in regression analysis.
With a solid understanding of these formulas, we can address the effect of variance on p-value in regression analysis. For detailed calculations of variance, you can refer to my previous article.
In this article, I will focus on the relationship between standard error and variance formulas in regression analysis. First, note that the standard error of regression is the square root of variance. The formula for standard error is as follows:
Next, let’s examine the formula for t-statistics. In regression analysis, the detailed formula for calculating t-statistics is as follows:
From this formula, it’s clear that to calculate t-statistics, you divide the estimated regression coefficient by its standard error. This means you calculate the standard error of regression for each independent variable coefficient.
Similarly, to calculate the standard error for each coefficient, you first need to compute the residual variance for each independent variable used.
From these formulas, we can conclude that if residual variance decreases, the standard error also decreases. A smaller standard error, according to the formula, implies a smaller denominator for the estimated regression coefficient, resulting in a larger t-statistic.
Conversely, if residual variance increases, the standard error also increases. Referring back to the t-statistics formula, dividing the estimated coefficient by a larger value yields a smaller t-statistic. What are the implications of this? Let’s discuss this in the next section.
Relationship Between T-Statistics and P-Value
You likely understand that t-statistics are closely related to p-values. In hypothesis testing, you can use either value to determine whether to accept or reject the null hypothesis.
In principle, if the t-statistic exceeds the critical value (T-table), the null hypothesis is rejected. Conversely, if the t-statistic is smaller than the critical value, the null hypothesis is accepted.
This principle also applies to p-values. P-values represent the probability of error in your research. If you set alpha at 5% and the p-value is smaller than 0.05, the null hypothesis is rejected. Conversely, if the p-value is greater than 0.05, the null hypothesis is accepted.
I’ve explained the calculation of p-values in another article on this website. In that article, I demonstrated that larger t-statistics yield smaller p-values. Conversely, smaller t-statistics yield larger p-values.
Now, let’s revisit the concept from the previous section regarding the relationship between residual variance and t-statistics. Smaller residual variance results in larger t-statistics. Conversely, larger residual variance results in smaller t-statistics.
This principle also applies to p-values: smaller residual variance leads to smaller p-values, while larger residual variance leads to larger p-values.
From this, we can conclude that smaller residual variance increases the likelihood of rejecting the null hypothesis. Conversely, larger residual variance decreases this likelihood. If you aim for larger t-statistics or smaller p-values, prioritize finding independent variables with smaller residual variance.
This concludes the article I can share with you. I hope it benefits and broadens your understanding of this topic. Thank you for reading, and see you in the next article on Kanda Data!
