Hello Kanda Data friends, I hope you are all in good health. On this occasion, I will share some knowledge again, specifically regarding data transformation. This is crucial to understand for those of you currently conducting data analysis.
Especially for those of you performing classical assumption tests in regression analysis, it is highly important that you read this to the end. You surely know that in Ordinary Least Squares (OLS) linear regression analysis, conducting a series of classical assumption tests is essential. This aims to ensure that the estimation results obtained are valid, unbiased, and robust.
However, in attempting to meet these classical assumptions, we are sometimes faced with the obstacle of unfulfilled requirements. For example, the test results might show that the data is not normally distributed. Various methods have been tried and tested, yet the desired outcome remains elusive. I am sure this causes you sleepless nights. Right?
When normality tests yield unexpected results, the most popular instant solution—and one I frequently see relied upon by many students and researchers—is transforming the data into a Natural Logarithm (Ln) form.
But a new problem arises: what if, after the Ln transformation, the data is still not normal? Can we apply an Ln transformation a second time, or apply a different type of transformation to the previously transformed data? This article will thoroughly explore the theory, practice, and rules of data transformation.
Basic Theory of Natural Logarithm Transformation
Before proceeding further, let’s understand the basic theory of natural logarithm transformation. Natural Logarithm transformation, often symbolized as Ln, is a mathematical technique used to change the scale of original measurement data into another form for specific purposes. Simply put, a natural logarithm is a logarithm to the base e, Euler’s number, which is approximately 2.718.
In statistical analysis, Ln transformation is highly useful for handling data with positive skewness. This transformation works by compressing the distance between very large values while expanding the distance between smaller values. I hope you catch my drift, hehe…
So, as a result, the data variance becomes more stable. The impact is that it corrects previously non-normally distributed data to approach a normal bell curve. Then, what is the difference compared to non-transformed data? Will this affect the interpretation of the subsequent analysis results? Let’s discuss this further!
Data is Not Normally Distributed: Is Data Transformation the Right Move?
Although data transformation is a legitimate and academically recognized technique, it should ideally be the last resort. Before rushing to transform your data, you need to evaluate several things:
- Presence of Outliers: This is highly important! Often, non-normal data is caused merely by one or two observations with extreme values. If this is the issue, removing or handling the outlier data is more appropriate than transforming the entire dataset.
- The Inherent Nature of the Data: Certain types of data naturally do not follow a normal distribution, such as categorical data, count data, or probability data.
- Model Specification: Sometimes, residual non-normality occurs because we incorrectly inputted variables or ignored non-linear relationships in the basic regression model.
If you have evaluated the points above and the data indeed has a very wide range of values or heterogeneous variance, then the decision to perform data transformation is spot on. How do you transform the data? It’s very easy! Let me tell you. You can even do it using just Excel. You can also watch the video tutorial that Kanda Data previously made on our YouTube channel. Just search for it yourself, hehehe…
How to Transform Data in Excel
You can perform data transformation independently in Excel; the method is quite easy. We can quickly execute Ln transformations using Microsoft Excel. Here are the steps:
- Prepare a new column next to your original data column (for instance, the original data is in cell A2).
- In the empty cell next to it (e.g., cell B2), type the formula: =LN(A2)
- Press Enter.
- Drag the small box in the bottom right corner of cell B2 downwards to apply the same formula to all your data rows. Don’t worry, even if you have a lot of data, it can be very fast using Excel.
OOh yes, this is an Important Note! Ln transformation can only be processed for positive numbers or values greater than zero (> 0). If your data contains zero or negative numbers, Excel will produce error #NUM!.
If the Transformed Data is Still Not Normal, Can We Apply Ln Transformation Again?
This is the core question I posed in the title of this article. The answer: Mathematically, it can be done, but statistically and practically, it is HIGHLY NOT RECOMMENDED. Performing an Ln transformation twice on the same variable ($Ln(Ln(Y))$) will cause two major problems:
- Biased Interpretation: If you logarithmize it twice, how will you read the results? Interpreting a “log-log percentage change” carries no logical meaning acceptable in research reporting or decision-making.
- Over-Transformation: Performing Ln twice risks reversing the problem entirely. Data that was initially right-skewed could drastically flip to become heavily left-skewed, ultimately remaining non-normal.
Is It Permissible to Apply a Different Transformation Afterward?
What if, after applying Ln (and failing), we stack another transformation on top of it, such as taking the square root? Just like the previous case, applying various types of transformations to the same single variable is bad practice in the data analysis process.
If the first Ln transformation fails to normalize the data, the correct step is to revert to the initial data (the original, untransformed data) and then try another transformation method. Several alternatives that can be used include:
- Square Root Transformation: Suitable for count data.
- Inverse Transformation (1/X): Suitable for data with high skewness.
- Box-Cox Transformation: This is a method where an algorithm autonomously searches for the most optimal transformation power to normalize the data.
However, if all single transformation methods on the original data fail, stop forcing the data to be normal. You can pivot to alternatives such as using Non-Parametric statistical tests or utilizing Generalized Linear Models (GLM), which are specifically designed to handle data without the assumption of normality.
Conclusion
Natural Logarithm (Ln) transformation is an analytical tool that is incredibly helpful when used correctly. However, its boundaries are clear: this transformation should only be performed once on a variable. Repeating the Ln transformation or stacking it with other methods will not only corrupt the integrity of the data structure but will also render your research results uninterpretable.
If a single transformation yields no results, take that as a signal to evaluate the data you have more deeply. For instance, try another transformation method on the original data, or switch to statistical methods that do not require the data to be normally distributed.
That concludes the article that Kanda Data can write for this occasion. Hopefully, it is beneficial and broadens the horizons for those of you currently conducting data analysis. Stay tuned for the next article update from Kanda Data!
