Residual Leverage Plot (Regression Diagnostic)

In linear or multiple regression, it is not enough to just fit the model into the dataset. But, it may not give the desired result. To apply the linear or multiple regression efficiently to the dataset. There are some assumptions that we need to check on the dataset that made linear/multiple regression efficient and generate better accuracy.

Assumptions of Regression

Regression analysis requires some assumptions to be followed by the dataset. These assumptions are:

• Observations are independent of each other. It should be correlated to another observation.
• Data is normally distributed.
• The relationship b/w the independent variable and the mean of the dependent variable is linear.
• The data is in homoscedasticity, which means the variance of the residual is the same for each value of the dependent variable.

To perform a good linear regression analysis, we also need to check whether these assumptions are violated:

• If the data contain non-linear trends then it will not be properly fitted by linear regression resulting in a high residual or error rate.
• To check for the normality in the dataset, draw a Q-Q plot on the data.
• The presence of correlation between observations is known as autocorrelation. We can check for the autocorrelation plot.
• The presence of homoscedasticity can be estimated with the plots such as the Scale Location plot, and the Residual vs Legacy plot.

Regression Diagnostic Plots

The above plots can be used to validate and test the above assumptions are part of Regression Diagnostic. This diagnostic can be used to check whether the assumptions. Before we discuss the diagnostic plot one by one let’s discuss some important terms:

• Outliers: Outliers are the points that are distinct and deviant from the bulk of the dataset. In general, the outliers have high residual values means that the difference is greater than the b/w observed and predicted value.
• Leverage Points: A leverage point is defined as an observation that has a value of x that is far away from the mean of x.
• Influential Points: An influential observation is defined as an observation that has a large influence on the fit of the model. One method to find influential points is to compare the fit of the model with and without each observation.

Below are the plots that we used in the diagnostic plot:

• Residual vs fitted plot: The residual can be calculated as:

[Tex]res = y_{observed} – y_{predicted}[/Tex]

This plot is used to check for linearity and homoscedasticity, if the model meets the condition of linear relationship then it should have a horizontal line with much deviation. If the model meets the condition for homoscedasticity, the graph should be equally spread around the y=0 line.

• Q-Q plot: This plot is used to check for the normality of the dataset, if there is normality that exists in the dataset then, the scatter points will be distributed along the 45 degrees dashed line.
• Scale-Location plot: It is a plot of square rooted standardized value vs predicted value. This plot is used for checking the homoscedasticity of residuals. Equally spread residuals across the horizontal line indicate the homoscedasticity of residuals.
• Residual vs Leverage plot/ Cook’s distance plot: The 4th point is the cook’s distance plot, which is used to measure the influence of the different plots. The Cook’s distance statistic for every observation measures the extent of change in model estimates when that particular observation is omitted. Cook distance plot the cook distance measure of each observation. whereas, Residual vs Leverage plot is the plot between standardized residuals and leverage points of the points.

Implementation

In this implementation, we will be plotting different diagnostic plots. For that, we use the Real-Estate dataset and apply the Ordinary Least Square (OLS) Regression. We then plot the regression diagnostic plot and Cook distance plot.

------------ # data No X1 X2 X3 X4 X5 X6 Y 0 1 2012.917 32.0 84.87882 10 24.98298 121.54024 37.9 1 2 2012.917 19.5 306.59470 9 24.98034 121.53951 42.2 2 3 2013.583 13.3 561.98450 5 24.98746 121.54391 47.3 3 4 2013.500 13.3 561.98450 5 24.98746 121.54391 54.8 4 5 2012.833 5.0 390.56840 5 24.97937 121.54245 43.1 ------------ OLS Regression Results ============================================================================== Dep. Variable: Y R-squared: 0.491 Model: OLS Adj. R-squared: 0.489 Method: Least Squares F-statistic: 198.3 Date: Thu, 19 Aug 2021 Prob (F-statistic): 5.07e-61 Time: 17:56:17 Log-Likelihood: -1527.9 No. Observations: 414 AIC: 3062. Df Residuals: 411 BIC: 3074. Df Model: 2 Covariance Type: nonrobust ============================================================================== coef std err t P>|t| [0.025 0.975] ------------------------------------------------------------------------------ Intercept 49.8856 0.968 51.547 0.000 47.983 51.788 X3 -0.0072 0.000 -18.997 0.000 -0.008 -0.006 X2 -0.2310 0.042 -5.496 0.000 -0.314 -0.148 ============================================================================== Omnibus: 161.397 Durbin-Watson: 2.130 Prob(Omnibus): 0.000 Jarque-Bera (JB): 1297.792 Skew: 1.443 Prob(JB): 1.54e-282 Kurtosis: 11.180 Cond. No. 3.37e+03 ============================================================================== Warnings: [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. [2] The condition number is large, 3.37e+03. This might indicate that there are strong multicollinearity or other numerical problems. --------------

References:

• BUMC Regression diagnostic
• University of Virginia