This Exploratory Data Analysis (EDA) aims to identify the top 3 features that correlate to the SalePrice by breaking down into each 5 MSZoning. The dataset is published through Kaggle (link)

Before we do anything, we need to load libraries, set work directory, load data, and create a theme.

First of all, let’s glimpse at the data

Hm, only 1,460 rows but with 81 columns excluding predictor and ID, then we only have 79 predictors. We have 43 factor variables and 36 integer variables.

Let’s check missing values

Well, 6,965 is a whole lot of NA. We need to pinpoint what features they belong.

The majority of NAs (5,407 out of 6,965 or 77%) is from Alley, PoolQC, Fence, MiscFeature. According to the data dictionary, NA in those categories indicates that they simply just don’t have the feature. For the EDA purpose, I will just keep them in. But I will remove them from Machine Learning step.

Let’s wait no more, let’s see the graphs and charts!

Almost 80% of the data is from RL or Residential Low Density. In term of price, RL also has the most outliers.

When it is the time for ML, I am quite inclined to break them down into MSZoning and run the technique to each MSZoning type. We sure can apply the log transformation to the entire dataset. But 20% of the data is somewhat fine. But we will see.

Let’s look at another view of those outliers

Most of the selling price is between $100,000 and $200,000 which represented by RL. Yet RL in the density plot has a long tail due to outliers. As RL influences the dataset and represents 78% of the dataset, we should spend most of our time on it. But before that, there are a couple of variables that is glaringly influential:  Lot Area, OverallQual, OverallCond, LotArea, and YearRemodel.

Let’s take a look at Overall Qual and Overall Cond.

Yes, the correlation is super clear: the higher the quality, the better sales prices. However, OverallCond does not exhibit the clear pattern as OverallQual. These two qualities are very different. OverallQual indicates the quality of material and finish of the house, which I think are the heart and soul of a house. OverallCond, on the contrary, indicates the overall condition of a house. So you may get a pristine clean house but made by asbestos materials which should get a high score on OverallCond but super low on OverallQual. On the contrary, there could be a house made from concrete with some mold on a wall, which should get a high score on OverallQual but low on OverallCond.

So far so good, what about LotArea?

On the left side, you can see that overall higher LotArea will lead to the higher sales price. The exception is in “C (all)” category in which the price seems to wiggle up and down. But for the rest MSZoning category, the intuition applies.

The last numerical variable that we will take a look is YearRemodAdd. If there is no remodeling, the YearRemodAdd will be the same as YearBuilt. A pretty clean data indeed.

The newer, the better!

At this point, we have an overall understanding of influential variables. We could have looked at the overall data through correlation diagram. But before that, I want to focus on RL outliers. Let’s create a new dataset.

Let’s take a look at the SalePrice and LotArea.

I kind of understand those super outliers. But if you really take a look where the cluster is and ignore outliers. This code should be better in revealing them.

What I am interested is why those with more than 20,000 lot area are sold lower than those with less than 20,000. Yeah, I know, there must be some features that separate them apart. Let’s take a look at them!

Wow, that is easy to see. Houses that were sold higher than $300,000 were built after 1990, while those with more space was built way earlier than 1990. Hm, that is sure interesting. However, the year built itself may just be the sign of some underlying features. Say… could it be the swimming pool? Or could it be some construction type that is only available after 1990?

No, the pool is not a factor. Well, we could repeat this with another 80 variables. Yuck. This is the time when statistical significance or PCA come into play. Too many variables? No problema. But I’d cover that in Machine Learning section.  For now, I’d like to focus on pure EDA approach… another tool is correlation matrix. Unfortunately, correlation works with numeric variables and without NA only. So we will select only numeric variables without NAs.

Now it’s time to run the correlation plot.

OverallQual is for sure has very high correlation with the SalePrice. YearRemodel, TotalBmstSF also have strong correlations. Although we can change the syntax in the correplot(), I am only interested in the correlation to the SalePrice. Hm, let’s just extract the column and use GGPLOT.

That’s much better. I’d think there are no new results that go against our common sense. The bigger space, quality, garage lead to high sell price. The only surprise is the OverCond which has a negative correlation with SalePrice. But that could also make sense if those with high OverCond have fewer features than other houses whose OverCond is low.

Now, let’s apply the same analyses to other MSZoning.

Yep, OverallQual and GrLivArea are the most positively correlated to the SalePrice. Let’s try RH.

Those two are still the most correlated. But it seems like for high-density residential property purchasers; the living area is more important that overall quality of the house.

Let’s see what commercial estate purchasers think.

Oh, that’s interesting. The most positively correlated is TotRmsAbvGrd, which stands for Total Rooms Above Grade followed by usual GrLivArea and 1STFlrSF or First Floor square feet.

What about FV or Floating Village Residential?

Yep, it is still the usual OverallQua and GrLiveArea.

I think we have uncovered different preferences among purchasers of different type of property. Let’s combine them all together for easy comparison.

It is interesting to see that OverallQual is not the most correlated factor for commercial property. I’d guess for commercial property; space is the most important thing. After all, they purchase property to turn them into a money-making machine, right? So space is king.

Now, what about the entire dataset? My guess is it’s going to be close to what RL is as RL represents almost 80%.

TL;DR; The preference of buyers differ by each type of zoning. Among numeric variables, RL MSZoning’s features mostly match to our common sense: newer YearBuilt, high OverQual, high LotArea correlated to higher price. But some MSZoning such as TotRmsAbvGrd, and 1STFlrSF are the most correlated. But as RL represents 78%, the entire train dataset is similar to that of RL.