#kmeans

Flag of I don't know how to use kmeans clustering on my midterm exam

Introduction

In today's highly competitive market, understanding your customers is more critical than ever. Companies across industries are leveraging data-driven strategies to gain insights into consumer behavior and tailor their marketing efforts accordingly. One effective method for achieving this is customer segmentation, which involves dividing a company's customer base into distinct groups based on shared characteristics. This approach enables businesses to personalize their offerings, enhance customer satisfaction, and ultimately increase profitability.

Unsupervised learning, a subset of machine learning, plays a significant role in customer segmentation. Unlike supervised learning, where models are trained on labeled data, unsupervised learning algorithms identify hidden patterns in data without predefined labels. One popular unsupervised learning technique is k-means clustering, which groups data points into clusters based on their similarity. In this blog, we'll explore how to apply k-means clustering in R Studio to segment customers based on purchasing behavior, equipping you with the practical skills needed for business analytics projects.

Dataset Overview

To illustrate the application of k-means clustering, let's consider a hypothetical dataset that captures customer purchase behavior. This dataset includes various features such as customer ID, age, gender, annual income, and spending score (a metric derived from customer spending habits and loyalty). Here's a brief look at what the dataset might contain:

This dataset provides a foundation for analyzing customer segments based on age, income, and spending behavior, offering valuable insights into different consumer groups.

Data Cleaning & Scaling

Before applying k-means clustering, it's essential to prepare the dataset appropriately. This involves data cleaning and scaling to ensure that the algorithm functions optimally.

Handling Missing Values

Begin by examining the dataset for any missing values. Missing data can skew results and lead to inaccurate clustering. In R Studio, you can use functions like is.na() and na.omit() to identify and remove or impute missing values. For instance:

# Check for missing values sum(is.na(dataset)) # Remove rows with missing values clean_dataset <- na.omit(dataset)

Standardization

K-means clustering is sensitive to the scale of the data. Therefore, it's crucial to standardize the dataset so that each feature contributes equally to the distance calculations. Standardization involves rescaling the data to have a mean of zero and a standard deviation of one. In R Studio, this can be achieved using the scale() function:

# Standardize the dataset scaled_dataset <- scale(clean_dataset[, c("Age", "Annual Income (k$)", "Spending Score (1-100)")])

Determining Optimal Clusters

Determining the optimal number of clusters is a critical step in k-means clustering. There are several methods to achieve this, with the elbow method and silhouette score being among the most popular.

Elbow Method

The elbow method involves plotting the total within-cluster sum of squares (WSS) against the number of clusters. The point where the WSS starts to decrease at a slower rate indicates the optimal number of clusters, resembling an "elbow" in the plot.

# Elbow method wss <- (nrow(scaled_dataset)-1)*sum(apply(scaled_dataset, 2, var)) for (i in 2:15) wss[i] <- sum(kmeans(scaled_dataset, centers=i)$withinss) plot(1:15, wss, type="b", xlab="Number of Clusters", ylab="Within groups sum of squares")

Silhouette Score

The silhouette score measures how similar an object is to its own cluster compared to other clusters. A high silhouette score indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.

# Silhouette score library(cluster) sil_width <- c(NA) for(i in 2:15){ km.res <- kmeans(scaled_dataset, centers = i, nstart = 25) sil_width[i] <- mean(silhouette(km.res$cluster, dist(scaled_dataset))[, 3]) } plot(1:15, sil_width, type="b", xlab="Number of Clusters", ylab="Silhouette Score")

Applying K-Means

Once the optimal number of clusters is determined, we can apply the k-means algorithm using R Studio's kmeans() function. This function partitions the dataset into clusters based on distance calculations.

Running kmeans() Function

To apply k-means clustering, specify the number of clusters and the dataset. The nstart parameter is recommended to ensure convergence to a global minimum.

# Applying k-means clustering set.seed(123) # for reproducibility kmeans_result <- kmeans(scaled_dataset, centers=5, nstart=25)

Interpreting Cluster Output

The k-means output includes several components, such as the cluster centers, the total within-cluster sum of squares, and the cluster assignments for each data point. These results help in understanding the distribution of data points across clusters.

# View cluster centers kmeans_result$centers # View cluster assignments head(kmeans_result$cluster)

Visualizing Clusters

Visualization is key to interpreting clustering results. Visual tools such as 2D scatter plots and cluster centroids provide a clear picture of how data points are grouped.

2D Scatter Plot

Using the ggplot2 package, you can create a scatter plot that visualizes the clusters in two dimensions, highlighting the distinct groups formed by the algorithm.

# Visualizing clusters library(ggplot2) ggplot(clean_dataset, aes(x=Annual.Income..k.., y=Spending.Score..1.100., color=factor(kmeans_result$cluster))) + geom_point(size=3) + labs(title="Customer Segments using K-Means Clustering", x="Annual Income (k$)", y="Spending Score (1-100)") + theme_minimal()

Cluster Centroids

Adding cluster centroids to the plot provides additional context, illustrating the central tendency of each cluster.

# Add cluster centroids centroids <- as.data.frame(kmeans_result$centers) ggplot(clean_dataset, aes(x=Annual.Income..k.., y=Spending.Score..1.100., color=factor(kmeans_result$cluster))) + geom_point(size=3) + geom_point(data=centroids, aes(x=V1, y=V2), color='red', size=4, shape=8) + labs(title="Customer Segments with Centroids", x="Annual Income (k$)", y="Spending Score (1-100)") + theme_minimal()

Business Interpretation

Translating clustering results into actionable business insights is crucial for decision-making. By analyzing the characteristics of each cluster, businesses can identify high-value customers and tailor marketing strategies to target specific segments.

Identifying High-Value Customers

Clusters often reveal customer segments with distinct purchasing behaviors. For instance, a cluster with high annual income and spending score might represent high-value customers who are more likely to respond to premium offerings.

Target Marketing Strategy

Understanding the unique characteristics of each segment allows businesses to develop targeted marketing strategies. For example, high-spending younger customers might be more receptive to digital marketing campaigns, while older, high-income customers may prefer personalized communication.

Conclusion

Customer segmentation is a powerful tool for businesses seeking to optimize their marketing strategies and improve customer satisfaction. By applying k-means clustering in R Studio, you can uncover meaningful patterns in customer behavior that inform strategic decisions. This blog has provided a practical guide to implementing k-means clustering, from data preparation to business interpretation. Armed with these insights, students and professionals in business analytics and data science can enhance their analytical skills and contribute to data-driven decision-making in their organizations.

Hi! I will be conducting one-on-one discussion with all channel members. Checkout the perks and Join membership if interested: … source

K-Means is one of the most popular machine learning algorithms you’ll encounter in data science interviews. In this video, I’ll … source

A k-means cluster analysis was conducted to identify underlying subgroups of individuals based on their similarity of responses on 12 variables that represent characteristics that could have an impact on maximum heart rate achieved.

Primarily, all python libraries need to be loaded that are required in creation for a lasso regression model. These also include the k-Means function from the sklearn.cluster library. Following are the libraries that are necessary to import:

Next, the required dataset is loaded. Here, I have uploaded the dataset available at Kaggle.com in the csv format using the read_csv() function. The dataset contains 14 attributes. These are age, sex, chest pain type (4 values), resting blood pressure, serum cholesterol in mg/dl, fasting blood sugar > 120 mg/dl, resting electrocardiographic results (values 0,1,2), maximum heart rate achieved, exercise induced angina, old peak = ST depression induced by exercise relative to rest, the slope of the peak exercise ST segment, number of major vessels (0-3) colored by fluoroscopy, thal: 0 = normal; 1 = fixed defect; 2 = reversible defect and output: 0= less chance of heart attack 1= more chance of heart attack.

Out of the 13 variables, only 12 were used in cluster analysis. Variable for maximum heart rate achieved is used for validation.

Before clustering, we need to standardize the variables measured on different scales. This is done so that the solution is not driven by variables measured on larger scales. The describe function is used to see statistical details of pandas dataframe. From the above data, we can see that our clustering variables are not standardized.

Here, we standardized the clustering variables to have a mean of 0, and a standard deviation of 1.The as type float 64 code ensures that all predictors will have a numeric format.

clustervar=cluster.copy()

clustervar['sex']=preprocessing.scale(clustervar['sex'].astype('float64'))

clustervar['age']=preprocessing.scale(clustervar['age'].astype('float64'))

clustervar['resting blood pressure']=preprocessing.scale(clustervar['resting blood pressure'].astype('float64'))

clustervar['cholestrol']=preprocessing.scale(clustervar['cholestrol'].astype('float64'))

clustervar['old peak']=preprocessing.scale(clustervar['old peak'].astype('float64'))

clustervar['chest pain']=preprocessing.scale(clustervar['chest pain'].astype('float64'))

clustervar['fasting blood sugar']=preprocessing.scale(clustervar['fasting blood sugar'].astype('float64'))

clustervar['resting ecg']=preprocessing.scale(clustervar['resting ecg'].astype('float64'))

clustervar['excercise included']=preprocessing.scale(clustervar['excercise included'].astype('float64'))

clustervar['slp']=preprocessing.scale(clustervar['slp'].astype('float64'))

clustervar['caa']=preprocessing.scale(clustervar['caa'].astype('float64'))

clustervar['THALL']=preprocessing.scale(clustervar['THALL'].astype('float64'))

clustervar['output']=preprocessing.scale(clustervar['output'].astype('float64'))

Now, dataset is divided into a training set and a test set. This can be achieved by using train_test_split() function. The size ratio is set as 70% for the training sample and 30% for the test sample. The random_state option specifies a random number seat(here I have selected as 123) to ensure that the data are randomly split the same way if the code is run again.

# split data into train and test sets

clus_train, clus_test = train_test_split(clustervar, test_size=.3, random_state=123)

cdist function from the scipy.spatial.distance library is used to calculate the average distance of the observations from the cluster centroids. We used 10 clusters for the analysis. Object meandist is used to store the average distance values that we will calculate for the 1 to 9 cluster solutions. The model is then initialized by calling the k-Means function from the sk learning cluster library. The function takes n_clusters which indicates the number of clusters as an argument. Here, we substituted n_clusters with k to tell Python to run the cluster analysis for 1 through 9 clusters. The model is then trained using the fit function which takes training features as argument. The code following meandist.append computes the average of the sum of the distances between each observation in the cluster centroids.

# k-means cluster analysis for 1-9 clusters

from scipy.spatial.distance import cdist

clusters=range(1,10)

meandist=[]

for k in clusters:

model=KMeans(n_clusters=k)

model.fit(clus_train)

clusassign=model.predict(clus_train)

meandist.append(sum(np.min(cdist(clus_train, model.cluster_centers_, 'euclidean'), axis=1))

/ clus_train.shape[0])

Next, we plot the elbow curve using the map plot lib plot function. The plot shows decrease in the average minimum distance of the observations from the cluster centroids for each of the cluster solutions.

From the above graph, we can see that the average distance decreases as the number of clusters increases. We can observe at two clusters, at three clusters, at seven clusters, and at eight clusters , there appear to be bends. These bends indicates that average distance value is leveling off such that adding more clusters doesn't decrease the average distance as much. Notice that these bends are not very much clear. This means that the elbow curve was inconclusive.

So we'll rerun the cluster analysis, this time asking for 3 clusters. To do so, simply initialize the model by calling the Kmeans function and set nclusters=3.

# Interpret 3 cluster solution

model3=KMeans(n_clusters=3)

model3.fit(clus_train)

clusassign=model3.predict(clus_train)

# plot clusters

Now, we used canonical discriminate analysis, which is a data reduction technique that creates a smaller number of variables that are linear combinations of the 3 clustering variables. To conduct the canonical discriminate analysis, we used the the PCA function and the sklearn decomposition library.

PCA(2) asks Python to return the two first canonical variables.Then we create a matrix called plot_columns that will include the two canonical variables estimated by the canonical discriminate analysis. PCA_2.fit asks Python to fit the canonical discriminate analysis that we specified with the PCA command, and the _transform applies the canonical discriminate analysis to the clus_train data set to calculate the canonical variables. We will plot the two canonical variables by the cluster assignment values from the 3 cluster solution in a scatter plot using the matplot libplot function.

From the above graph we can see that none of the cluster did not overlap very much with the other clusters. This indicates less correlation among the observations. The observations in the green and yellow clusters had greater spread. Also the observation in the green cluster were spread out more than the other clusters, showing high within cluster variance. The results of this plot suggest that the best cluster solution may have fewer than 3 clusters, so it will be especially important to also evaluate the cluster solutions with fewer than 3 clusters.

The means on the clustering variables showed that compared to the other clusters, individuals in cluster 0 had the highest likelihood of having a heart attack. They are less likely to get a blood disorder called thalassemia than cluster 1, highest slope of the peak exercise ST segment, lesser exercise induced angina,moderate old peak and greater chances of having a chest pain. On the other hand, indivuals in cluster 1 had the least chance of having a heart attack. Compared to individuals in the other clusters, they were lower chances for having a chest pain, greater resting blood pressure, lower resting ecg, highest exercise induced angina, highest old peak, lowest slope of the peak exercise ST segment, most number of major vessels colored by fluoroscopy, and highest chances for getting a blood disorder called thalassemia. Individuals in cluster 2 appeared to have moderate chances for having a heart attack as compared to the other two clusters. They had higher levels of resting ecg and had the least levels of old peak, least number of major vessels colored by fluoroscopy and had the lowest chances for getting thalassemia.

Finally, let's see how the clusters differ on maximum heart rate achieved. We'll use analysis of variance to test whether there are significant differences between clusters on the quantitative max_heart_rate variable. To do this, we have to import the statsmodels.formula.api and the statsmodels.stats.multicomp libraries. We use the ols function to test the analysis of variance.

# validate clusters in training data by examining cluster differences in GPA using ANOVA

# first have to merge GPA with clustering variables and cluster assignment data

mhr_data=data['max_heart_rate']

# split GPA data into train and test sets

mhr_train, mhr_test = train_test_split(mhr_data, test_size=.3, random_state=123)

mhr_train1=pd.DataFrame(mhr_train)

mhr_train1.reset_index(level=0, inplace=True)

merged_train_all=pd.merge(mhr_train1, merged_train, on='index')

sub1 = merged_train_all[['max_heart_rate', 'cluster']].dropna()

import statsmodels.formula.api as smf

import statsmodels.stats.multicomp as multi

mhrmod = smf.ols(formula='max_heart_rate ~ C(cluster)', data=sub1).fit()

print (mhrmod.summary())

The analysis of variance summary table indicates that the clusters differed significantly on maximum heart rate achieved.

When we examine the means we find that individuals in cluster 0 had previously achieved highest heart rate(mean= 158.276923 and sd = 17.619687) and the individuals in cluster 1 had achieved the lowest levels for maximum heart rate(mean= 138.205479 and sd= 24.218885).

The tukey post hoc comparisons showed significant differences between clusters on maximum heart rate achieved.

K-means clustering analysis was performed on the raw data consisting of the dummy details of candidates that are considered in the job application such as age, marks obtained during 10th, 12th, graduation and post graduation education, work experience, current CTC and notice period. 

The objective of this analysis is to identify the underlying subgroups and their underlying contribution for selection of that candidate for the job role. The variables considered have different scales. Thus, in order to standardize the scale, preprocessing.scale functionality was used to achieve mean =0 and standard deviation =1 for all the variables.

The dummy data that I have considered only has 50 rows. Hence, I have skipped the step of splitting the data into train and test set. In order to select the ideal value of k, elbow curve was plotted. In this, for every cluster, euclidean distance was calculated for all the observations from their respective centroids and it was observed that at k=2, the average distance was minimum. Hence, further analysis has been carried out using value of k=2.

In all, there are 8 variables which are considered for clustering exercise. In order to represent the scatter plot of the observations, it is not possible to plot the points in 8 dimensional space. Hence, canonical discriminate analysis was performed to reduce the 8 clustering variables to 2 canonical variables. Below is the scatter plot of the same.

It is evident from the plot that the clusters have high variance within and the observations are not densely populated. This  could also be due to the fact that limited records were used for training the data.

Further, while analyzing the behavior of variables in every cluster, it was found that notice period was an important variable in cluster 1, whereas candidates having higher marks in graduation & HSC in cluster 2. Further, cluster 1 candidates had higher CTC than cluster 2 and 3. 

Below is the code that is used for performing above analysis.

##Importing the libraries import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.model_selection import train_test_split from sklearn import preprocessing from sklearn.cluster import KMeans import numpy as np

##Loading the data data=pd.read_csv("Job_Application.csv")

##Retrieving the columns of present in the data data.columns

###Selecting cluster variables and storing it in a new dataframe cluster = data[['Age', 'SSC', 'HSC', 'Grad', 'PG','Work_ex','Current_CTC','Notice_period']] cluster.describe()

##Standardizing clustering variables to have a mean=0 and std dev=1 clustervar = cluster.copy() clustervar['Age'] = preprocessing.scale(clustervar['Age'].astype('float64')) clustervar['SSC'] = preprocessing.scale(clustervar['SSC'].astype('float64')) clustervar['HSC'] = preprocessing.scale(clustervar['HSC'].astype('float64')) clustervar['Grad'] = preprocessing.scale(clustervar['Grad'].astype('float64')) clustervar['PG'] = preprocessing.scale(clustervar['PG'].astype('float64')) clustervar['Work_ex'] = preprocessing.scale(clustervar['Work_ex'].astype('float64')) clustervar['Current_CTC'] = preprocessing.scale(clustervar['Current_CTC'].astype('float64')) clustervar['Notice_period'] = preprocessing.scale(clustervar['Notice_period'].astype('float64'))

##Splitting the data into training and test set ##clus_train,clus_test = train_test_split(clustervar,test_size=0.0,random_state=123) clus_train = clustervar ##Performing cluster analysis with value of k from 1 to 9 from scipy.spatial.distance import cdist clusters=range(1,10) meandist=[]

for k in clusters:    modek=KMeans(n_clusters=k)    model.fit(clus_train)    ##Denotes the cluster that has been assigned to each row    clusassign = model.predict(clus_train)    ##Finds the average distance of observations from the centroids for all observations    meandist.append(sum(np.min(cdist(clus_train,model.cluster_centers_,'euclidean'),axis=1))/clus_train.shape[0])

##Plotting the elbow curve plt.plot(clusters,meandist) plt.xlabel('No. of clusters') plt.ylabel('Average distance') plt.title('Selecting K with elbow method')

##Interpretting 2 cluster solution model2 = KMeans(n_clusters=2) model2.fit(clus_train) clusassign = model.predict(clus_train)

##Plotting the clusters from sklearn.decomposition import PCA pca_2 = PCA(2) plot_columns = pca_2.fit_transform(clus_train) plt.scatter(x=plot_columns[:,0],y=plot_columns[:,1],c=model2.labels_) plt.xlabel('Canonical variable 1') plt.ylabel('Canonical variable 2') plt.title('Scatter plot of canonical variables for 2 clusters')

##to identify variable specific property of every cluster

##Creating index to merge cluster labels with the training data clus_train.reset_index(level=0,inplace=True) clu_list = list(clus_train['index']) labels = list(model2.labels_) new_list = dict(zip(clu_list,labels))

new_clus = DataFrame.from_dict(new_list,orient = 'index') new_clus

new_clus.columns = ['cluster']