Datadriven

A Lasso regression analysis was conducted for the use case of job application. Total 14 variables were considered as predictor variables. Since the data set is small, having more variables that do not add much value to the model might overfit the model. Hence, using lasso, we will be able to eliminate the variables that marginally impact the dependent variable. Independent variables considered for this case study are Age, SSC marks, HSC marks, Graduation grade, post graduation grade, skills in python/ sql and excel, work experience, current ctc and job location - Mumbai, Bangalore and Delhi. The target variable is to predict whether the candidate will be selected or not.

Since the scale of independent variables are different from one another, it is important to standardize all the variables with the mean =0 and standard deviation =1. Hence, the standardization acitivity is performed using preprocessing.scale module of sklearn library. Data is  split into training and test set in the ratio of 70:30. The value pf folds i.e cv is considered as 5 while building the model on the training set. The mean square error is plotted at each step to identify the best set of predictor variables. 

From the regression coefficients of the predictor variables, it can be observed that age is highly negatively correlated with the job selection variable where as python skill is positively correlated with the y variable. Also, variables such as SQL skill, Delhi and Bangalore job location were eliminated from the final model as their regression coefficients are penalized to 0. The 11 variables accounted for the r square value of 66% variance in the job selection variable.

Below is the code used for executing this analysis.

##Importing the libraries

import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.model_selection import train_test_split from sklearn.linear_model import LassoLarsCV

##Loading the dataset data = pd.read_csv('Job_Application.csv')

##Selecting predictors and target predictors = data[['Age', 'SSC', 'HSC', 'Grad', 'PG', 'Python', 'SQL', 'Excel', 'Work_ex',       'Current_CTC', 'Mumbai', 'Bangalore', 'Delhi', 'Notice_period']]

target = data.Selected

##Standardizing variables to have mean=0 and standard deviation =1

predictors_1 = predictors.copy()

from sklearn import preprocessing predictors_1['Age'] = preprocessing.scale(predictors_1['Age'].astype('float64')) predictors_1['SSC'] = preprocessing.scale(predictors_1['SSC'].astype('float64')) predictors_1['HSC'] = preprocessing.scale(predictors_1['HSC'].astype('float64')) predictors_1['Grad'] = preprocessing.scale(predictors_1['Grad'].astype('float64')) predictors_1['PG'] = preprocessing.scale(predictors_1['PG'].astype('float64')) predictors_1['Python'] = preprocessing.scale(predictors_1['Python'].astype('float64')) predictors_1['SQL'] = preprocessing.scale(predictors_1['SQL'].astype('float64')) predictors_1['Excel'] = preprocessing.scale(predictors_1['Excel'].astype('float64')) predictors_1['Work_ex'] = preprocessing.scale(predictors_1['Work_ex'].astype('float64')) predictors_1['Current_CTC'] = preprocessing.scale(predictors_1['Current_CTC'].astype('float64')) predictors_1['Mumbai'] = preprocessing.scale(predictors_1['Mumbai'].astype('float64')) predictors_1['Bangalore'] = preprocessing.scale(predictors_1['Bangalore'].astype('float64')) predictors_1['Delhi'] = preprocessing.scale(predictors_1['Delhi'].astype('float64')) predictors_1['Notice_period'] = preprocessing.scale(predictors_1['Notice_period'].astype('float64'))

##Splitting the data into train and test set pred_train,pred_test,tar_train,tar_test = train_test_split(predictors_1,target,test_size=0.3,random_state=123)

##Model building using Lasso model = LassoLarsCV(cv=5,precompute=False).fit(pred_train,tar_train)

##Depicting coefficients of the predictor variables dict(zip(predictors_1.columns,model.coef_))

##Plotting regression coefficients m_log_alphas = -np.log10(model.alphas_) ax = plt.gca() plt.plot(m_log_alphas, model.coef_path_.T) plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',            label='alpha CV') plt.ylabel('Regression Coefficients') plt.xlabel('-log(alpha)') plt.title('Regression Coefficients Progression for Lasso Paths')

m_log_alphascv = -np.log10(model.cv_alphas_) plt.figure() plt.plot(m_log_alphascv, model.mse_path_, ':') plt.plot(m_log_alphascv, model.mse_path_.mean(axis=-1), 'k',         label='Average across the folds', linewidth=2) plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',            label='alpha CV') plt.legend() plt.xlabel('-log(alpha)') plt.ylabel('Mean squared error') plt.title('Mean squared error on each fold')

from sklearn.metrics import mean_squared_error train_error = mean_squared_error(tar_train, model.predict(pred_train)) test_error = mean_squared_error(tar_test, model.predict(pred_test)) print ('training data MSE') print(train_error) print ('test data MSE') print(test_error)

K-means clustering analysis was performed on the raw data consisting of the dummy details of candidates that are considered in the job application such as age, marks obtained during 10th, 12th, graduation and post graduation education, work experience, current CTC and notice period. 

The objective of this analysis is to identify the underlying subgroups and their underlying contribution for selection of that candidate for the job role. The variables considered have different scales. Thus, in order to standardize the scale, preprocessing.scale functionality was used to achieve mean =0 and standard deviation =1 for all the variables.

The dummy data that I have considered only has 50 rows. Hence, I have skipped the step of splitting the data into train and test set. In order to select the ideal value of k, elbow curve was plotted. In this, for every cluster, euclidean distance was calculated for all the observations from their respective centroids and it was observed that at k=2, the average distance was minimum. Hence, further analysis has been carried out using value of k=2.

In all, there are 8 variables which are considered for clustering exercise. In order to represent the scatter plot of the observations, it is not possible to plot the points in 8 dimensional space. Hence, canonical discriminate analysis was performed to reduce the 8 clustering variables to 2 canonical variables. Below is the scatter plot of the same.

It is evident from the plot that the clusters have high variance within and the observations are not densely populated. This  could also be due to the fact that limited records were used for training the data.

Further, while analyzing the behavior of variables in every cluster, it was found that notice period was an important variable in cluster 1, whereas candidates having higher marks in graduation & HSC in cluster 2. Further, cluster 1 candidates had higher CTC than cluster 2 and 3. 

Below is the code that is used for performing above analysis.

##Importing the libraries import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.model_selection import train_test_split from sklearn import preprocessing from sklearn.cluster import KMeans import numpy as np

##Loading the data data=pd.read_csv("Job_Application.csv")

##Retrieving the columns of present in the data data.columns

###Selecting cluster variables and storing it in a new dataframe cluster = data[['Age', 'SSC', 'HSC', 'Grad', 'PG','Work_ex','Current_CTC','Notice_period']] cluster.describe()

##Standardizing clustering variables to have a mean=0 and std dev=1 clustervar = cluster.copy() clustervar['Age'] = preprocessing.scale(clustervar['Age'].astype('float64')) clustervar['SSC'] = preprocessing.scale(clustervar['SSC'].astype('float64')) clustervar['HSC'] = preprocessing.scale(clustervar['HSC'].astype('float64')) clustervar['Grad'] = preprocessing.scale(clustervar['Grad'].astype('float64')) clustervar['PG'] = preprocessing.scale(clustervar['PG'].astype('float64')) clustervar['Work_ex'] = preprocessing.scale(clustervar['Work_ex'].astype('float64')) clustervar['Current_CTC'] = preprocessing.scale(clustervar['Current_CTC'].astype('float64')) clustervar['Notice_period'] = preprocessing.scale(clustervar['Notice_period'].astype('float64'))

##Splitting the data into training and test set ##clus_train,clus_test = train_test_split(clustervar,test_size=0.0,random_state=123) clus_train = clustervar ##Performing cluster analysis with value of k from 1 to 9 from scipy.spatial.distance import cdist clusters=range(1,10) meandist=[]

for k in clusters:    modek=KMeans(n_clusters=k)    model.fit(clus_train)    ##Denotes the cluster that has been assigned to each row    clusassign = model.predict(clus_train)    ##Finds the average distance of observations from the centroids for all observations    meandist.append(sum(np.min(cdist(clus_train,model.cluster_centers_,'euclidean'),axis=1))/clus_train.shape[0])

##Plotting the elbow curve plt.plot(clusters,meandist) plt.xlabel('No. of clusters') plt.ylabel('Average distance') plt.title('Selecting K with elbow method')

##Interpretting 2 cluster solution model2 = KMeans(n_clusters=2) model2.fit(clus_train) clusassign = model.predict(clus_train)

##Plotting the clusters from sklearn.decomposition import PCA pca_2 = PCA(2) plot_columns = pca_2.fit_transform(clus_train) plt.scatter(x=plot_columns[:,0],y=plot_columns[:,1],c=model2.labels_) plt.xlabel('Canonical variable 1') plt.ylabel('Canonical variable 2') plt.title('Scatter plot of canonical variables for 2 clusters')

##to identify variable specific property of every cluster

##Creating index to merge cluster labels with the training data clus_train.reset_index(level=0,inplace=True) clu_list = list(clus_train['index']) labels = list(model2.labels_) new_list = dict(zip(clu_list,labels))

new_clus = DataFrame.from_dict(new_list,orient = 'index') new_clus

new_clus.columns = ['cluster']

Random Forest Classifier was used to develop a model that can predict whether a candidate will be selected for the job role or not. So, ‘Selected' variable was considered as the target variable. Predictor variables considered are 'Age','SSC','HSC','Grad','PG','Python','SQL','Excel','Work_ex','Current_CTC','Mumbai','Bangalore','Delhi' and 'Notice_period'.

The model was split into the ratio of 70:30 for training and testing. Value of n_estimators was passed as 25. The accuracy of the model was 73%. While checking the feature importance, it was observed that the notice period was one of the most important feature for classification whereas SQL skills did not contribute much in the classification activity.

Below is the python code that was used for this case study.

##Imporing the libraries import pandas as pd import matplotlib.pylab as plt from sklearn.model_selection import train_test_split import sklearn.metrics from sklearn.ensemble import ExtraTreesClassifier

##Loading the dataset data=pd.read_csv("Job_Application.csv")

##Defining predictor and target variables predictors = data[['Age','SSC','HSC','Grad','PG','Python','SQL','Excel','Work_ex','Current_CTC','Mumbai','Bangalore','Delhi','Notice_period']] target = data.Selected

##Splitting the data into training and test set pred_train,pred_test,tar_train,tar_test = train_test_split(predictors,target,test_size=0.3)

pred_train.shape pred_test.shape tar_train.shape tar_test.shape

##Building Random Forest Model

from sklearn.ensemble import RandomForestClassifier classifier = RandomForestClassifier(n_estimators=10) classifier = classifier.fit(pred_train,tar_train) predictions = classifier.predict(pred_test)

##Deriving confusion matrix & accuracy score sklearn.metrics.confusion_matrix(tar_test, predictions) sklearn.metrics.accuracy_score(tar_test, predictions)

##Finding relatively important variables model = ExtraTreesClassifier() model.fit(pred_train,tar_train) print(model.feature_importances_)

##Finding accuracy of each tree trees = range(25) accuracy = np.zeros(25)

for id in range(len(trees)):    classifier = RandomForestClassifier(n_estimators=id+1)    classifier = classifier.fit(pred_train,tar_train)    predictions = classifier.predict(pred_test)    accuracy [id] = sklearn.metrics.accuracy_score(tar_test,predictions)    print(accuracy [id])

plt.cla() plt.plot(trees,accuracy)

IMAGE

PROBLEM STATEMENT

The presented use case includes several parameters that are analyzed while selecting the prospective employee such as age, marks of 10th, 12th, graduation & post graduation, hands on tools experience, work experience, current ctc and location preference. These are considered as predictors and whether the person is selected or not is considered as target variable. 

CODE:

import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report import sklearn.metrics

data=pd.read_csv('Job_Application.csv')

predictors = data[['Age','SSC','HSC','Grad','PG','Python','SQL','Excel','Work_ex','Current_CTC','Mumbai','Bangalore','Delhi','Notice_period']] targets = data.Selected

pred_train,pred_test,tar_train,tar_test = train_test_split(predictors, targets,test_size=0.4)

pred_train.shape pred_test.shape tar_train.shape tar_test.shape

classifier = DecisionTreeClassifier() classifier=classifier.fit(pred_train,tar_train) predictions=classifier.predict(pred_test)

sklearn.metrics.confusion_matrix(tar_test,predictions) sklearn.metrics.accuracy_score(tar_test,predictions)

from sklearn import tree from sklearn.tree._export import plot_tree import graphviz feature_names=['Age','SSC','HSC','Grad','PG','Python','SQL','Excel','Work_ex','Current_CTC','Mumbai','Bangalore','Delhi','Notice_period'] cn=['Yes','No'] dot_data=tree.export_graphviz(classifier,out_file='tree.dot',feature_names=feature_names,class_names=cn,filled=True) graph=graphviz.Source(dot_data)

After this, we need to run a command in the command prompt to convert the dot file into an image.

dot -Tpng tree.dot -o tree.png

SUMMARY