#classification tree

Building  Decision Tree Model : 

In This project assignment  I will explore  non linear relationships among a series of explanatory variables and binary, categorical response variables.

Code

import pandas as pd

import numpy as np

from sklearn.metrics import*

from sklearn.model_selection import train_test_split

from sklearn.tree import DecisionTreeClassifier

from sklearn import tree

from io import StringIO

from IPython.display import Image

import pydotplus

from sklearn.manifold import TSNE

from matplotlib import pyplot as plt

%matplotlib inline

rnd_state = 23468

Load data

data = pd.read_csv('Data/breast_cancer.csv')

data.info()

In the output above there is an empty column 'Unnamed: 32', so next it should be dropped.

Plots

For visualization purposes, the number of dimensions was reduced to two by applying t-SNE method. The plot illustrates that our classes are not clearly divided into two parts, so the nonlinear methods (like Decision tree) may solve this problem.

Decision tree

Confusion matrix:

Actual 0 1 All

Predicted

0 96 8 104

1 5 62 67

All 101 70 171

Accuracy: 0.9239766081871345

Results

Generated decision tree can be found below:

Decision tree analysis was performed to test nonlinear relationships among a series of explanatory variables and a binary, categorical response variable (breast cancer diagnosis: malignant or benign).

The dataset was splitted into train and test samples in ratio 70\30.

After fitting the classifier the key metrics were calculated - confusion matrix and accuracy = 0.924. This is a good result for a model trained on a small dataset.

From decision tree we can observe:

The malignant tumor is tend to have much more visible affected areas, texture and concave points, while the benign's characteristics are significantly lower.

The most important features are:

concave points_worst = 0.707688

area_worst = 0.114771

concave points_mean = 0.034234

fractal_dimension_se = 0.026301

texture_worst = 0.026300

area_se = 0.025201

concavity_se = 0.024540

texture_mean = 0.023671

perimeter_mean = 0.010415

it's been decided select the following vars:

HISPANIC WHITE','BLACK','NAMERICAN','ASIAN','age'

in order to try to predict a regular smoked.

Previously I have install all library needed via conda gui.

[code]

-- coding: utf-8 --

""" Created on Sun JAN 09 21:12:54 2023

@author: malarcono """

-- coding: utf-8 --

from pandas import Series, DataFrame import pandas as pd import numpy as np import os import matplotlib.pylab as plt from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import classification_report import sklearn.metrics

os.chdir("C:\TREES")

""" Data Engineering and Analysis """

Load the dataset

AH_data = pd.read_csv("tree_addhealth.csv")

data_clean = AH_data.dropna()

data_clean.dtypes data_clean.describe()

""" Modeling and Prediction """

Split into training and testing sets

predictors = data_clean[['HISPANIC','WHITE','BLACK','NAMERICAN','ASIAN', 'age']]

targets = data_clean.TREG1

pred_train, pred_test, tar_train, tar_test = train_test_split(predictors, targets, test_size=.4)

pred_train.shape pred_test.shape tar_train.shape tar_test.shape

Build model on training data

classifier=DecisionTreeClassifier() classifier=classifier.fit(pred_train,tar_train)

predictions=classifier.predict(pred_test)

sklearn.metrics.confusion_matrix(tar_test,predictions) sklearn.metrics.accuracy_score(tar_test, predictions)

from sklearn import tree

from io import StringIO

from IPython.display import Image out = StringIO() tree.export_graphviz(classifier, out_file=out) import pydotplus graph=pydotplus.graph_from_dot_data(out.getvalue()).write_png('tree_dec.png')

and graphic result below

Part 13

Program in SAS

The program in Python [link]

Decision tree analysis was performed to test nonlinear relationships among a series of explanatory variables and a binary, categorical response variable. All possible separations (categorical) or cut points (quantitative) are tested. For the present analyses, the entropy “goodness of split” criterion was used to grow the tree and a cost complexity algorithm was used for pruning the full tree into a final subtree.

The following explanatory variables were included as possible contributors to a classification tree model evaluating alcohol consumption rate (my response variable), urbanrate, polityscore, incomeperperson, internetuserate.

Take a look at the output.

We can see in the model information table that the decision tree that SAS grew has 30 leaves before pruning and 3 leaves following pruning. Model event level let us confirm that the tree is predicting the value 1, that is high level (> 6 liters per adult) alcohol consumption (our target variable). That is why it is necessary to recode the low-level alcohol consumption to 2, keeping 1 equal to high-level alcohol consumption.

Notice too that the number of observations read from gapminder dataset was 152 and the number of observations used was 152 too. These two numbers is equal because there were some data preparations in the data management part of the program (left only the valid data for the target variable and each explanatory variables). Those observations with missing data on even one variable have been set aside.

Next SAS creates a plot of the cross validated average standard error (ASE) based on number of leaves each of the trees generated on the training sample.

A vertical reference line is drawn for the tree with the number of leaves that has lowest cross-validated ASE. In this case, the 3-leaf tree. The horizontal reference line represents the average standard error plus one standard error for this complexity parameter. Often the 1-SE rule is applied when you are pruning via the cost-complexity method to potentially select a smaller tree that has only a slightly higher error rate than the minimum ASE selecting the smallest tree that has an ASE below the horizontal reference line is in effect implementing the 1-SE rule.

The final smaller tree

Model-based confusion matrix which shows how well the final classification tree performed.

The total model classified 81.6% of the sample correctly, 67% sensitivity and 96.1% specificity.

The total model correctly classified 1-0.3289=67.11% of those countries which have high-level alcohol consumption and 96.05% of those which have low-level alcohol consumption.

ROC-curve shows sensitivity, that is the true positive rate, and specificity, the true negative rate plotted against each other.

Variable impotence table

Elements Of Statistical Learning

C O N T E N T S:

KEY TOPICS

Elements of Statistical Learning: data mining, inference and prediction (2nd Edition) (with J. Friedman, Springer-Verlag, 2009).(More…)

This new two-day course gives a detailed and modern overview of statistical models used by data scientists for prediction and inference, including sparse models and deep learning.(More…)

It focuses on formulation of prediction…

Here I am using the GapMiinder data set to analyze which explanatory variables does the best job of classifying life expectancy among countries.

I am running a decision tree analysis on the below variables:

Response Variable: Life Expectancy (lifeexpectancy), converted to a categorized variable le_cat with two levels

1: High – Life expectancy greater than 65 years

2: Low – Life expectancy less than or equal to 65 years

Explanatory Variables:

             urbanrate

             incomeperperson

              employrate

              alcconsumption

              hivrate

              suicideper100th

              femaleemployrate

Here all my explanatory variables are quantitative.

A decision tree analysis was executed to test non-linear relationships among the above explanatory variables and the categorical response variable and all possible cut points are tested. I have used the entropy “goodness of split” criterion to grow the tree and a cost complexity algorithm for pruning the full tree into the final subtree.

The SAS code for my analysis is given below:

Output:

The final subtree (after pruning) generated by the snippet is: 

Confusion Matrix:

Observations and Interpretations

1.      Out of 213 countries available in the data set, only 142 were considered for the analysis as these records had data for all variables considered.

2.      The initial tree after the “grow” step had 8 leaf nodes. After pruning, the adjusted subtree has 4 leaves.

3.      The HIV rate was the first variable to separate the sample into two subgroups. Countries with HIV rate < 0.577 (82 countries) were put into one bucket where 91% have high life expectancy and only 9% have low life expectancy.

4.      For the 60 Countries with high HIV rate (>= 0.577) 33.33% have high life expectancy and 66.67% have low.

5.      This subgroup is then further divided by income per person at a cut point of 637.169.

6.      Out of the countries with high HIV rate and low income per person (30 countries) none have high life expectancy.

7.      On the other hand, 66.67% countries with high HIV rate and high income per person (30 countries) have high life expectancy

8.      The group of countries with high HIV rate (>= 0.577) and high income per person (> 637.169) are further subdivided by using HIV Rate again into two subgroups:

           a.      HIV rate below 3.161. In this group, 91% countries have high life expectancy.

           b.      HIV rate above 3.161. In this group, no country has high life expectancy.

9.      From the confusion matrix,

         a.     The classification tree, correctly classifies countries with high life expectancy 100% of the time.