Predictive Modeling

What is Predictive Modeling/Analytics?

Predictive Modeling/Analytics involves manipulations on data from exisiting data sets with the goal of identifying some new trends or patterns, in which are used to predict future outcomes and trends.

What’s the difference between Predictive Modeling/Analytics and Machine Learning?

• Predictive Modeling is the mathematical approach for which we use statistics and past trends for “future predictions”. While Machine Learning uses various ML models to solve complex problems.

Why is Predictive Analysis Important?

Well we use historical data to predict future outcomes, thus it improves decision making and helps increase profit rates of business/engineering (KPIs).

Predictive Analysis can be used in many fields (domains) including:

• Online Retail
• Engineering
• Eduction
• Predicting Weather Forecasting
• Social Media Analysis

What are the Steps to Perform Predictive Analysis?

1. Define Problem Statement
2. Collect Data
3. Clean Data
4. Data Analysis
5. Build a Predictive Model
6. Validate the Model
7. Deployment of the Model
8. Monitor the Model

Predictive Modeling

• The task of predicting the value of a target variable (y) as a function of the predictor variables (X)

• Problem of learning a mapping function from inputs to outputs called function approximation

  y = f(X)

Classification vs Regression

Regression

Regression is the process of finding a model or function for distinguishing the data into continuous real values instead of using classes

Method of Calculation: Measurement of root mean square error

Examples of Regression Regression Tree (random forest) , Linear Regression

Let’s see how Linear Regression works

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

seed = 1            # seed for random number generation 
numInstances = 200  # number of data instances
np.random.seed(seed)
X = np.random.rand(numInstances,1).reshape(-1,1)
y_true = 2*X + 1 
y = y_true + np.random.normal(size=numInstances).reshape(-1,1)

plt.scatter(X, y,  color='black')
plt.plot(X, y_true, color='blue', linewidth=3)
plt.title('True function: y = 2X + 1')
plt.xlabel('X')
plt.ylabel('y')

Output:

Classification

Classification is the task of classifying things into sub-categories. Definition: Classification is the task of learning a Target function f that maps each attribute set x to one of the predefined class labels y.

It has two types:

1. Binary Classification: Categorizes given data into two distinct classes

2. Multiclass Classification: The number of classes is more than 2

Involves prediction of Discrete Values (individually seperate and distinct)

Method of Calculation: Measuring Accuracy

Examples of Classification: Decision Tree, Logestic Regression

Predictive Modeling

Terminology for Predicitve Modeling

• Training Set: is the set used for model building, in which contains a set of labeled examples, including the target variablue values are known

• Test Set: is used to predict the target values of the unknown data or it’s used to evaluate the performance of the model (if its for evaluation, the target values of test examples must be known)

• Predictive Model: An abstract representation of the relationship between the predictor and the target variables

Predictive Modeling Techniques

Single Models

• Decison Tree Methods (examples below!!!)
• Support Vector Machine/Regression
• Artifical Neural Network

Ensemble Models

• Boosting, Random Forest, Bagging

Lets work with a Decision Tree Classifier

A Classification Technique is a systematic approach to build a Classification models from an input of data!

A decision tree classifier is organized a series of test questions and conditions in a tree structure.

In a decision tree, the root and internal nodes contain attribute test conditions to “separate” records that have different characteristics, all the terminal node assigned a class label yes or no.

How do we Build a Decision Tree?

• It’s called/referred to as decision tree induction

Building an optimal decision tee is key in decision tree classifiers, there are many efficient algorithms that use Greedy approach/strategy. Greedy Algorithms is an algorithm paradigm that builds up a solution piece-by-piece, which makes locally optimum decisions about what attributes to use for dividing the data.

Examples of different Greedy Algorithms:

1. Hunts
2. ID3
3. C4.5
4. CART
5. SPRINT

All 5 of those algorithms are used in Greedy decision tree induction.

How do we Determine the Best Attribute test Condition?

We have to split the “Nodes”. We can do a two-way split or a multi-way split. A two-way split is easy for when there are binary attributes (Yes or No). If there is a Nominal (many names) attributes which could have many values, a test condition can be expressed into a multiway split. For continuous attributes (example of money $0- 1,000,000), we could split into comparison tests ($0-100,000, $100,001-500,000, $500,001-1,000,000) (< & >). Since there are many ways to specify the test conditions from a given training set, we need a measurement to determine the best way to split the records.

The goal of the best test condition is whether it leads to a homogenous distribution of the nodes, which is the purity of child nodes before and after splitting. The larger the degree of purity, the better the class distribution.

Measurement of impurity can be determined by:

1. Gina Index (used in our example below)
2. Entropy
3. Misclassification Error

Decision Tree Example Notes:

Example of a Decision Tree Classification

• using sklearn in Python

• Using a NBA Draft Combine Information file to predict if the player will be drafted based on picking a few different categories.

Heres the csv file -> file

Steps

Step 1: Load the data

import pandas as p

data = p.read_csv('nba_draft_combine_all_years.csv', header = 0)
data

Output:

Step 2 Lets clean this data because there are missing values, here are the columns were intrested in using!

feature_columns = ["Height (No Shoes)", "Wingspan", "Standing reach","Vertical (Max)", "Weight", "Agility", "Sprint"] 
update_data = data[feature_columns]
update_data

Output:

Now lets check for missing values!!!

update_data.describe()

Output:

We can see the count for the total players is 517, and in different columns like Vertical, Weight, Agility, and Sprint here is missing data. Ill just insert the Median value for the missing values.

## Vertical Max

update_data['Vertical (Max)'] = update_data['Vertical (Max)'].fillna(update_data['Vertical (Max)'].median())

## Weight

update_data['Weight'] = update_data['Weight'].fillna(update_data['Weight'].median())

## Agility

update_data['Agility'] = update_data['Agility'].fillna(update_data['Agility'].median())


## Sprint

update_data['Sprint'] = update_data['Sprint'].fillna(update_data['Sprint'].median())

Now all the Columns are filled in, no more missing data!

update_data

Output:

Step 3: Extract the predictor (X) and the Target (y) variables

X = update_data
y = data["Drafted"]

Now that we have the data split.

Step 3: Divide the data into training and test sets

from sklearn.tree import DecisionTreeClassifier 
from sklearn.model_selection import train_test_split
from sklearn import metrics 

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1) 

Note: test_size 0.3 means that 70% training and 30% test

Lets look at X_train,X_test, y_train, and y_test

 X_train

Output:

There are 361 rows in X_train!

 X_test

Output:

There are 156 rows in the X_test!

y_train

Output:

There are 361 rows in y_train.

y_test

Output:

There are 156 rows in y_test.

Step 4: Build the decision tree model and apply it to the test data

from sklearn import tree
clf = DecisionTreeClassifier()
clf = clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)

Step 5: Evaluate the accuracy of the model on the test data

from sklearn.metrics import accuracy_score

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

Output:

Accuracy: 0.6794871794871795

We can Visualize the Decision Tree using Scikit-learn’s export_graphviz function:

from sklearn import tree
import pydotplus 
dot_data = tree.export_graphviz(clf, out_file=None) 
graph = pydotplus.graph_from_dot_data(dot_data) 
graph.write_pdf("NBA_Draft_Prediction.pdf") 

from IPython.display import Image
Image(graph.create_png())

Output:

Model Evaluation

• Accuracy

Alternative Measures for Classification

ROC Curve

What is a ROC Curve? An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. The curve plots two parameters: True Positive Rate (TPR) and False Positive Rate (FPR).

Python Example Confusion Matrix Using the NBA Draft Prediction that we’ve been working with above!!

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred, labels=['No','Yes'])
cm

Output:

array([[13, 26],
       [27, 90]])

Python Example F Score

F-Measure: (or F-Score) is a measure of a test’s accuracy

from sklearn.metrics import f1_score

f1 = f1_score(y_test, y_pred, labels=['No','Yes'], pos_label = "Yes")
f1

Output:

0.7725321888412018

Y_prob = clf.predict_proba(X_test)
Y_prob

Output:

array([[0.33333333, 0.66666667],
       [1.        , 0.        ],
       [0.33333333, 0.66666667],
       ...,
       [0.88888889, 0.11111111],
       [0.33333333, 0.66666667],
       [0.33333333, 0.66666667]])

import matplotlib.pyplot as plt

from sklearn.metrics import roc_curve, auc
%matplotlib inline

true_labels = (Y_test == 'Yes');

fpr, tpr, thresholds = roc_curve(true_labels, Y_prob[:,1])
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr)

Output:

Holdout Method

• add later

5-Fold Cross Validation Method

from sklearn.model_selection import cross_val_score
clf = tree.DecisionTreeClassifier(max_depth=3)
scores = cross_val_score(clf, X, Y, cv=5)
scores

Output:

array([0.72727273, 0.72077922, 0.74025974, 0.7124183 , 0.74509804])

print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Output:

Accuracy: 0.73 (+/- 0.02)

Extras

import pandas as p

data = p.read_csv('vehicle.csv',header=0)
data.head()

Output:

data[['compactness','circularity','distance_circularity','radius_ratio']].corr()

Output:

Histogram

%matplotlib inline
data['compactness'].hist()

Box Plot

data[['compactness','circularity','distance_circularity','radius_ratio']].boxplot()

from sklearn.model_selection import train_test_split

Y = data['class']
X = data.drop('class',axis=1)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.23, random_state=1)

Y

Output:

Model Selection and Overfitting (More on Predictive Modeling)

Model Selection

What is Model Overfitting?

• Model Overfitting is error that occur when a function is too closely fit to a limited set of data points

• Too many details and noise

• The goal of Predictive Modeling is to build a model with low training and test errors

• Which is challenging because a model with low training errors does not guarantee it will have low test error (due to model overfitting problem)

An example of Model Overfitting:

import numpy as np
import matplotlib.pyplot as plt
from numpy.random import random

%matplotlib inline

N = 1500

mean1 = [6, 14]
mean2 = [10, 6]
mean3 = [14, 14]
cov = [[3.5, 0], [0, 3.5]]  # diagonal covariance

np.random.seed(50)
x1, y1 = np.random.multivariate_normal(mean1, cov, N//6).T
x2, y2 = np.random.multivariate_normal(mean2, cov, N//6).T
x3, y3 = np.random.multivariate_normal(mean3, cov, N//6).T
x4 = 20*np.random.random(N//2)
y4 = 20*np.random.random(N//2)

plt.plot(x1,y1,'ro',x2,y2,'ro',x3,y3,'ro',x4,y4,'g+',ms=4)

Output:

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.8, random_state=1)

from sklearn import tree
from sklearn.metrics import accuracy_score



maxdepths = [2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50]

trainAcc = np.zeros(len(maxdepths))
testAcc = np.zeros(len(maxdepths))

index = 0
for depth in maxdepths:
    clf = tree.DecisionTreeClassifier(max_depth=depth)
    clf = clf.fit(X_train, Y_train)
    Y_predTrain = clf.predict(X_train)
    Y_predTest = clf.predict(X_test)
    trainAcc[index] = accuracy_score(Y_train, Y_predTrain)
    testAcc[index] = accuracy_score(Y_test, Y_predTest)
    index += 1
    
plt.plot(maxdepths,trainAcc,'ro-',maxdepths,testAcc,'bv--')
plt.legend(['Training Accuracy','Test Accuracy'])
plt.xlabel('Max depth')
plt.ylabel('Accuracy')

Output:

You can see the plot above shows the training accuracy will continue to imporive as the “maximum depth” becomes more complex. However, the test accuracy initally improves up to a maximum depth of 5, before it sharply decreases due to model overfitting.

Building Trees of Different Sizes

from sklearn import tree
from sklearn.metrics import accuracy_score

maxdepths = [2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50]

trainAcc = np.zeros(len(maxdepths))
testAcc = np.zeros(len(maxdepths))
index = 0

for depth in maxdepths:
    clf = tree.DecisionTreeClassifier(max_depth=depth)
    clf = clf.fit(X_train, Y_train)
    Y_predTrain = clf.predict(X_train)
    Y_predTest = clf.predict(X_test)
    trainAcc[index] = accuracy_score(Y_train, Y_predTrain)
    testAcc[index] = accuracy_score(Y_test, Y_predTest)
    index += 1

Lets plot a Tree with MaxDepth = 4

depth = 4
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
AccTrain = accuracy_score(Y_train, Y_predTrain)
AccTest = accuracy_score(Y_test, Y_predTest)
[AccTrain, AccTest]

Output:

[0.8888888888888888, 0.6959349593495935]

What do these numbers tell us?

• With a tree of depth 4, our training accuracy was 88.88% which is pretty solid. Once we tested our model on our test data, the accuracy was 69.59% which is also solid.

Lets Plot the Tree with Depth = 4

import pydotplus 
from IPython.display import Image

dot_data = tree.export_graphviz(clf, out_file=None) 
graph = pydotplus.graph_from_dot_data(dot_data) 
Image(graph.create_png())

Output:

Lets plot a Tree with MaxDepth = 20

A tree with a MaxDepth of 20 should be very acccurate for the training data set. But how about when we test our model on the test data??

depth = 20
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
AccTrain = accuracy_score(Y_train, Y_predTrain)
AccTest = accuracy_score(Y_test, Y_predTest)
[AccTrain, AccTest]

Output:

[1.0, 0.640650406504065]

Accuracy Training = 100%

Accuracy Test = 64.07%

As you can see our training data was 100% accurate, but it was so overfitted aka too complex, that the test data accuracy was lower than the tree with a max depth of 4!

• The model may fit exceptional cases (rather than normal) in the training data!!

Lets Plot the Tree with Depth = 20

import pydotplus 
from IPython.display import Image

dot_data = tree.export_graphviz(clf, out_file=None) 
graph = pydotplus.graph_from_dot_data(dot_data) 
Image(graph.create_png())

Lets Plot Model Overfitting and Underfitting

plt.plot(maxdepths,trainAcc,'ro-',maxdepths,testAcc,'bv--')
plt.legend(['Training Accuracy','Test Accuracy'])

Output:

Model Overfitting and Underfitting

Defintions:

• Underfitting: When the model is too simple

• Overfitting: When the model is too complex

Model Selection Using Validation Set

• A general model selection approach that does not depend on the predicitive modeling technique used

• It divides the training data into two parts:

  • Training Set: for model building
  • Validation set: for estimating generalization error

A drawback is that less data would be available for training

Lets do an example of Model Selection Using Validation Set!

Other Predictive Modeling Techniques with Classification in Sklearn (L10)

Nearest-Neighbor Methods

What is Nearest-Neighbors?

• One of the simple/basic classification algorithms in Machine Learning.
• Supervised Learning

• Most People use K Nearest Neighbors

• Linear Method
• Nonlinear Method
• Ensemble Methods

K Nearest Neighbors

• Common algorithm used to solve classification model problems

K Nearest Neighbors in Sklearn

Linked to Dataset

• First opened the .data file and added a “header row” with the columns:

id,clump_thickness,unif_cell_size,unif_cell_shape,marg_adhesion,single_epith_cell_size,bare_nuclei,bland_chrom,norm_nucleoli,mitoses,class

import numpy as np
from sklearn import preprocessing, neighbors
from sklearn.model_selection import cross_validate

import pandas as pd

df = pd.read_csv('breast-cancer-wisconsin.data')
# Replace missing data!
df.replace('?',-99999, inplace=True)
## Drop ID, completly useless
df.drop(['id'], 1, inplace=True)

df

Output:

X = np.array(df.drop(['class'],1))
y = np.array(df['class'])

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2)

clf = neighbors.KNeighborsClassifier()
clf.fit(X_train, y_train)

accuracy = clf.score(X_test, y_test)
accuracy

Output:

0.9785714285714285

Lets add

example_measures = np.array([4,2,1,1,1,2,3,2,1])
example_measures = example_measures.reshape(1, -1)
prediction = clf.predict(example_measures)

Output:

array([2])

Ensemble Methods

• Ensemble Methods is taking the orginal training data

1. Creating Multiple Data Sets
2. Build Multiple Base Classifiers
3. Combine the Base Classifiers

Why use Ensemble models?

• Better accuracy
• Higher Consistency
• Reduces bias and variance errors

When and Where to use Ensemble Models?

• When single model overfits (decision trees usually overfit)

What are the different Ensemble Methods?

• Bagging: Build multiple classifiers by resampling the data with replacement
• Boosting: Build multiple classifiers by resampling the (weighted) data with replacement
• Random Forests: Train multiple decision tree classifiers and combine their prediction

Python Example