# Predictive Modeling

What is Predictive Modeling/Analytics?

Predictive Modeling/Analytics involves manipulations on data from exisiting data sets with the goal of identifying some new trends or patterns, in which are used to predict future outcomes and trends.

What’s the difference between Predictive Modeling/Analytics and Machine Learning?

• Predictive Modeling is the mathematical approach for which we use statistics and past trends for “future predictions”. While Machine Learning uses various ML models to solve complex problems.

Why is Predictive Analysis Important?

Well we use historical data to predict future outcomes, thus it improves decision making and helps increase profit rates of business/engineering (KPIs).

Predictive Analysis can be used in many fields (domains) including:

• Online Retail
• Engineering
• Eduction
• Predicting Weather Forecasting
• Social Media Analysis

What are the Steps to Perform Predictive Analysis?

1. Define Problem Statement
2. Collect Data
3. Clean Data
4. Data Analysis
5. Build a Predictive Model
6. Validate the Model
7. Deployment of the Model
8. Monitor the Model

Predictive Modeling

• The task of predicting the value of a target variable (y) as a function of the predictor variables (X)

• Problem of learning a mapping function from inputs to outputs called function approximation

y = f(X)

Regression Quantitative (ratio/interval) Stock Market Prediction, revenue/sales Forecast, Temperature Prediction, Calorie Prediction
Classification Qualitative (nominal) Disease Prediction, Image Classification, Twitter Mood Prediction

# Classification vs Regression

## Regression

Regression is the process of finding a model or function for distinguishing the data into continuous real values instead of using classes

Method of Calculation: Measurement of root mean square error

Examples of Regression Regression Tree (random forest) , Linear Regression

Let’s see how Linear Regression works

``````%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

seed = 1            # seed for random number generation
numInstances = 200  # number of data instances
np.random.seed(seed)
X = np.random.rand(numInstances,1).reshape(-1,1)
y_true = 2*X + 1
y = y_true + np.random.normal(size=numInstances).reshape(-1,1)

plt.scatter(X, y,  color='black')
plt.plot(X, y_true, color='blue', linewidth=3)
plt.title('True function: y = 2X + 1')
plt.xlabel('X')
plt.ylabel('y')
``````

Output: ## Classification

Classification is the task of classifying things into sub-categories. Definition: Classification is the task of learning a Target function f that maps each attribute set x to one of the predefined class labels y. It has two types:

1. Binary Classification: Categorizes given data into two distinct classes

2. Multiclass Classification: The number of classes is more than 2

Involves prediction of Discrete Values (individually seperate and distinct)

Method of Calculation: Measuring Accuracy

Examples of Classification: Decision Tree, Logestic Regression

## Predictive Modeling

Terminology for Predicitve Modeling

• Training Set: is the set used for model building, in which contains a set of labeled examples, including the target variablue values are known

• Test Set: is used to predict the target values of the unknown data or it’s used to evaluate the performance of the model (if its for evaluation, the target values of test examples must be known)

• Predictive Model: An abstract representation of the relationship between the predictor and the target variables Predictive Modeling Techniques

Single Models

• Decison Tree Methods (examples below!!!)
• Support Vector Machine/Regression
• Artifical Neural Network

Ensemble Models

• Boosting, Random Forest, Bagging

Lets work with a Decision Tree Classifier

A Classification Technique is a systematic approach to build a Classification models from an input of data!

A decision tree classifier is organized a series of test questions and conditions in a tree structure.

In a decision tree, the root and internal nodes contain attribute test conditions to “separate” records that have different characteristics, all the terminal node assigned a class label yes or no.

How do we Build a Decision Tree?

• It’s called/referred to as decision tree induction

Building an optimal decision tee is key in decision tree classifiers, there are many efficient algorithms that use Greedy approach/strategy. Greedy Algorithms is an algorithm paradigm that builds up a solution piece-by-piece, which makes locally optimum decisions about what attributes to use for dividing the data.

Examples of different Greedy Algorithms:

1. Hunts
2. ID3
3. C4.5
4. CART
5. SPRINT

All 5 of those algorithms are used in Greedy decision tree induction.

How do we Determine the Best Attribute test Condition?

We have to split the “Nodes”. We can do a two-way split or a multi-way split. A two-way split is easy for when there are binary attributes (Yes or No). If there is a Nominal (many names) attributes which could have many values, a test condition can be expressed into a multiway split. For continuous attributes (example of money \$0- 1,000,000), we could split into comparison tests (\$0-100,000, \$100,001-500,000, \$500,001-1,000,000) (< & >). Since there are many ways to specify the test conditions from a given training set, we need a measurement to determine the best way to split the records.

The goal of the best test condition is whether it leads to a homogenous distribution of the nodes, which is the purity of child nodes before and after splitting. The larger the degree of purity, the better the class distribution.

Measurement of impurity can be determined by:

1. Gina Index (used in our example below)
2. Entropy
3. Misclassification Error

## Decision Tree Example Notes:          ### Example of a Decision Tree Classification

• using sklearn in Python

• Using a NBA Draft Combine Information file to predict if the player will be drafted based on picking a few different categories.

Heres the csv file -> file

Steps

``````import pandas as p

data
``````

Output:

Player Year Drafted Height (No Shoes) Height (With Shoes) Wingspan Standing reach Vertical (Max) Vertical (Max Reach) Vertical (No Step) Vertical (No Step Reach) Weight Body Fat Hand (Length) Hand (Width) Bench Agility Sprint
0 Blake Griffin 2009 Yes 80.50 82.00 83.25 105.0 35.5 140.5 32.0 137.0 248.0 8.2 NaN NaN 22.0 10.95 3.28
1 Terrence Williams 2009 Yes 77.00 78.25 81.00 103.5 37.0 140.5 30.5 134.0 213.0 5.1 NaN NaN 9.0 11.15 3.18
2 Gerald Henderson 2009 Yes 76.00 77.00 82.25 102.5 35.0 137.5 31.5 134.0 215.0 4.4 NaN NaN 8.0 11.17 3.14
3 Tyler Hansbrough 2009 Yes 80.25 81.50 83.50 106.0 34.0 140.0 27.5 133.5 234.0 8.5 NaN NaN 18.0 11.12 3.27
4 Earl Clark 2009 Yes 80.50 82.25 86.50 109.5 33.0 142.5 28.5 138.0 228.0 5.2 NaN NaN 5.0 11.17 3.35
512 Peter Jok 2017 No 76.25 77.75 80.00 102.0 31.0 133.0 26.5 128.5 202.0 11.0 8.25 9.50 NaN 11.34 3.41
513 Rawle Alkins 2017 No 74.50 75.75 80.75 99.0 40.5 139.5 31.5 130.5 223.0 11.0 8.75 10.00 NaN 11.99 3.30
514 Sviatoslav Mykhailiuk 2017 No 78.50 79.50 77.00 100.0 33.0 133.0 27.0 127.0 220.0 11.4 8.00 9.25 NaN 12.40 3.53
515 Thomas Welsh 2017 No 83.50 84.50 84.00 109.5 NaN NaN NaN NaN 254.0 10.9 9.00 10.50 NaN NaN NaN
516 V.J. Beachem 2017 No 78.25 80.00 82.25 104.5 37.0 141.5 30.0 134.5 193.0 6.8 8.50 9.00 NaN 11.18 3.26

Step 2 Lets clean this data because there are missing values, here are the columns were intrested in using!

``````feature_columns = ["Height (No Shoes)", "Wingspan", "Standing reach","Vertical (Max)", "Weight", "Agility", "Sprint"]
update_data = data[feature_columns]
update_data
``````

Output:

Height (No Shoes) Wingspan Standing reach Vertical (Max) Weight Agility Sprint
0 80.50 83.25 105.0 35.5 248.0 10.95 3.28
1 77.00 81.00 103.5 37.0 213.0 11.15 3.18
2 76.00 82.25 102.5 35.0 215.0 11.17 3.14
3 80.25 83.50 106.0 34.0 234.0 11.12 3.27
4 80.50 86.50 109.5 33.0 228.0 11.17 3.35
512 76.25 80.00 102.0 31.0 202.0 11.34 3.41
513 74.50 80.75 99.0 40.5 223.0 11.99 3.30
514 78.50 77.00 100.0 33.0 220.0 12.40 3.53
515 83.50 84.00 109.5 NaN 254.0 NaN NaN
516 78.25 82.25 104.5 37.0 193.0 11.18 3.26

Now lets check for missing values!!!

``````update_data.describe()
``````

Output:

Height (No Shoes) Wingspan Standing reach Vertical (Max) Weight Agility Sprint
count 517.000000 517.000000 517.000000 450.000000 516.000000 444.000000 446.000000
mean 77.609284 82.497292 103.275629 35.136667 214.833333 11.330248 3.299664
std 3.287633 3.943068 4.897515 3.561688 24.683537 0.563144 0.128422
min 68.250000 70.000000 88.500000 25.000000 149.000000 10.070000 3.010000
25% 75.250000 79.750000 100.000000 32.500000 196.000000 10.940000 3.200000
50% 77.750000 82.500000 103.500000 35.000000 213.500000 11.255000 3.280000
75% 80.000000 85.500000 107.000000 37.500000 232.000000 11.660000 3.380000
max 85.250000 92.500000 115.000000 44.500000 303.000000 13.440000 3.810000

We can see the count for the total players is 517, and in different columns like Vertical, Weight, Agility, and Sprint here is missing data. Ill just insert the Median value for the missing values.

``````## Vertical Max

update_data['Vertical (Max)'] = update_data['Vertical (Max)'].fillna(update_data['Vertical (Max)'].median())

## Weight

update_data['Weight'] = update_data['Weight'].fillna(update_data['Weight'].median())

## Agility

update_data['Agility'] = update_data['Agility'].fillna(update_data['Agility'].median())

## Sprint

update_data['Sprint'] = update_data['Sprint'].fillna(update_data['Sprint'].median())

``````

Now all the Columns are filled in, no more missing data!

``````update_data
``````

Output:

Height (No Shoes) Wingspan Standing reach Vertical (Max) Weight Agility Sprint
0 80.50 83.25 105.0 35.5 248.0 10.950 3.28
1 77.00 81.00 103.5 37.0 213.0 11.150 3.18
2 76.00 82.25 102.5 35.0 215.0 11.170 3.14
3 80.25 83.50 106.0 34.0 234.0 11.120 3.27
4 80.50 86.50 109.5 33.0 228.0 11.170 3.35
512 76.25 80.00 102.0 31.0 202.0 11.340 3.41
513 74.50 80.75 99.0 40.5 223.0 11.990 3.30
514 78.50 77.00 100.0 33.0 220.0 12.400 3.53
515 83.50 84.00 109.5 35.0 254.0 11.255 3.28
516 78.25 82.25 104.5 37.0 193.0 11.180 3.26

Step 3: Extract the predictor (X) and the Target (y) variables

``````X = update_data
y = data["Drafted"]

``````

Now that we have the data split.

Step 3: Divide the data into training and test sets

``````from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn import metrics
``````
`````` X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
``````

Note: test_size 0.3 means that 70% training and 30% test

Lets look at X_train,X_test, y_train, and y_test

`````` X_train
``````

Output:

Height (No Shoes) Wingspan Standing reach Vertical (Max) Weight Agility Sprint
13 79.75 81.25 106.5 32.5 211.0 11.150 3.28
61 76.75 80.50 104.0 35.5 208.0 11.860 3.19
453 75.25 81.75 97.5 35.5 212.0 10.770 3.45
39 71.25 74.00 95.0 33.0 175.0 10.870 3.10
373 76.50 80.25 100.5 34.5 209.0 11.255 3.27
129 76.00 81.75 101.5 35.5 198.0 11.200 3.09
144 78.50 85.25 106.5 30.5 242.0 12.430 3.46
72 75.75 81.00 101.0 40.0 203.0 11.380 3.15
235 79.50 86.50 106.5 38.0 236.0 11.820 3.52
37 72.50 75.75 97.0 31.0 193.0 10.990 3.22

There are 361 rows in X_train!

`````` X_test
``````

Output:

Height (No Shoes) Wingspan Standing reach Vertical (Max) Weight Agility Sprint
270 73.00 76.00 97.50 35.0 179.0 11.255 3.28
90 78.75 79.75 103.00 34.5 211.0 11.730 3.22
133 82.00 87.75 109.50 36.0 217.0 11.880 3.35
221 81.75 87.50 111.50 35.0 230.0 11.255 3.28
224 76.50 79.00 101.00 37.5 199.0 10.680 3.25
494 75.00 78.50 99.50 35.0 185.0 11.380 3.35
95 76.00 78.75 101.75 32.0 207.0 11.430 3.17
122 79.75 84.25 106.50 30.5 247.0 12.740 3.45
165 75.25 80.50 98.00 36.5 212.0 11.360 3.37
23 80.75 85.00 107.00 35.0 240.0 11.980 3.14

There are 156 rows in the X_test!

``````y_train
``````

Output:

13 Yes
61 Yes
453 No
39 Yes
373 No

129 Yes
144 No
72 Yes
235 Yes
37 Yes
Name: Drafted, Length: 361, dtype: object

There are 361 rows in y_train.

``````y_test
``````

Output:

270 No
90 Yes
133 Yes
221 Yes
224 Yes

494 No
95 No
122 Yes
165 Yes
23 Yes
Name: Drafted, Length: 156, dtype: object

There are 156 rows in y_test.

Step 4: Build the decision tree model and apply it to the test data

``````from sklearn import tree
clf = DecisionTreeClassifier()
clf = clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)
``````

Step 5: Evaluate the accuracy of the model on the test data

``````from sklearn.metrics import accuracy_score

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
``````

Output:

``````Accuracy: 0.6794871794871795
``````

We can Visualize the Decision Tree using Scikit-learn’s export_graphviz function:

``````from sklearn import tree
import pydotplus
dot_data = tree.export_graphviz(clf, out_file=None)
graph = pydotplus.graph_from_dot_data(dot_data)
graph.write_pdf("NBA_Draft_Prediction.pdf")

from IPython.display import Image
Image(graph.create_png())
``````

Output: ## Model Evaluation

• Accuracy

Alternative Measures for Classification

### ROC Curve

What is a ROC Curve? An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. The curve plots two parameters: True Positive Rate (TPR) and False Positive Rate (FPR).

Python Example Confusion Matrix Using the NBA Draft Prediction that we’ve been working with above!!

``````from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred, labels=['No','Yes'])
cm
``````

Output:

``````array([[13, 26],
[27, 90]])
``````

Python Example F Score

F-Measure: (or F-Score) is a measure of a test’s accuracy

``````from sklearn.metrics import f1_score

f1 = f1_score(y_test, y_pred, labels=['No','Yes'], pos_label = "Yes")
f1
``````

Output:

``````0.7725321888412018
``````
``````Y_prob = clf.predict_proba(X_test)
Y_prob
``````

Output:

``````array([[0.33333333, 0.66666667],
[1.        , 0.        ],
[0.33333333, 0.66666667],
...,
[0.88888889, 0.11111111],
[0.33333333, 0.66666667],
[0.33333333, 0.66666667]])
``````
``````import matplotlib.pyplot as plt

from sklearn.metrics import roc_curve, auc
%matplotlib inline

true_labels = (Y_test == 'Yes');

fpr, tpr, thresholds = roc_curve(true_labels, Y_prob[:,1])
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr)

``````

Output: Holdout Method

5-Fold Cross Validation Method

``````from sklearn.model_selection import cross_val_score
clf = tree.DecisionTreeClassifier(max_depth=3)
scores = cross_val_score(clf, X, Y, cv=5)
scores
``````

Output:

``````array([0.72727273, 0.72077922, 0.74025974, 0.7124183 , 0.74509804])
``````
``````print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
``````

Output:

``````Accuracy: 0.73 (+/- 0.02)
``````

### Extras

``````import pandas as p

``````

Output:

compactness circularity distance_circularity radius_ratio pr_axis_aspect_ratio max_length_aspect_ratio scatter_ratio elongatedness pr_axisrectangular lengthrectangular majorvariance minorvariance gyrationradius majorskewness minorskewness minorkurtosis majorkurtosis hollows_ratio class
0 95 43 96 202 65 10 189 35 22 143 217 534 166 71 6 27 190 197 opel
1 96 52 104 222 67 9 198 33 23 163 217 589 226 67 12 20 192 201 opel
2 107 52 101 218 64 11 202 33 23 164 219 610 192 65 17 2 197 206 opel
3 97 37 78 181 62 8 161 41 20 131 182 389 117 62 2 28 203 211 opel
4 96 54 104 175 58 10 215 31 24 175 221 682 222 75 13 23 186 194 opel
``````data[['compactness','circularity','distance_circularity','radius_ratio']].corr()
``````

Output:

compactness 1.000000 0.692869 0.792444 0.691659
circularity 0.692869 1.000000 0.798492 0.622778
distance_circularity 0.792444 0.798492 1.000000 0.771644

Histogram

``````%matplotlib inline
data['compactness'].hist()
`````` Box Plot

``````data[['compactness','circularity','distance_circularity','radius_ratio']].boxplot()
`````` ``````from sklearn.model_selection import train_test_split

Y = data['class']
X = data.drop('class',axis=1)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.23, random_state=1)
``````
``````Y
``````

Output:

0 opel
1 opel
2 opel
3 opel
4 opel

841 van
842 van
843 van
844 van
845 van

# Model Selection and Overfitting (More on Predictive Modeling)

Model Selection

What is Model Overfitting?

• Model Overfitting is error that occur when a function is too closely fit to a limited set of data points

• Too many details and noise

• The goal of Predictive Modeling is to build a model with low training and test errors

• Which is challenging because a model with low training errors does not guarantee it will have low test error (due to model overfitting problem)

An example of Model Overfitting:

``````import numpy as np
import matplotlib.pyplot as plt
from numpy.random import random

%matplotlib inline

N = 1500

mean1 = [6, 14]
mean2 = [10, 6]
mean3 = [14, 14]
cov = [[3.5, 0], [0, 3.5]]  # diagonal covariance

np.random.seed(50)
x1, y1 = np.random.multivariate_normal(mean1, cov, N//6).T
x2, y2 = np.random.multivariate_normal(mean2, cov, N//6).T
x3, y3 = np.random.multivariate_normal(mean3, cov, N//6).T
x4 = 20*np.random.random(N//2)
y4 = 20*np.random.random(N//2)

plt.plot(x1,y1,'ro',x2,y2,'ro',x3,y3,'ro',x4,y4,'g+',ms=4)
``````

Output: ``````from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.8, random_state=1)

from sklearn import tree
from sklearn.metrics import accuracy_score

maxdepths = [2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50]

trainAcc = np.zeros(len(maxdepths))
testAcc = np.zeros(len(maxdepths))

index = 0
for depth in maxdepths:
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
trainAcc[index] = accuracy_score(Y_train, Y_predTrain)
testAcc[index] = accuracy_score(Y_test, Y_predTest)
index += 1

plt.plot(maxdepths,trainAcc,'ro-',maxdepths,testAcc,'bv--')
plt.legend(['Training Accuracy','Test Accuracy'])
plt.xlabel('Max depth')
plt.ylabel('Accuracy')

``````

Output: You can see the plot above shows the training accuracy will continue to imporive as the “maximum depth” becomes more complex. However, the test accuracy initally improves up to a maximum depth of 5, before it sharply decreases due to model overfitting.

## Building Trees of Different Sizes

``````from sklearn import tree
from sklearn.metrics import accuracy_score

maxdepths = [2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50]

trainAcc = np.zeros(len(maxdepths))
testAcc = np.zeros(len(maxdepths))
index = 0

for depth in maxdepths:
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
trainAcc[index] = accuracy_score(Y_train, Y_predTrain)
testAcc[index] = accuracy_score(Y_test, Y_predTest)
index += 1
``````

Lets plot a Tree with MaxDepth = 4

``````depth = 4
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
AccTrain = accuracy_score(Y_train, Y_predTrain)
AccTest = accuracy_score(Y_test, Y_predTest)
[AccTrain, AccTest]
``````

Output:

``````[0.8888888888888888, 0.6959349593495935]
``````

What do these numbers tell us?

• With a tree of depth 4, our training accuracy was 88.88% which is pretty solid. Once we tested our model on our test data, the accuracy was 69.59% which is also solid.

Lets Plot the Tree with Depth = 4

``````import pydotplus
from IPython.display import Image

dot_data = tree.export_graphviz(clf, out_file=None)
graph = pydotplus.graph_from_dot_data(dot_data)
Image(graph.create_png())
``````

Output: Lets plot a Tree with MaxDepth = 20

A tree with a MaxDepth of 20 should be very acccurate for the training data set. But how about when we test our model on the test data??

``````depth = 20
clf = tree.DecisionTreeClassifier(max_depth=depth)
clf = clf.fit(X_train, Y_train)
Y_predTrain = clf.predict(X_train)
Y_predTest = clf.predict(X_test)
AccTrain = accuracy_score(Y_train, Y_predTrain)
AccTest = accuracy_score(Y_test, Y_predTest)
[AccTrain, AccTest]
``````

Output:

``````[1.0, 0.640650406504065]
``````

Accuracy Training = 100%

Accuracy Test = 64.07%

As you can see our training data was 100% accurate, but it was so overfitted aka too complex, that the test data accuracy was lower than the tree with a max depth of 4!

• The model may fit exceptional cases (rather than normal) in the training data!!

Lets Plot the Tree with Depth = 20

``````import pydotplus
from IPython.display import Image

dot_data = tree.export_graphviz(clf, out_file=None)
graph = pydotplus.graph_from_dot_data(dot_data)
Image(graph.create_png())
`````` ### Lets Plot Model Overfitting and Underfitting

``````plt.plot(maxdepths,trainAcc,'ro-',maxdepths,testAcc,'bv--')
plt.legend(['Training Accuracy','Test Accuracy'])
``````

Output:

Model Overfitting and Underfitting Defintions:

• Underfitting: When the model is too simple

• Overfitting: When the model is too complex

### Model Selection Using Validation Set

• A general model selection approach that does not depend on the predicitive modeling technique used

• It divides the training data into two parts:

• Training Set: for model building
• Validation set: for estimating generalization error

A drawback is that less data would be available for training

Lets do an example of Model Selection Using Validation Set!

## Nearest-Neighbor Methods

What is Nearest-Neighbors?

• One of the simple/basic classification algorithms in Machine Learning.
• Supervised Learning
• Most People use K Nearest Neighbors

• Linear Method
• Nonlinear Method
• Ensemble Methods

## K Nearest Neighbors

• Common algorithm used to solve classification model problems K Nearest Neighbors in Sklearn

• First opened the .data file and added a “header row” with the columns:

``````import numpy as np
from sklearn import preprocessing, neighbors
from sklearn.model_selection import cross_validate

import pandas as pd

# Replace missing data!
df.replace('?',-99999, inplace=True)
## Drop ID, completly useless
df.drop(['id'], 1, inplace=True)

df
``````

Output:

clump_thickness uniform_cell_size uniform_cell_shape marginal_adhesion single_epi_cell_size bare_nuclei bland_chromation normal_nucleoli mitoses class
0 5 1 1 1 2 1 3 1 1 2
1 5 4 4 5 7 10 3 2 1 2
2 3 1 1 1 2 2 3 1 1 2
3 6 8 8 1 3 4 3 7 1 2
4 4 1 1 3 2 1 3 1 1 2
694 3 1 1 1 3 2 1 1 1 2
695 2 1 1 1 2 1 1 1 1 2
696 5 10 10 3 7 3 8 10 2 4
697 4 8 6 4 3 4 10 6 1 4
698 4 8 8 5 4 5
``````X = np.array(df.drop(['class'],1))
y = np.array(df['class'])

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2)

clf = neighbors.KNeighborsClassifier()
clf.fit(X_train, y_train)

accuracy = clf.score(X_test, y_test)
accuracy
``````

Output:

``````0.9785714285714285
``````

``````example_measures = np.array([4,2,1,1,1,2,3,2,1])
example_measures = example_measures.reshape(1, -1)
prediction = clf.predict(example_measures)

``````

Output:

``````array()
``````

### Ensemble Methods

• Ensemble Methods is taking the orginal training data
1. Creating Multiple Data Sets
2. Build Multiple Base Classifiers
3. Combine the Base Classifiers

Why use Ensemble models?

• Better accuracy
• Higher Consistency
• Reduces bias and variance errors

When and Where to use Ensemble Models?

• When single model overfits (decision trees usually overfit)

What are the different Ensemble Methods?

• Bagging: Build multiple classifiers by resampling the data with replacement
• Boosting: Build multiple classifiers by resampling the (weighted) data with replacement
• Random Forests: Train multiple decision tree classifiers and combine their prediction

Python Example