# Practical Machine Learning with R and Python – Part 5

This is the 5th and probably penultimate part of my series on ‘Practical Machine Learning with R and Python’. The earlier parts of this series included

1. Practical Machine Learning with R and Python – Part 1 In this initial post, I touch upon univariate, multivariate, polynomial regression and KNN regression in R and Python
2.Practical Machine Learning with R and Python – Part 2 In this post, I discuss Logistic Regression, KNN classification and cross validation error for both LOOCV and K-Fold in both R and Python
3.Practical Machine Learning with R and Python – Part 3 This post covered ‘feature selection’ in Machine Learning. Specifically I touch best fit, forward fit, backward fit, ridge(L2 regularization) & lasso (L1 regularization). The post includes equivalent code in R and Python.
4.Practical Machine Learning with R and Python – Part 4 In this part I discussed SVMs, Decision Trees, validation, precision recall, and roc curves

This post ‘Practical Machine Learning with R and Python – Part 5’ discusses regression with B-splines, natural splines, smoothing splines, generalized additive models (GAMS), bagging, random forest and boosting

As with my previous posts in this series, this post is largely based on the following 2 MOOC courses

1. Statistical Learning, Prof Trevor Hastie & Prof Robert Tibesherani, Online Stanford
2. Applied Machine Learning in Python Prof Kevyn-Collin Thomson, University Of Michigan, Coursera

You can download this R Markdown file and associated data files from Github at MachineLearning-RandPython-Part5

Check out my compact and minimal book  “Practical Machine Learning with R and Python:Second edition- Machine Learning in stereo”  available in Amazon in paperback($10.99) and kindle($7.99) versions. My book includes implementations of key ML algorithms and associated measures and metrics. The book is ideal for anybody who is familiar with the concepts and would like a quick reference to the different ML algorithms that can be applied to problems and how to select the best model. Pick your copy today!!

For this part I have used the data sets from UCI Machine Learning repository(Communities and Crime and Auto MPG)

## 1. Splines

When performing regression (continuous or logistic) between a target variable and a feature (or a set of features), a single polynomial for the entire range of the data set usually does not perform a good fit.Rather we would need to provide we could fit
regression curves for different section of the data set.

There are several techniques which do this for e.g. piecewise-constant functions, piecewise-linear functions, piecewise-quadratic/cubic/4th order polynomial functions etc. One such set of functions are the cubic splines which fit cubic polynomials to successive sections of the dataset. The points where the cubic splines join, are called ‘knots’.

Since each section has a different cubic spline, there could be discontinuities (or breaks) at these knots. To prevent these discontinuities ‘natural splines’ and ‘smoothing splines’ ensure that the seperate cubic functions have 2nd order continuity at these knots with the adjacent splines. 2nd order continuity implies that the value, 1st order derivative and 2nd order derivative at these knots are equal.

A cubic spline with knots $\alpha_{k}$ , k=1,2,3,..K is a piece-wise cubic polynomial with continuous derivative up to order 2 at each knot. We can write $y_{i} = \beta_{0} +\beta_{1}b_{1}(x_{i}) +\beta_{2}b_{2}(x_{i}) + .. + \beta_{K+3}b_{K+3}(x_{i}) + \epsilon_{i}$.
For each ($x{i},y{i}$), $b_{i}$ are called ‘basis’ functions, where  $b_{1}(x_{i})=x_{i}$$b_{2}(x_{i})=x_{i}^2$, $b_{3}(x_{i})=x_{i}^3$, $b_{k+3}(x_{i})=(x_{i} -\alpha_{k})^3$ where k=1,2,3… K The 1st and 2nd derivatives of cubic splines are continuous at the knots. Hence splines provide a smooth continuous fit to the data by fitting different splines to different sections of the data

## 1.1a Fit a 4th degree polynomial – R code

In the code below a non-linear function (a 4th order polynomial) is used to fit the data. Usually when we fit a single polynomial to the entire data set the tails of the fit tend to vary a lot particularly if there are fewer points at the ends. Splines help in reducing this variation at the extremities

library(dplyr)
library(ggplot2)
source('RFunctions-1.R')
df=read.csv("auto_mpg.csv",stringsAsFactors = FALSE) # Data from UCI
df1 <- as.data.frame(sapply(df,as.numeric))
#Select specific columns
df2 <- df1 %>% dplyr::select(cylinder,displacement, horsepower,weight, acceleration, year,mpg)
auto <- df2[complete.cases(df2),]
# Fit a 4th degree polynomial
fit=lm(mpg~poly(horsepower,4),data=auto)
#Display a summary of fit
summary(fit)
##
## Call:
## lm(formula = mpg ~ poly(horsepower, 4), data = auto)
##
## Residuals:
##      Min       1Q   Median       3Q      Max
## -14.8820  -2.5802  -0.1682   2.2100  16.1434
##
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)
## (Intercept)            23.4459     0.2209 106.161   <2e-16 ***
## poly(horsepower, 4)1 -120.1377     4.3727 -27.475   <2e-16 ***
## poly(horsepower, 4)2   44.0895     4.3727  10.083   <2e-16 ***
## poly(horsepower, 4)3   -3.9488     4.3727  -0.903    0.367
## poly(horsepower, 4)4   -5.1878     4.3727  -1.186    0.236
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.373 on 387 degrees of freedom
## Multiple R-squared:  0.6893, Adjusted R-squared:  0.6861
## F-statistic: 214.7 on 4 and 387 DF,  p-value: < 2.2e-16
#Get the range of horsepower
hp <- range(auto$horsepower) #Create a sequence to be used for plotting hpGrid <- seq(hp[1],hp[2],by=10) #Predict for these values of horsepower. Set Standard error as TRUE pred=predict(fit,newdata=list(horsepower=hpGrid),se=TRUE) #Compute bands on either side that is 2xSE seBands=cbind(pred$fit+2*pred$se.fit,pred$fit-2*pred$se.fit) #Plot the fit with Standard Error bands plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower", ylab="MPG", main="Polynomial of degree 4") lines(hpGrid,pred$fit,lwd=2,col="blue")
matlines(hpGrid,seBands,lwd=2,col="blue",lty=3)

## 1.1b Fit a 4th degree polynomial – Python code

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
# Select columns
autoDF1=autoDF[['mpg','cylinder','displacement','horsepower','weight','acceleration','year']]
# Convert all columns to numeric
autoDF2 = autoDF1.apply(pd.to_numeric, errors='coerce')

#Drop NAs
autoDF3=autoDF2.dropna()
autoDF3.shape
X=autoDF3[['horsepower']]
y=autoDF3['mpg']
#Create a polynomial of degree 4
poly = PolynomialFeatures(degree=4)
X_poly = poly.fit_transform(X)

# Fit a polynomial regression line
linreg = LinearRegression().fit(X_poly, y)
# Create a range of values
hpGrid = np.arange(np.min(X),np.max(X),10)
hp=hpGrid.reshape(-1,1)
# Transform to 4th degree
poly = PolynomialFeatures(degree=4)
hp_poly = poly.fit_transform(hp)

#Create a scatter plot
plt.scatter(X,y)
# Fit the prediction
ypred=linreg.predict(hp_poly)
plt.title("Poylnomial of degree 4")
fig2=plt.xlabel("Horsepower")
fig2=plt.ylabel("MPG")
# Draw the regression curve
plt.plot(hp,ypred,c="red")
plt.savefig('fig1.png', bbox_inches='tight')

## 1.1c Fit a B-Spline – R Code

In the code below a B- Spline is fit to data. The B-spline requires the manual selection of knots

#Splines
library(splines)
# Fit a B-spline to the data. Select knots at 60,75,100,150
fit=lm(mpg~bs(horsepower,df=6,knots=c(60,75,100,150)),data=auto)
# Use the fitted regresion to predict
pred=predict(fit,newdata=list(horsepower=hpGrid),se=T)
# Create a scatter plot
plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower",
ylab="MPG", main="B-Spline with 4 knots")
#Draw lines with 2 Standard Errors on either side
lines(hpGrid,pred$fit,lwd=2) lines(hpGrid,pred$fit+2*pred$se,lty="dashed") lines(hpGrid,pred$fit-2*pred$se,lty="dashed") abline(v=c(60,75,100,150),lty=2,col="darkgreen") ## 1.1d Fit a Natural Spline – R Code Here a ‘Natural Spline’ is used to fit .The Natural Spline extrapolates beyond the boundary knots and the ends of the function are much more constrained than a regular spline or a global polynomoial where the ends can wag a lot more. Natural splines do not require the explicit selection of knots # There is no need to select the knots here. There is a smoothing parameter which # can be specified by the degrees of freedom 'df' parameter. The natural spline fit2=lm(mpg~ns(horsepower,df=4),data=auto) pred=predict(fit2,newdata=list(horsepower=hpGrid),se=T) plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower", ylab="MPG", main="Natural Splines") lines(hpGrid,pred$fit,lwd=2)
lines(hpGrid,pred$fit+2*pred$se,lty="dashed")
lines(hpGrid,pred$fit-2*pred$se,lty="dashed")

## 1.1.e Fit a Smoothing Spline – R code

Here a smoothing spline is used. Smoothing splines also do not require the explicit setting of knots. We can change the ‘degrees of freedom(df)’ paramater to get the best fit

# Smoothing spline has a smoothing parameter, the degrees of freedom
# This is too wiggly
plot(auto$horsepower,auto$mpg,xlim=hp,cex=.5,col="black",xlab="Horsepower",
ylab="MPG", main="Smoothing Splines")

# Here df is set to 16. This has a lot of variance
fit=smooth.spline(auto$horsepower,auto$mpg,df=16)
lines(fit,col="red",lwd=2)

# We can use Cross Validation to allow the spline to pick the value of this smpopothing paramter. We do not need to set the degrees of freedom 'df'
fit=smooth.spline(auto$horsepower,auto$mpg,cv=TRUE)
lines(fit,col="blue",lwd=2)

## 1.1e Splines – Python

There isn’t as much treatment of splines in Python and SKLearn. I did find the LSQUnivariate, UnivariateSpline spline. The LSQUnivariate spline requires the explcit setting of knots

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from scipy.interpolate import LSQUnivariateSpline
autoDF.shape
autoDF.columns
autoDF1=autoDF[['mpg','cylinder','displacement','horsepower','weight','acceleration','year']]
autoDF2 = autoDF1.apply(pd.to_numeric, errors='coerce')
auto=autoDF2.dropna()
auto=auto[['horsepower','mpg']].sort_values('horsepower')

# Set the knots manually
knots=[65,75,100,150]
# Create an array for X & y
X=np.array(auto['horsepower'])
y=np.array(auto['mpg'])
# Fit a LSQunivariate spline
s = LSQUnivariateSpline(X,y,knots)

#Plot the spline
xs = np.linspace(40,230,1000)
ys = s(xs)
plt.scatter(X, y)
plt.plot(xs, ys)
plt.savefig('fig2.png', bbox_inches='tight')


## 1.2 Generalized Additiive models (GAMs)

Generalized Additive Models (GAMs) is a really powerful ML tool.

$y_{i} = \beta_{0} + f_{1}(x_{i1}) + f_{2}(x_{i2}) + .. +f_{p}(x_{ip}) + \epsilon_{i}$

In GAMs we use a different functions for each of the variables. GAMs give a much better fit since we can choose any function for the different sections

## 1.2a Generalized Additive Models (GAMs) – R Code

The plot below show the smooth spline that is fit for each of the features horsepower, cylinder, displacement, year and acceleration. We can use any function for example loess, 4rd order polynomial etc.

library(gam)
# Fit a smoothing spline for horsepower, cyliner, displacement and acceleration
gam=gam(mpg~s(horsepower,4)+s(cylinder,5)+s(displacement,4)+s(year,4)+s(acceleration,5),data=auto)
# Display the summary of the fit. This give the significance of each of the paramwetr
# Also an ANOVA is given for each combination of the features
summary(gam)
##
## Call: gam(formula = mpg ~ s(horsepower, 4) + s(cylinder, 5) + s(displacement,
##     4) + s(year, 4) + s(acceleration, 5), data = auto)
## Deviance Residuals:
##     Min      1Q  Median      3Q     Max
## -8.3190 -1.4436 -0.0261  1.2279 12.0873
##
## (Dispersion Parameter for gaussian family taken to be 6.9943)
##
##     Null Deviance: 23818.99 on 391 degrees of freedom
## Residual Deviance: 2587.881 on 370 degrees of freedom
## AIC: 1898.282
##
## Number of Local Scoring Iterations: 3
##
## Anova for Parametric Effects
##                     Df  Sum Sq Mean Sq  F value    Pr(>F)
## s(horsepower, 4)     1 15632.8 15632.8 2235.085 < 2.2e-16 ***
## s(cylinder, 5)       1   508.2   508.2   72.666 3.958e-16 ***
## s(displacement, 4)   1   374.3   374.3   53.514 1.606e-12 ***
## s(year, 4)           1  2263.2  2263.2  323.583 < 2.2e-16 ***
## s(acceleration, 5)   1   372.4   372.4   53.246 1.809e-12 ***
## Residuals          370  2587.9     7.0
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Anova for Nonparametric Effects
##                    Npar Df Npar F     Pr(F)
## (Intercept)
## s(horsepower, 4)         3 13.825 1.453e-08 ***
## s(cylinder, 5)           3 17.668 9.712e-11 ***
## s(displacement, 4)       3 44.573 < 2.2e-16 ***
## s(year, 4)               3 23.364 7.183e-14 ***
## s(acceleration, 5)       4  3.848  0.004453 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
par(mfrow=c(2,3))
plot(gam,se=TRUE)

## 1.2b Generalized Additive Models (GAMs) – Python Code

I did not find the equivalent of GAMs in SKlearn in Python. There was an early prototype (2012) in Github. Looks like it is still work in progress or has probably been abandoned.

## 1.3 Tree based Machine Learning Models

Tree based Machine Learning are all based on the ‘bootstrapping’ technique. In bootstrapping given a sample of size N, we create datasets of size N by sampling this original dataset with replacement. Machine Learning models are built on the different bootstrapped samples and then averaged.

Decision Trees as seen above have the tendency to overfit. There are several techniques that help to avoid this namely a) Bagging b) Random Forests c) Boosting

### Bagging, Random Forest and Gradient Boosting

Bagging: Bagging, or Bootstrap Aggregation decreases the variance of predictions, by creating separate Decisiion Tree based ML models on the different samples and then averaging these ML models

Random Forests: Bagging is a greedy algorithm and tries to produce splits based on all variables which try to minimize the error. However the different ML models have a high correlation. Random Forests remove this shortcoming, by using a variable and random set of features to split on. Hence the features chosen and the resulting trees are uncorrelated. When these ML models are averaged the performance is much better.

Boosting: Gradient Boosted Decision Trees also use an ensemble of trees but they don’t build Machine Learning models with random set of features at each step. Rather small and simple trees are built. Successive trees try to minimize the error from the earlier trees.

Out of Bag (OOB) Error: In Random Forest and Gradient Boosting for each bootstrap sample taken from the dataset, there will be samples left out. These are known as Out of Bag samples.Classification accuracy carried out on these OOB samples is known as OOB error

## 1.31a Decision Trees – R Code

The code below creates a Decision tree with the cancer training data. The summary of the fit is output. Based on the ML model, the predict function is used on test data and a confusion matrix is output.

# Read the cancer data
library(tree)
library(caret)
library(e1071)
cancer <- cancer[,2:32]
cancer$target <- as.factor(cancer$target)
train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5)
train <- cancer[train_idx, ]
test <- cancer[-train_idx, ]

# Create Decision Tree
cancerStatus=tree(target~.,train)
summary(cancerStatus)
##
## Classification tree:
## tree(formula = target ~ ., data = train)
## Variables actually used in tree construction:
## [1] "worst.perimeter"      "worst.concave.points" "area.error"
## [4] "worst.texture"        "mean.texture"         "mean.concave.points"
## Number of terminal nodes:  9
## Residual mean deviance:  0.1218 = 50.8 / 417
## Misclassification error rate: 0.02347 = 10 / 426
pred <- predict(cancerStatus,newdata=test,type="class")
confusionMatrix(pred,test$target) ## Confusion Matrix and Statistics ## ## Reference ## Prediction 0 1 ## 0 49 7 ## 1 8 78 ## ## Accuracy : 0.8944 ## 95% CI : (0.8318, 0.9397) ## No Information Rate : 0.5986 ## P-Value [Acc > NIR] : 4.641e-15 ## ## Kappa : 0.7795 ## Mcnemar's Test P-Value : 1 ## ## Sensitivity : 0.8596 ## Specificity : 0.9176 ## Pos Pred Value : 0.8750 ## Neg Pred Value : 0.9070 ## Prevalence : 0.4014 ## Detection Rate : 0.3451 ## Detection Prevalence : 0.3944 ## Balanced Accuracy : 0.8886 ## ## 'Positive' Class : 0 ##  # Plot decision tree with labels plot(cancerStatus) text(cancerStatus,pretty=0) ## 1.31b Decision Trees – Cross Validation – R Code We can also perform a Cross Validation on the data to identify the Decision Tree which will give the minimum deviance. library(tree) cancer <- read.csv("cancer.csv",stringsAsFactors = FALSE) cancer <- cancer[,2:32] cancer$target <- as.factor(cancer$target) train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5) train <- cancer[train_idx, ] test <- cancer[-train_idx, ] # Create Decision Tree cancerStatus=tree(target~.,train) # Execute 10 fold cross validation cvCancer=cv.tree(cancerStatus) plot(cvCancer) # Plot the plot(cvCancer$size,cvCancer$dev,type='b') prunedCancer=prune.tree(cancerStatus,best=4) plot(prunedCancer) text(prunedCancer,pretty=0) pred <- predict(prunedCancer,newdata=test,type="class") confusionMatrix(pred,test$target)
## Confusion Matrix and Statistics
##
##           Reference
## Prediction  0  1
##          0 50  7
##          1  7 78
##
##                Accuracy : 0.9014
##                  95% CI : (0.8401, 0.945)
##     No Information Rate : 0.5986
##     P-Value [Acc > NIR] : 7.988e-16
##
##                   Kappa : 0.7948
##  Mcnemar's Test P-Value : 1
##
##             Sensitivity : 0.8772
##             Specificity : 0.9176
##          Pos Pred Value : 0.8772
##          Neg Pred Value : 0.9176
##              Prevalence : 0.4014
##          Detection Rate : 0.3521
##    Detection Prevalence : 0.4014
##       Balanced Accuracy : 0.8974
##
##        'Positive' Class : 0
## 

## 1.31c Decision Trees – Python Code

Below is the Python code for creating Decision Trees. The accuracy, precision, recall and F1 score is computed on the test data set.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification, make_blobs
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import graphviz

(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)

X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
random_state = 0)
clf = DecisionTreeClassifier().fit(X_train, y_train)

print('Accuracy of Decision Tree classifier on training set: {:.2f}'
.format(clf.score(X_train, y_train)))
print('Accuracy of Decision Tree classifier on test set: {:.2f}'
.format(clf.score(X_test, y_test)))

y_predicted=clf.predict(X_test)
confusion = confusion_matrix(y_test, y_predicted)
print('Accuracy: {:.2f}'.format(accuracy_score(y_test, y_predicted)))
print('Precision: {:.2f}'.format(precision_score(y_test, y_predicted)))
print('Recall: {:.2f}'.format(recall_score(y_test, y_predicted)))
print('F1: {:.2f}'.format(f1_score(y_test, y_predicted)))

# Plot the Decision Tree
clf = DecisionTreeClassifier(max_depth=2).fit(X_train, y_train)
dot_data = tree.export_graphviz(clf, out_file=None,
feature_names=cancer.feature_names,
class_names=cancer.target_names,
filled=True, rounded=True,
special_characters=True)
graph = graphviz.Source(dot_data)
graph
## Accuracy of Decision Tree classifier on training set: 1.00
## Accuracy of Decision Tree classifier on test set: 0.87
## Accuracy: 0.87
## Precision: 0.97
## Recall: 0.82
## F1: 0.89

## 1.31d Decision Trees – Cross Validation – Python Code

In the code below 5-fold cross validation is performed for different depths of the tree and the accuracy is computed. The accuracy on the test set seems to plateau when the depth is 8. But it is seen to increase again from 10 to 12. More analysis needs to be done here


import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
from sklearn.cross_validation import train_test_split, KFold
def computeCVAccuracy(X,y,folds):
accuracy=[]
foldAcc=[]
depth=[1,2,3,4,5,6,7,8,9,10,11,12]
nK=len(X)/float(folds)
xval_err=0
for i in depth:
kf = KFold(len(X),n_folds=folds)
for train_index, test_index in kf:
X_train, X_test = X.iloc[train_index], X.iloc[test_index]
y_train, y_test = y.iloc[train_index], y.iloc[test_index]
clf = DecisionTreeClassifier(max_depth = i).fit(X_train, y_train)
score=clf.score(X_test, y_test)
accuracy.append(score)

foldAcc.append(np.mean(accuracy))

return(foldAcc)

cvAccuracy=computeCVAccuracy(pd.DataFrame(X_cancer),pd.DataFrame(y_cancer),folds=10)

df1=pd.DataFrame(cvAccuracy)
df1.columns=['cvAccuracy']
df=df1.reindex([1,2,3,4,5,6,7,8,9,10,11,12])
df.plot()
plt.title("Decision Tree - 10-fold Cross Validation Accuracy vs Depth of tree")
plt.xlabel("Depth of tree")
plt.ylabel("Accuracy")
plt.savefig('fig3.png', bbox_inches='tight')

## 1.4a Random Forest – R code

A Random Forest is fit using the Boston data. The summary shows that 4 variables were randomly chosen at each split and the resulting ML model explains 88.72% of the test data. Also the variable importance is plotted. It can be seen that ‘rooms’ and ‘status’ are the most influential features in the model

library(randomForest)
df=read.csv("Boston.csv",stringsAsFactors = FALSE) # Data from MASS - SL

# Select specific columns
Boston <- df %>% dplyr::select("crimeRate","zone","indus","charles","nox","rooms","age",                          "distances","highways","tax","teacherRatio","color",
"status","medianValue")

# Fit a Random Forest on the Boston training data
rfBoston=randomForest(medianValue~.,data=Boston)
# Display the summatu of the fit. It can be seen that the MSE is 10.88
# and the percentage variance explained is 86.14%. About 4 variables were tried at each # #split for a maximum tree of 500.
# The MSE and percent variance is on Out of Bag trees
rfBoston
##
## Call:
##  randomForest(formula = medianValue ~ ., data = Boston)
##                Type of random forest: regression
##                      Number of trees: 500
## No. of variables tried at each split: 4
##
##           Mean of squared residuals: 9.521672
##                     % Var explained: 88.72
#List and plot the variable importances
importance(rfBoston)
##              IncNodePurity
## crimeRate        2602.1550
## zone              258.8057
## indus            2599.6635
## charles           240.2879
## nox              2748.8485
## rooms           12011.6178
## age              1083.3242
## distances        2432.8962
## highways          393.5599
## tax              1348.6987
## teacherRatio     2841.5151
## color             731.4387
## status          12735.4046
varImpPlot(rfBoston)

## 1.4b Random Forest-OOB and Cross Validation Error – R code

The figure below shows the OOB error and the Cross Validation error vs the ‘mtry’. Here mtry indicates the number of random features that are chosen at each split. The lowest test error occurs when mtry = 8

library(randomForest)
df=read.csv("Boston.csv",stringsAsFactors = FALSE) # Data from MASS - SL

# Select specific columns
Boston <- df %>% dplyr::select("crimeRate","zone","indus","charles","nox","rooms","age",                          "distances","highways","tax","teacherRatio","color",
"status","medianValue")
# Split as training and tst sets
train_idx <- trainTestSplit(Boston,trainPercent=75,seed=5)
train <- Boston[train_idx, ]
test <- Boston[-train_idx, ]

#Initialize OOD and testError
oobError <- NULL
testError <- NULL
# In the code below the number of variables to consider at each split is increased
# from 1 - 13(max features) and the OOB error and the MSE is computed
for(i in 1:13){
fitRF=randomForest(medianValue~.,data=train,mtry=i,ntree=400)
oobError[i] <-fitRF$mse[400] pred <- predict(fitRF,newdata=test) testError[i] <- mean((pred-test$medianValue)^2)
}

# We can see the OOB and Test Error. It can be seen that the Random Forest performs
# best with the lowers MSE at mtry=6
matplot(1:13,cbind(testError,oobError),pch=19,col=c("red","blue"),
type="b",xlab="mtry(no of varaibles at each split)", ylab="Mean Squared Error",
main="Random Forest - OOB and Test Error")
legend("topright",legend=c("OOB","Test"),pch=19,col=c("red","blue"))

## 1.4c Random Forest – Python code

The python code for Random Forest Regression is shown below. The training and test score is computed. The variable importance shows that ‘rooms’ and ‘status’ are the most influential of the variables

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax',
'teacherRatio','color','status']]
y=df['medianValue']

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

regr = RandomForestRegressor(max_depth=4, random_state=0)
regr.fit(X_train, y_train)

print('R-squared score (training): {:.3f}'
.format(regr.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
.format(regr.score(X_test, y_test)))

feature_names=['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax',
'teacherRatio','color','status']
print(regr.feature_importances_)
plt.figure(figsize=(10,6),dpi=80)
c_features=X_train.shape[1]
plt.barh(np.arange(c_features),regr.feature_importances_)
plt.xlabel("Feature importance")
plt.ylabel("Feature name")

plt.yticks(np.arange(c_features), feature_names)
plt.tight_layout()

plt.savefig('fig4.png', bbox_inches='tight')

## R-squared score (training): 0.917
## R-squared score (test): 0.734
## [ 0.03437382  0.          0.00580335  0.          0.00731004  0.36461548
##   0.00638577  0.03432173  0.0041244   0.01732328  0.01074148  0.0012638
##   0.51373683]

## 1.4d Random Forest – Cross Validation and OOB Error – Python code

As with R the ‘max_features’ determines the random number of features the random forest will use at each split. The plot shows that when max_features=8 the MSE is lowest

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score

X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax',
'teacherRatio','color','status']]
y=df['medianValue']

cvError=[]
oobError=[]
oobMSE=[]
for i in range(1,13):
regr = RandomForestRegressor(max_depth=4, n_estimators=400,max_features=i,oob_score=True,random_state=0)
mse= np.mean(cross_val_score(regr, X, y, cv=5,scoring = 'neg_mean_squared_error'))
# Since this is neg_mean_squared_error I have inverted the sign to get MSE
cvError.append(-mse)
# Fit on all data to compute OOB error
regr.fit(X, y)
# Record the OOB error for each max_features=i setting
oob = 1 - regr.oob_score_
oobError.append(oob)
# Get the Out of Bag prediction
oobPred=regr.oob_prediction_
# Compute the Mean Squared Error between OOB Prediction and target
mseOOB=np.mean(np.square(oobPred-y))
oobMSE.append(mseOOB)

# Plot the CV Error and OOB Error
# Set max_features
maxFeatures=np.arange(1,13)
cvError=pd.DataFrame(cvError,index=maxFeatures)
oobMSE=pd.DataFrame(oobMSE,index=maxFeatures)
#Plot
fig8=df.plot()
fig8=plt.title('Random forest - CV Error and OOB Error vs max_features')
fig8.figure.savefig('fig8.png', bbox_inches='tight')

#Plot the OOB Error vs max_features
plt.plot(range(1,13),oobError)
fig2=plt.title("Random Forest - OOB Error vs max_features (variable no of features)")
fig2=plt.xlabel("max_features (variable no of features)")
fig2=plt.ylabel("OOB Error")
fig2.figure.savefig('fig7.png', bbox_inches='tight')


## 1.5a Boosting – R code

Here a Gradient Boosted ML Model is built with a n.trees=5000, with a learning rate of 0.01 and depth of 4. The feature importance plot also shows that rooms and status are the 2 most important features. The MSE vs the number of trees plateaus around 2000 trees

library(gbm)
# Perform gradient boosting on the Boston data set. The distribution is gaussian since we
# doing MSE. The interaction depth specifies the number of splits
boostBoston=gbm(medianValue~.,data=train,distribution="gaussian",n.trees=5000,
shrinkage=0.01,interaction.depth=4)
#The summary gives the variable importance. The 2 most significant variables are
# number of rooms and lower status
summary(boostBoston)

##                       var    rel.inf
## rooms               rooms 42.2267200
## status             status 27.3024671
## distances       distances  7.9447972
## crimeRate       crimeRate  5.0238827
## nox                   nox  4.0616548
## teacherRatio teacherRatio  3.1991999
## age                   age  2.7909772
## color               color  2.3436295
## tax                   tax  2.1386213
## charles           charles  1.3799109
## highways         highways  0.7644026
## indus               indus  0.7236082
## zone                 zone  0.1001287
# The plots below show how each variable relates to the median value of the home. As
# the number of roomd increase the median value increases and with increase in lower status
# the median value decreases
par(mfrow=c(1,2))
#Plot the relation between the top 2 features and the target
plot(boostBoston,i="rooms")
plot(boostBoston,i="status")

# Create a sequence of trees between 100-5000 incremented by 50
nTrees=seq(100,5000,by=50)
# Predict the values for the test data
pred <- predict(boostBoston,newdata=test,n.trees=nTrees)
# Compute the mean for each of the MSE for each of the number of trees
boostError <- apply((pred-test$medianValue)^2,2,mean) #Plot the MSE vs the number of trees plot(nTrees,boostError,pch=19,col="blue",ylab="Mean Squared Error", main="Boosting Test Error") ## 1.5b Cross Validation Boosting – R code Included below is a cross validation error vs the learning rate. The lowest error is when learning rate = 0.09 cvError <- NULL s <- c(.001,0.01,0.03,0.05,0.07,0.09,0.1) for(i in seq_along(s)){ cvBoost=gbm(medianValue~.,data=train,distribution="gaussian",n.trees=5000, shrinkage=s[i],interaction.depth=4,cv.folds=5) cvError[i] <- mean(cvBoost$cv.error)
}

# Create a data frame for plotting
a <- rbind(s,cvError)
b <- as.data.frame(t(a))
# It can be seen that a shrinkage parameter of 0,05 gives the lowes CV Error
ggplot(b,aes(s,cvError)) + geom_point() + geom_line(color="blue") +
xlab("Shrinkage") + ylab("Cross Validation Error") +
ggtitle("Gradient boosted trees - Cross Validation error vs Shrinkage")

## 1.5c Boosting – Python code

A gradient boost ML model in Python is created below. The Rsquared score is computed on the training and test data.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax',
'teacherRatio','color','status']]
y=df['medianValue']

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

regr.fit(X_train, y_train)

print('R-squared score (training): {:.3f}'
.format(regr.score(X_train, y_train)))
print('R-squared score (test): {:.3f}'
.format(regr.score(X_test, y_test)))
## R-squared score (training): 0.983
## R-squared score (test): 0.821

## 1.5c Cross Validation Boosting – Python code

the cross validation error is computed as the learning rate is varied. The minimum CV eror occurs when lr = 0.04

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score

X=df[['crimeRate','zone', 'indus','charles','nox','rooms', 'age','distances','highways','tax',
'teacherRatio','color','status']]
y=df['medianValue']

cvError=[]
learning_rate =[.001,0.01,0.03,0.05,0.07,0.09,0.1]
for lr in learning_rate:
mse= np.mean(cross_val_score(regr, X, y, cv=10,scoring = 'neg_mean_squared_error'))
# Since this is neg_mean_squared_error I have inverted the sign to get MSE
cvError.append(-mse)
learning_rate =[.001,0.01,0.03,0.05,0.07,0.09,0.1]
plt.plot(learning_rate,cvError)
plt.title("Gradient Boosting - 5-fold CV- Mean Squared Error vs max_features (variable no of features)")
plt.xlabel("max_features (variable no of features)")
plt.ylabel("Mean Squared Error")
plt.savefig('fig6.png', bbox_inches='tight')

Conclusion This post covered Splines and Tree based ML models like Bagging, Random Forest and Boosting. Stay tuned for further updates.

You may also like

To see all posts see Index of posts

# Practical Machine Learning with R and Python – Part 4

This is the 4th installment of my ‘Practical Machine Learning with R and Python’ series. In this part I discuss classification with Support Vector Machines (SVMs), using both a Linear and a Radial basis kernel, and Decision Trees. Further, a closer look is taken at some of the metrics associated with binary classification, namely accuracy vs precision and recall. I also touch upon Validation curves, Precision-Recall, ROC curves and AUC with equivalent code in R and Python

This post is a continuation of my 3 earlier posts on Practical Machine Learning in R and Python
1. Practical Machine Learning with R and Python – Part 1
2. Practical Machine Learning with R and Python – Part 2
3. Practical Machine Learning with R and Python – Part 3

The RMarkdown file with the code and the associated data files can be downloaded from Github at MachineLearning-RandPython-Part4

Check out my compact and minimal book  “Practical Machine Learning with R and Python:Second edition- Machine Learning in stereo”  available in Amazon in paperback($10.99) and kindle($7.99) versions. My book includes implementations of key ML algorithms and associated measures and metrics. The book is ideal for anybody who is familiar with the concepts and would like a quick reference to the different ML algorithms that can be applied to problems and how to select the best model. Pick your copy today!!

Support Vector Machines (SVM) are another useful Machine Learning model that can be used for both regression and classification problems. SVMs used in classification, compute the hyperplane, that separates the 2 classes with the maximum margin. To do this the features may be transformed into a larger multi-dimensional feature space. SVMs can be used with different kernels namely linear, polynomial or radial basis to determine the best fitting model for a given classification problem.

In the 2nd part of this series Practical Machine Learning with R and Python – Part 2, I had mentioned the various metrics that are used in classification ML problems namely Accuracy, Precision, Recall and F1 score. Accuracy gives the fraction of data that were correctly classified as belonging to the +ve or -ve class. However ‘accuracy’ in itself is not a good enough measure because it does not take into account the fraction of the data that were incorrectly classified. This issue becomes even more critical in different domains. For e.g a surgeon who would like to detect cancer, would like to err on the side of caution, and classify even a possibly non-cancerous patient as possibly having cancer, rather than mis-classifying a malignancy as benign. Here we would like to increase recall or sensitivity which is  given by Recall= TP/(TP+FN) or we try reduce mis-classification by either increasing the (true positives) TP or reducing (false negatives) FN

On the other hand, search algorithms would like to increase precision which tries to reduce the number of irrelevant results in the search result. Precision= TP/(TP+FP). In other words we do not want ‘false positives’ or irrelevant results to come in the search results and there is a need to reduce the false positives.

When we try to increase ‘precision’, we do so at the cost of ‘recall’, and vice-versa. I found this diagram and explanation in Wikipedia very useful Source: Wikipedia

“Consider a brain surgeon tasked with removing a cancerous tumor from a patient’s brain. The surgeon needs to remove all of the tumor cells since any remaining cancer cells will regenerate the tumor. Conversely, the surgeon must not remove healthy brain cells since that would leave the patient with impaired brain function. The surgeon may be more liberal in the area of the brain she removes to ensure she has extracted all the cancer cells. This decision increases recall but reduces precision. On the other hand, the surgeon may be more conservative in the brain she removes to ensure she extracts only cancer cells. This decision increases precision but reduces recall. That is to say, greater recall increases the chances of removing healthy cells (negative outcome) and increases the chances of removing all cancer cells (positive outcome). Greater precision decreases the chances of removing healthy cells (positive outcome) but also decreases the chances of removing all cancer cells (negative outcome).”

## 1.1a. Linear SVM – R code

In R code below I use SVM with linear kernel

source('RFunctions-1.R')
library(dplyr)
library(e1071)
library(caret)
library(reshape2)
library(ggplot2)
# Read data. Data from SKLearn
cancer$target <- as.factor(cancer$target)

# Split into training and test sets
train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5)
train <- cancer[train_idx, ]
test <- cancer[-train_idx, ]

# Fit a linear basis kernel. DO not scale the data
svmfit=svm(target~., data=train, kernel="linear",scale=FALSE)
ypred=predict(svmfit,test)
#Print a confusion matrix
confusionMatrix(ypred,test$target) ## Confusion Matrix and Statistics ## ## Reference ## Prediction 0 1 ## 0 54 3 ## 1 3 82 ## ## Accuracy : 0.9577 ## 95% CI : (0.9103, 0.9843) ## No Information Rate : 0.5986 ## P-Value [Acc > NIR] : <2e-16 ## ## Kappa : 0.9121 ## Mcnemar's Test P-Value : 1 ## ## Sensitivity : 0.9474 ## Specificity : 0.9647 ## Pos Pred Value : 0.9474 ## Neg Pred Value : 0.9647 ## Prevalence : 0.4014 ## Detection Rate : 0.3803 ## Detection Prevalence : 0.4014 ## Balanced Accuracy : 0.9560 ## ## 'Positive' Class : 0 ##  ## 1.1b Linear SVM – Python code The code below creates a SVM with linear basis in Python and also dumps the corresponding classification metrics import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.svm import LinearSVC from sklearn.datasets import make_classification, make_blobs from sklearn.metrics import confusion_matrix from matplotlib.colors import ListedColormap from sklearn.datasets import load_breast_cancer # Load the cancer data (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) clf = LinearSVC().fit(X_train, y_train) print('Breast cancer dataset') print('Accuracy of Linear SVC classifier on training set: {:.2f}' .format(clf.score(X_train, y_train))) print('Accuracy of Linear SVC classifier on test set: {:.2f}' .format(clf.score(X_test, y_test))) ## Breast cancer dataset ## Accuracy of Linear SVC classifier on training set: 0.92 ## Accuracy of Linear SVC classifier on test set: 0.94 ## 1.2 Dummy classifier Often when we perform classification tasks using any ML model namely logistic regression, SVM, neural networks etc. it is very useful to determine how well the ML model performs agains at dummy classifier. A dummy classifier uses some simple computation like frequency of majority class, instead of fitting and ML model. It is essential that our ML model does much better that the dummy classifier. This problem is even more important in imbalanced classes where we have only about 10% of +ve samples. If any ML model we create has a accuracy of about 0.90 then it is evident that our classifier is not doing any better than a dummy classsfier which can just take a majority count of this imbalanced class and also come up with 0.90. We need to be able to do better than that. In the examples below (1.3a & 1.3b) it can be seen that SVMs with ‘radial basis’ kernel with unnormalized data, for both R and Python, do not perform any better than the dummy classifier. ## 1.2a Dummy classifier – R code R does not seem to have an explicit dummy classifier. I created a simple dummy classifier that predicts the majority class. SKlearn in Python also includes other strategies like uniform, stratified etc. but this should be possible to create in R also. # Create a simple dummy classifier that computes the ratio of the majority class to the totla DummyClassifierAccuracy <- function(train,test,type="majority"){ if(type=="majority"){ count <- sum(train$target==1)/dim(train)[1]
}
count
}

cancer$target <- as.factor(cancer$target)

# Create training and test sets
train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5)
train <- cancer[train_idx, ]
test <- cancer[-train_idx, ]

#Dummy classifier majority class
acc=DummyClassifierAccuracy(train,test)
sprintf("Accuracy is %f",acc)
## [1] "Accuracy is 0.638498"

## 1.2b Dummy classifier – Python code

This dummy classifier uses the majority class.

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.metrics import confusion_matrix
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
random_state = 0)

# Negative class (0) is most frequent
dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train)
y_dummy_predictions = dummy_majority.predict(X_test)

print('Dummy classifier accuracy on test set: {:.2f}'
.format(dummy_majority.score(X_test, y_test)))

## Dummy classifier accuracy on test set: 0.63

## 1.3a – Radial SVM (un-normalized) – R code

SVMs perform better when the data is normalized or scaled. The 2 examples below show that SVM with radial basis kernel does not perform any better than the dummy classifier

library(dplyr)
library(e1071)
library(caret)
library(reshape2)
library(ggplot2)

train_idx <- trainTestSplit(cancer,trainPercent=75,seed=5)
train <- cancer[train_idx, ]
test <- cancer[-train_idx, ]
# Unnormalized data
ypred=predict(svmfit,test)
confusionMatrix(ypred,test$target) ## Confusion Matrix and Statistics ## ## Reference ## Prediction 0 1 ## 0 0 0 ## 1 57 85 ## ## Accuracy : 0.5986 ## 95% CI : (0.5131, 0.6799) ## No Information Rate : 0.5986 ## P-Value [Acc > NIR] : 0.5363 ## ## Kappa : 0 ## Mcnemar's Test P-Value : 1.195e-13 ## ## Sensitivity : 0.0000 ## Specificity : 1.0000 ## Pos Pred Value : NaN ## Neg Pred Value : 0.5986 ## Prevalence : 0.4014 ## Detection Rate : 0.0000 ## Detection Prevalence : 0.0000 ## Balanced Accuracy : 0.5000 ## ## 'Positive' Class : 0 ##  ## 1.4b – Radial SVM (un-normalized) – Python code import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.svm import SVC # Load the cancer data (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) clf = SVC(C=10).fit(X_train, y_train) print('Breast cancer dataset (unnormalized features)') print('Accuracy of RBF-kernel SVC on training set: {:.2f}' .format(clf.score(X_train, y_train))) print('Accuracy of RBF-kernel SVC on test set: {:.2f}' .format(clf.score(X_test, y_test))) ## Breast cancer dataset (unnormalized features) ## Accuracy of RBF-kernel SVC on training set: 1.00 ## Accuracy of RBF-kernel SVC on test set: 0.63 ## 1.5a – Radial SVM (Normalized) -R Code The data is scaled (normalized ) before using the SVM model. The SVM model has 2 paramaters a) C – Large C (less regularization), more regularization b) gamma – Small gamma has larger decision boundary with more misclassfication, and larger gamma has tighter decision boundary The R code below computes the accuracy as the regularization paramater is changed trainingAccuracy <- NULL testAccuracy <- NULL C1 <- c(.01,.1, 1, 10, 20) for(i in C1){ svmfit=svm(target~., data=train, kernel="radial",cost=i,scale=TRUE) ypredTrain <-predict(svmfit,train) ypredTest=predict(svmfit,test) a <-confusionMatrix(ypredTrain,train$target)
b <-confusionMatrix(ypredTest,test$target) trainingAccuracy <-c(trainingAccuracy,a$overall[1])
testAccuracy <-c(testAccuracy,b$overall[1]) } print(trainingAccuracy) ## Accuracy Accuracy Accuracy Accuracy Accuracy ## 0.6384977 0.9671362 0.9906103 0.9976526 1.0000000 print(testAccuracy) ## Accuracy Accuracy Accuracy Accuracy Accuracy ## 0.5985915 0.9507042 0.9647887 0.9507042 0.9507042 a <-rbind(C1,as.numeric(trainingAccuracy),as.numeric(testAccuracy)) b <- data.frame(t(a)) names(b) <- c("C1","trainingAccuracy","testAccuracy") df <- melt(b,id="C1") ggplot(df) + geom_line(aes(x=C1, y=value, colour=variable),size=2) + xlab("C (SVC regularization)value") + ylab("Accuracy") + ggtitle("Training and test accuracy vs C(regularization)") ## 1.5b – Radial SVM (normalized) – Python The Radial basis kernel is used on normalized data for a range of ‘C’ values and the result is plotted. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.svm import SVC from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler() # Load the cancer data (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) print('Breast cancer dataset (normalized with MinMax scaling)') trainingAccuracy=[] testAccuracy=[] for C1 in [.01,.1, 1, 10, 20]: clf = SVC(C=C1).fit(X_train_scaled, y_train) acctrain=clf.score(X_train_scaled, y_train) accTest=clf.score(X_test_scaled, y_test) trainingAccuracy.append(acctrain) testAccuracy.append(accTest) # Create a dataframe C1=[.01,.1, 1, 10, 20] trainingAccuracy=pd.DataFrame(trainingAccuracy,index=C1) testAccuracy=pd.DataFrame(testAccuracy,index=C1) # Plot training and test R squared as a function of alpha df=pd.concat([trainingAccuracy,testAccuracy],axis=1) df.columns=['trainingAccuracy','trainingAccuracy'] fig1=df.plot() fig1=plt.title('Training and test accuracy vs C (SVC)') fig1.figure.savefig('fig1.png', bbox_inches='tight') ## Breast cancer dataset (normalized with MinMax scaling) Output image: ## 1.6a Validation curve – R code Sklearn includes code creating validation curves by varying paramaters and computing and plotting accuracy as gamma or C or changd. I did not find this R but I think this is a useful function and so I have created the R equivalent of this. # The R equivalent of np.logspace seqLogSpace <- function(start,stop,len){ a=seq(log10(10^start),log10(10^stop),length=len) 10^a } # Read the data. This is taken the SKlearn cancer data cancer <- read.csv("cancer.csv") cancer$target <- as.factor(cancer$target) set.seed(6) # Create the range of C1 in log space param_range = seqLogSpace(-3,2,20) # Initialize the overall training and test accuracy to NULL overallTrainAccuracy <- NULL overallTestAccuracy <- NULL # Loop over the parameter range of Gamma for(i in param_range){ # Set no of folds noFolds=5 # Create the rows which fall into different folds from 1..noFolds folds = sample(1:noFolds, nrow(cancer), replace=TRUE) # Initialize the training and test accuracy of folds to 0 trainingAccuracy <- 0 testAccuracy <- 0 # Loop through the folds for(j in 1:noFolds){ # The training is all rows for which the row is != j (k-1 folds -> training) train <- cancer[folds!=j,] # The rows which have j as the index become the test set test <- cancer[folds==j,] # Create a SVM model for this svmfit=svm(target~., data=train, kernel="radial",gamma=i,scale=TRUE) # Add up all the fold accuracy for training and test separately ypredTrain <-predict(svmfit,train) ypredTest=predict(svmfit,test) # Create confusion matrix a <-confusionMatrix(ypredTrain,train$target)
b <-confusionMatrix(ypredTest,test$target) # Get the accuracy trainingAccuracy <-trainingAccuracy + a$overall[1]
testAccuracy <-testAccuracy+b$overall[1] } # Compute the average of accuracy for K folds for number of features 'i' overallTrainAccuracy=c(overallTrainAccuracy,trainingAccuracy/noFolds) overallTestAccuracy=c(overallTestAccuracy,testAccuracy/noFolds) } #Create a dataframe a <- rbind(param_range,as.numeric(overallTrainAccuracy), as.numeric(overallTestAccuracy)) b <- data.frame(t(a)) names(b) <- c("C1","trainingAccuracy","testAccuracy") df <- melt(b,id="C1") #Plot in log axis ggplot(df) + geom_line(aes(x=C1, y=value, colour=variable),size=2) + xlab("C (SVC regularization)value") + ylab("Accuracy") + ggtitle("Training and test accuracy vs C(regularization)") + scale_x_log10() ## 1.6b Validation curve – Python Compute and plot the validation curve as gamma is varied. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import MinMaxScaler from sklearn.svm import SVC from sklearn.model_selection import validation_curve # Load the cancer data (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) scaler = MinMaxScaler() X_scaled = scaler.fit_transform(X_cancer) # Create a gamma values from 10^-3 to 10^2 with 20 equally spaced intervals param_range = np.logspace(-3, 2, 20) # Compute the validation curve train_scores, test_scores = validation_curve(SVC(), X_scaled, y_cancer, param_name='gamma', param_range=param_range, cv=10) #Plot the figure fig2=plt.figure() #Compute the mean train_scores_mean = np.mean(train_scores, axis=1) train_scores_std = np.std(train_scores, axis=1) test_scores_mean = np.mean(test_scores, axis=1) test_scores_std = np.std(test_scores, axis=1) fig2=plt.title('Validation Curve with SVM') fig2=plt.xlabel('$\gamma$(gamma)') fig2=plt.ylabel('Score') fig2=plt.ylim(0.0, 1.1) lw = 2 fig2=plt.semilogx(param_range, train_scores_mean, label='Training score', color='darkorange', lw=lw) fig2=plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color='darkorange', lw=lw) fig2=plt.semilogx(param_range, test_scores_mean, label='Cross-validation score', color='navy', lw=lw) fig2=plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color='navy', lw=lw) fig2.figure.savefig('fig2.png', bbox_inches='tight')  Output image: ## 1.7a Validation Curve (Preventing data leakage) – Python code In this course Applied Machine Learning in Python, the Professor states that when we apply the same data transformation to a entire dataset, it will cause a data leakage. “The proper way to do cross-validation when you need to scale the data is not to scale the entire dataset with a single transform, since this will indirectly leak information into the training data about the whole dataset, including the test data (see the lecture on data leakage later in the course). Instead, scaling/normalizing must be computed and applied for each cross-validation fold separately” So I apply separate scaling to the training and testing folds and plot. In the lecture the Prof states that this can be done using pipelines. import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.cross_validation import KFold from sklearn.preprocessing import MinMaxScaler from sklearn.svm import SVC # Read the data (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) # Set the parameter range param_range = np.logspace(-3, 2, 20) # Set number of folds folds=5 #Initialize overallTrainAccuracy=[] overallTestAccuracy=[] # Loop over the paramater range for c in param_range: trainingAccuracy=0 testAccuracy=0 kf = KFold(len(X_cancer),n_folds=folds) # Partition into training and test folds for train_index, test_index in kf: # Partition the data acccording the fold indices generated X_train, X_test = X_cancer[train_index], X_cancer[test_index] y_train, y_test = y_cancer[train_index], y_cancer[test_index] # Scale the X_train and X_test scaler = MinMaxScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # Fit a SVC model for each C clf = SVC(C=c).fit(X_train_scaled, y_train) #Compute the training and test score acctrain=clf.score(X_train_scaled, y_train) accTest=clf.score(X_test_scaled, y_test) trainingAccuracy += np.sum(acctrain) testAccuracy += np.sum(accTest) # Compute the mean training and testing accuracy overallTrainAccuracy.append(trainingAccuracy/folds) overallTestAccuracy.append(testAccuracy/folds) overallTrainAccuracy=pd.DataFrame(overallTrainAccuracy,index=param_range) overallTestAccuracy=pd.DataFrame(overallTestAccuracy,index=param_range) # Plot training and test R squared as a function of alpha df=pd.concat([overallTrainAccuracy,overallTestAccuracy],axis=1) df.columns=['trainingAccuracy','testAccuracy'] fig3=plt.title('Validation Curve with SVM') fig3=plt.xlabel('$\gamma$(gamma)') fig3=plt.ylabel('Score') fig3=plt.ylim(0.5, 1.1) lw = 2 fig3=plt.semilogx(param_range, overallTrainAccuracy, label='Training score', color='darkorange', lw=lw) fig3=plt.semilogx(param_range, overallTestAccuracy, label='Cross-validation score', color='navy', lw=lw) fig3=plt.legend(loc='best') fig3.figure.savefig('fig3.png', bbox_inches='tight')  Output image: ## 1.8 a Decision trees – R code Decision trees in R can be plotted using RPart package library(rpart) library(rpart.plot) rpart = NULL # Create a decision tree m <-rpart(Species~.,data=iris) #Plot rpart.plot(m,extra=2,main="Decision Tree - IRIS") ## 1.8 b Decision trees – Python code from sklearn.datasets import load_iris from sklearn.tree import DecisionTreeClassifier from sklearn import tree from sklearn.model_selection import train_test_split import graphviz iris = load_iris() X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state = 3) clf = DecisionTreeClassifier().fit(X_train, y_train) print('Accuracy of Decision Tree classifier on training set: {:.2f}' .format(clf.score(X_train, y_train))) print('Accuracy of Decision Tree classifier on test set: {:.2f}' .format(clf.score(X_test, y_test))) dot_data = tree.export_graphviz(clf, out_file=None, feature_names=iris.feature_names, class_names=iris.target_names, filled=True, rounded=True, special_characters=True) graph = graphviz.Source(dot_data) graph ## Accuracy of Decision Tree classifier on training set: 1.00 ## Accuracy of Decision Tree classifier on test set: 0.97 ## 1.9a Feature importance – R code I found the following code which had a snippet for feature importance. Sklean has a nice method for this. For some reason the results in R and Python are different. Any thoughts? set.seed(3) # load the library library(mlbench) library(caret) # load the dataset cancer <- read.csv("cancer.csv") cancer$target <- as.factor(cancer$target) # Split as data data <- cancer[,1:31] target <- cancer[,32] # Train the model model <- train(data, target, method="rf", preProcess="scale", trControl=trainControl(method = "cv")) # Compute variable importance importance <- varImp(model) # summarize importance print(importance) # plot importance plot(importance) ## 1.9b Feature importance – Python code import numpy as np import pandas as pd import os import matplotlib.pyplot as plt from sklearn.tree import DecisionTreeClassifier from sklearn.model_selection import train_test_split from sklearn.datasets import load_breast_cancer import numpy as np # Read the data cancer= load_breast_cancer() (X_cancer, y_cancer) = load_breast_cancer(return_X_y = True) X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = 0) # Use the DecisionTreClassifier clf = DecisionTreeClassifier(max_depth = 4, min_samples_leaf = 8, random_state = 0).fit(X_train, y_train) c_features=len(cancer.feature_names) print('Breast cancer dataset: decision tree') print('Accuracy of DT classifier on training set: {:.2f}' .format(clf.score(X_train, y_train))) print('Accuracy of DT classifier on test set: {:.2f}' .format(clf.score(X_test, y_test))) # Plot the feature importances fig4=plt.figure(figsize=(10,6),dpi=80) fig4=plt.barh(range(c_features), clf.feature_importances_) fig4=plt.xlabel("Feature importance") fig4=plt.ylabel("Feature name") fig4=plt.yticks(np.arange(c_features), cancer.feature_names) fig4=plt.tight_layout() plt.savefig('fig4.png', bbox_inches='tight')  ## Breast cancer dataset: decision tree ## Accuracy of DT classifier on training set: 0.96 ## Accuracy of DT classifier on test set: 0.94 Output image: ## 1.10a Precision-Recall, ROC curves & AUC- R code I tried several R packages for plotting the Precision and Recall and AUC curve. PRROC seems to work well. The Precision-Recall curves show the tradeoff between precision and recall. The higher the precision, the lower the recall and vice versa.AUC curves that hug the top left corner indicate a high sensitivity,specificity and an excellent accuracy. source("RFunctions-1.R") library(dplyr) library(caret) library(e1071) library(PRROC) # Read the data (this data is from sklearn!) d <- read.csv("digits.csv") digits <- d[2:66] digits$X64 <- as.factor(digits$X64) # Split as training and test sets train_idx <- trainTestSplit(digits,trainPercent=75,seed=5) train <- digits[train_idx, ] test <- digits[-train_idx, ] # Fit a SVM model with linear basis kernel with probabilities svmfit=svm(X64~., data=train, kernel="linear",scale=FALSE,probability=TRUE) ypred=predict(svmfit,test,probability=TRUE) head(attr(ypred,"probabilities")) ## 0 1 ## 6 7.395947e-01 2.604053e-01 ## 8 9.999998e-01 1.842555e-07 ## 12 1.655178e-05 9.999834e-01 ## 13 9.649997e-01 3.500032e-02 ## 15 9.994849e-01 5.150612e-04 ## 16 9.999987e-01 1.280700e-06 # Store the probability of 0s and 1s m0<-attr(ypred,"probabilities")[,1] m1<-attr(ypred,"probabilities")[,2] # Create a dataframe of scores scores <- data.frame(m1,test$X64)

# Class 0 is data points of +ve class (in this case, digit 1) and -ve class (digit 0)
#Compute Precision Recall
pr <- pr.curve(scores.class0=scores[scores$test.X64=="1",]$m1,
scores.class1=scores[scores$test.X64=="0",]$m1,
curve=T)

# Plot precision-recall curve
plot(pr)

#Plot the ROC curve
roc<-roc.curve(m0, m1,curve=TRUE)
plot(roc)

## 1.10b Precision-Recall, ROC curves & AUC- Python code

For Python Logistic Regression is used to plot Precision Recall, ROC curve and compute AUC

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import roc_curve, auc
X, y = dataset.data, dataset.target
#Create 2 classes -i) Digit 1 (from digit 1) ii) Digit 0 (from all other digits)
# Make a copy of the target
z= y.copy()
# Replace all non 1's as 0
z[z != 1] = 0

X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=0)
# Fit a LR model
lr = LogisticRegression().fit(X_train, y_train)

#Compute the decision scores
y_scores_lr = lr.fit(X_train, y_train).decision_function(X_test)
y_score_list = list(zip(y_test[0:20], y_scores_lr[0:20]))

#Show the decision_function scores for first 20 instances
y_score_list

precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
#Plot
plt.figure()
plt.xlim([0.0, 1.01])
plt.ylim([0.0, 1.01])
plt.plot(precision, recall, label='Precision-Recall Curve')
plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3)
plt.xlabel('Precision', fontsize=16)
plt.ylabel('Recall', fontsize=16)
plt.axes().set_aspect('equal')
plt.savefig('fig5.png', bbox_inches='tight')

#Compute and plot the ROC
y_score_lr = lr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)

plt.figure()
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])
plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
plt.axes()
plt.savefig('fig6.png', bbox_inches='tight')


output

## 1.10c Precision-Recall, ROC curves & AUC- Python code

In the code below classification probabilities are used to compute and plot precision-recall, roc and AUC

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV

X, y = dataset.data, dataset.target
# Make a copy of the target
z= y.copy()
# Replace all non 1's as 0
z[z != 1] = 0

X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=0)
svm = LinearSVC()
# Need to use CalibratedClassifierSVC to redict probabilities for lInearSVC
clf = CalibratedClassifierCV(svm)
clf.fit(X_train, y_train)
y_proba_lr = clf.predict_proba(X_test)
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, y_proba_lr[:,1])
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
#plt.figure(figsize=(15,15),dpi=80)
plt.figure()
plt.xlim([0.0, 1.01])
plt.ylim([0.0, 1.01])
plt.plot(precision, recall, label='Precision-Recall Curve')
plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3)
plt.xlabel('Precision', fontsize=16)
plt.ylabel('Recall', fontsize=16)
plt.axes().set_aspect('equal')
plt.savefig('fig7.png', bbox_inches='tight')

# Analysis of IPL T20 matches with yorkr templates

## Introduction

In this post I create RMarkdown templates for end-to-end analysis of IPL T20 matches, that are available on Cricsheet based on my R package yorkr.  With these templates you can convert all IPL data which is in yaml format to R dataframes. Further I create data and the necessary templates for analyzing IPL matches, teams and players. All of these can be accessed at yorkrIPLTemplate.

If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at $12.99 for the paperback and$4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr and Beaten by sheer pace-Cricket analytics with yorkr A must read for any cricket lover! Check it out!! 9/Rs 320 and$6.99/Rs448 respectively

The templates are

1. Template for conversion and setup – IPLT20Template.Rmd
2. Any IPL match – IPLMatchtemplate.Rmd
3. IPL matches between 2 nations – IPLMatches2TeamTemplate.Rmd
4. A IPL nations performance against all other IPL nations – IPLAllMatchesAllOppnTemplate.Rmd
5. Analysis of IPL batsmen and bowlers of all IPL nations – IPLBatsmanBowlerTemplate.Rmd

Besides the templates the repository also includes the converted data for all IPL matches I downloaded from Cricsheet in Dec 2016. So this data is complete till the 2016 IPL season. You can recreate the files as more matches are added to Cricsheet site in IPL 2017 and future seasons. This post contains all the steps needed for detailed analysis of IPL matches, teams and IPL player. This will also be my reference in future if I decide to analyze IPL in future!

See my earlier posts where I analyze IPL T20
1. yorkr crashes the IPL party ! – Part 1
2. yorkr crashes the IPL party! – Part 2
3. yorkr crashes the IPL party! – Part 3!
4. yorkr crashes the IPL party! – Part 4

There will be 5 folders at the root

1. IPLdata – Match files as yaml from Cricsheet
2. IPLMatches – Yaml match files converted to dataframes
3. IPLMatchesBetween2Teams – All Matches between any 2 IPL teams
4. allMatchesAllOpposition – An IPL teams’s performance against all other teams
5. BattingBowlingDetails – Batting and bowling details of all IPL teams
library(yorkr)
library(dplyr)

The first few steps take care of the data setup. This needs to be done before any of the analysis of IPL batsmen, bowlers, any IPL match, matches between any 2 IPL countries or analysis of a teams performance against all other countries

There will be 5 folders at the root

1. data
2. IPLMatches
3. IPLMatchesBetween2Teams
4. allMatchesAllOpposition
5. BattingBowlingDetails

# 1.Create directory of IPLMatches

Some files may give conversions errors. You could try to debug the problem or just remove it from the IPLdata folder. At most 2-4 file will have conversion problems and I usally remove then from the files to be converted.

Also take a look at my GooglyPlus shiny app which was created after performing the same conversion on the Dec 16 data .

convertAllYaml2RDataframesT20("data","IPLMatches")

### 2.Save all matches between all combinations of IPL nations

This function will create the set of all matches between each IPL team against every other IPL team. This uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function.

setwd("./IPLMatchesBetween2Teams")
saveAllMatchesBetween2IPLTeams("../IPLMatches")

### 3.Save all matches against all opposition

This will create a consolidated dataframe of all matches played by every IPL playing nation against all other nattions. This also uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function.

setwd("../allMatchesAllOpposition")
saveAllMatchesAllOppositionIPLT20("../IPLMatches")

### 4. Create batting and bowling details for each IPL team

These are the current IPL playing teams. You can add to this vector as newer IPL teams start playing IPL. You will get to know all IPL teams by also look at the directory created above namely allMatchesAllOpposition. This also uses the data that was created in IPLMatches, with the convertAllYaml2RDataframesIPL() function.

setwd("../BattingBowlingDetails")
ipl_teams <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings XI Punjab",
"Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors",
"Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
"Rising Pune Supergiants")

for(i in seq_along(ipl_teams)){
print(ipl_teams[i])
val <- paste(ipl_teams[i],"-details",sep="")
val <- getTeamBattingDetails(ipl_teams[i],dir="../IPLMatches", save=TRUE)

}

for(i in seq_along(ipl_teams)){
print(ipl_teams[i])
val <- paste(ipl_teams[i],"-details",sep="")
val <- getTeamBowlingDetails(ipl_teams[i],dir="../IPLMatches", save=TRUE)

}

### 5. Get the list of batsmen for a particular IPL team

The following code is needed for analyzing individual IPL batsmen. In IPL a player could have played in multiple IPL teams.

getBatsmen <- function(df){
bmen <- df %>% distinct(batsman)
bmen <- as.character(bmen$batsman) batsmen <- sort(bmen) } load("Chennai Super Kings-BattingDetails.RData") csk_details <- battingDetails load("Deccan Chargers-BattingDetails.RData") dc_details <- battingDetails load("Delhi Daredevils-BattingDetails.RData") dd_details <- battingDetails load("Kings XI Punjab-BattingDetails.RData") kxip_details <- battingDetails load("Kochi Tuskers Kerala-BattingDetails.RData") ktk_details <- battingDetails load("Kolkata Knight Riders-BattingDetails.RData") kkr_details <- battingDetails load("Mumbai Indians-BattingDetails.RData") mi_details <- battingDetails load("Pune Warriors-BattingDetails.RData") pw_details <- battingDetails load("Rajasthan Royals-BattingDetails.RData") rr_details <- battingDetails load("Royal Challengers Bangalore-BattingDetails.RData") rcb_details <- battingDetails load("Sunrisers Hyderabad-BattingDetails.RData") sh_details <- battingDetails load("Gujarat Lions-BattingDetails.RData") gl_details <- battingDetails load("Rising Pune Supergiants-BattingDetails.RData") rps_details <- battingDetails #Get the batsmen for each IPL team csk_batsmen <- getBatsmen(csk_details) dc_batsmen <- getBatsmen(dc_details) dd_batsmen <- getBatsmen(dd_details) kxip_batsmen <- getBatsmen(kxip_details) ktk_batsmen <- getBatsmen(ktk_details) kkr_batsmen <- getBatsmen(kkr_details) mi_batsmen <- getBatsmen(mi_details) pw_batsmen <- getBatsmen(pw_details) rr_batsmen <- getBatsmen(rr_details) rcb_batsmen <- getBatsmen(rcb_details) sh_batsmen <- getBatsmen(sh_details) gl_batsmen <- getBatsmen(gl_details) rps_batsmen <- getBatsmen(rps_details) # Save the dataframes save(csk_batsmen,file="csk.RData") save(dc_batsmen, file="dc.RData") save(dd_batsmen, file="dd.RData") save(kxip_batsmen, file="kxip.RData") save(ktk_batsmen, file="ktk.RData") save(kkr_batsmen, file="kkr.RData") save(mi_batsmen , file="mi.RData") save(pw_batsmen, file="pw.RData") save(rr_batsmen, file="rr.RData") save(rcb_batsmen, file="rcb.RData") save(sh_batsmen, file="sh.RData") save(gl_batsmen, file="gl.RData") save(rps_batsmen, file="rps.RData") ### 6. Get the list of bowlers for a particular IPL team The method below can get the list of bowler names for any IPL team.The following code is needed for analyzing individual IPL bowlers. In IPL a player could have played in multiple IPL teams. getBowlers <- function(df){ bwlr <- df %>% distinct(bowler) bwlr <- as.character(bwlr$bowler)
bowler <- sort(bwlr)
}

csk_details <- bowlingDetails
dc_details <- bowlingDetails
dd_details <- bowlingDetails
kxip_details <- bowlingDetails
ktk_details <- bowlingDetails
kkr_details <- bowlingDetails
mi_details <- bowlingDetails
pw_details <- bowlingDetails
rr_details <- bowlingDetails
rcb_details <- bowlingDetails
sh_details <- bowlingDetails
gl_details <- bowlingDetails
rps_details <- bowlingDetails

# Get the bowlers for each team
csk_bowlers <- getBowlers(csk_details)
dc_bowlers <- getBowlers(dc_details)
dd_bowlers <- getBowlers(dd_details)
kxip_bowlers <- getBowlers(kxip_details)
ktk_bowlers <- getBowlers(ktk_details)
kkr_bowlers <- getBowlers(kkr_details)
mi_bowlers <- getBowlers(mi_details)
pw_bowlers <- getBowlers(pw_details)
rr_bowlers <- getBowlers(rr_details)
rcb_bowlers <- getBowlers(rcb_details)
sh_bowlers <- getBowlers(sh_details)
gl_bowlers <- getBowlers(gl_details)
rps_bowlers <- getBowlers(rps_details)

#Save the dataframes
save(csk_bowlers,file="csk1.RData")
save(dc_bowlers, file="dc1.RData")
save(dd_bowlers, file="dd1.RData")
save(kxip_bowlers, file="kxip1.RData")
save(ktk_bowlers, file="ktk1.RData")
save(kkr_bowlers, file="kkr1.RData")
save(mi_bowlers , file="mi1.RData")
save(pw_bowlers, file="pw1.RData")
save(rr_bowlers, file="rr1.RData")
save(rcb_bowlers, file="rcb1.RData")
save(sh_bowlers, file="sh1.RData")
save(gl_bowlers, file="gl1.RData")
save(rps_bowlers, file="rps1.RData")

### 1 IPL Match Analysis

Load any match data from the ./IPLMatches folder for e.g. Chennai Super Kings-Deccan Chargers-2008-05-06.RData

setwd("./IPLMatches")
csk_dc<- overs
#The steps are
IPLTeam1_IPLTeam2 <- overs

All analysis for this match can be done now

### 2. Scorecard

teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 3.Batting Partnerships

teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam2","IPLTeam1")

### 4. Batsmen vs Bowler Plot

teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=TRUE)
teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)

### 5. Team bowling scorecard

teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 6. Team bowling Wicket kind match

teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 7. Team Bowling Wicket Runs Match

teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 8. Team Bowling Wicket Match

m <-teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m
teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 9. Team Bowler vs Batsmen

teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <- teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 10. Match Worm chart

matchWormGraph(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 1 IPL Match Analysis

Load any match data from the ./IPLMatches folder for e.g. Chennai Super Kings-Deccan Chargers-2008-05-06.RData

setwd("./IPLMatches")
csk_dc<- overs
#The steps are
IPLTeam1_IPLTeam2 <- overs

All analysis for this match can be done now

### 2. Scorecard

teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBattingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 3.Batting Partnerships

teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
teamBatsmenPartnershipMatch(IPLTeam1_IPLTeam2,"IPLTeam2","IPLTeam1")

### 4. Batsmen vs Bowler Plot

teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=TRUE)
teamBatsmenVsBowlersMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)

### 5. Team bowling scorecard

teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam1")
teamBowlingScorecardMatch(IPLTeam1_IPLTeam2,"IPLTeam2")

### 6. Team bowling Wicket kind match

teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketKindMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 7. Team Bowling Wicket Runs Match

teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <-teamBowlingWicketRunsMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 8. Team Bowling Wicket Match

m <-teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m
teamBowlingWicketMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 9. Team Bowler vs Batsmen

teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")
m <- teamBowlersVsBatsmenMatch(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2",plot=FALSE)
m

### 10. Match Worm chart

matchWormGraph(IPLTeam1_IPLTeam2,"IPLTeam1","IPLTeam2")

### 1. IPL Matches for a team against all other teams

Load the data between for a IPL team against all other countries ./allMatchesAllOpposition for e.g all matches of Kolkata Knight Riders

load("allMatchesAllOpposition-Kolkata Knight Riders.RData")
kkr_matches <- matches
IPLTeam="IPLTeam1"
allMatches <- paste("allMatchesAllOposition-",IPLTeam,".RData",sep="")
IPLTeam1AllMatches <- matches


### 2. Team’s batting scorecard all Matches

m <-teamBattingScorecardAllOppnAllMatches(IPLTeam1AllMatches,theTeam="IPLTeam1")
m

### 3. Batting scorecard of opposing team

m <-teamBattingScorecardAllOppnAllMatches(matches=IPLTeam1AllMatches,theTeam="IPLTeam2")

### 4. Team batting partnerships

m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam="IPLTeam1")
m
m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam='IPLTeam1',report="detailed")
m <- teamBatsmenPartnershipAllOppnAllMatches(IPLTeam1AllMatches,theTeam='IPLTeam1',report="summary")
m

### 5. Team batting partnerships plot

teamBatsmenPartnershipAllOppnAllMatchesPlot(IPLTeam1AllMatches,"IPLTeam1",main="IPLTeam1")
teamBatsmenPartnershipAllOppnAllMatchesPlot(IPLTeam1AllMatches,"IPLTeam1",main="IPLTeam2")

### 6, Team batsmen vs bowlers report

m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=0)
m
m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=1,dispRows=30)
m
m <-teamBatsmenVsBowlersAllOppnAllMatchesRept(matches=IPLTeam1AllMatches,theTeam="IPLTeam2",rank=1,dispRows=25)
m

### 7. Team batsmen vs bowler plot

d <- teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=1,dispRows=50)
d
teamBatsmenVsBowlersAllOppnAllMatchesPlot(d)
d <- teamBatsmenVsBowlersAllOppnAllMatchesRept(IPLTeam1AllMatches,"IPLTeam1",rank=2,dispRows=50)
teamBatsmenVsBowlersAllOppnAllMatchesPlot(d)

### 8. Team bowling scorecard

teamBowlingScorecardAllOppnAllMatchesMain(matches=IPLTeam1AllMatches,theTeam="IPLTeam1")
teamBowlingScorecardAllOppnAllMatches(IPLTeam1AllMatches,'IPLTeam2')

### 9. Team bowler vs batsmen

teamBowlersVsBatsmenAllOppnAllMatchesMain(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=0)
teamBowlersVsBatsmenAllOppnAllMatchesMain(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=2)
teamBowlersVsBatsmenAllOppnAllMatchesRept(matches=IPLTeam1AllMatches,theTeam="IPLTeam1",rank=0)

### 10. Team Bowler vs bastmen

df <- teamBowlersVsBatsmenAllOppnAllMatchesRept(IPLTeam1AllMatches,theTeam="IPLTeam1",rank=1)
teamBowlersVsBatsmenAllOppnAllMatchesPlot(df,"IPLTeam1","IPLTeam1")

### 11. Team bowler wicket kind

teamBowlingWicketKindAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="All")
teamBowlingWicketKindAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="IPLTeam2")


### 12.

teamBowlingWicketRunsAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="All",plot=TRUE)
teamBowlingWicketRunsAllOppnAllMatches(IPLTeam1AllMatches,t1="IPLTeam1",t2="IPLTeam2",plot=TRUE)

### 1 IPL Batsman setup functions

Get the batsman’s details for a batsman

setwd("../BattingBowlingDetails")
# IPL Team names
IPLTeamNames <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings Xi Punjab",
"Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors",
"Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
"Rising Pune Supergiants")

# Check and get the team indices of IPL teams in which the batsman has played
getTeamIndex <- function(batsman){
setwd("./BattingBowlingDetails")
setwd("..")
getwd()
print(ls())
teams_batsmen = list(csk_batsmen,dc_batsmen,dd_batsmen,kxip_batsmen,ktk_batsmen,kkr_batsmen,mi_batsmen,
pw_batsmen,rr_batsmen,rcb_batsmen,sh_batsmen,gl_batsmen,rps_batsmen)
b <- NULL
for (i in 1:length(teams_batsmen)){
a <- which(teams_batsmen[[i]] == batsman)

if(length(a) != 0)
b <- c(b,i)
}
b
}

# Get the list of the IPL team names from the indices passed
getTeams <- function(x){

l <- NULL
# Get the teams passed in as indexes
for (i in seq_along(x)){

l <- c(l, IPLTeamNames[[x[i]]])

}
l
}

# Create a consolidated data frame with all teams the IPL batsman has played for
getIPLBatsmanDF <- function(teamNames){
batsmanDF <- NULL
# Create a consolidated Data frame of batsman for all IPL teams played
for (i in seq_along(teamNames)){
df <- getBatsmanDetails(team=teamNames[i],name=IPLBatsman,dir="./BattingBowlingDetails")
batsmanDF <- rbind(batsmanDF,df)

}
batsmanDF
}


### 2. Create a consolidated IPL batsman data frame

# Since an IPL batsman coculd have played in multiple teams we need to determine these teams and
# create a consolidated data frame for the analysis
# For example to check MS Dhoni we need to do the following

IPLBatsman = "MS Dhoni"
#Check and get the team indices of IPL teams in which the batsman has played
i <- getTeamIndex(IPLBatsman)

# Get the team names in which the IPL batsman has played
teamNames <- getTeams(i)
# Check if file exists in the directory. This check is necessary when moving between matchType

############## Create a consolidated IPL batsman dataframe for analysis
batsmanDF <- getIPLBatsmanDF(teamNames)


### 3. Runs vs deliveries

# For e.g. batsmanName="MS Dhoni""
#batsmanRunsVsDeliveries(batsmanDF, "MS Dhoni")
batsmanRunsVsDeliveries(batsmanDF,"batsmanName")

### 4. Batsman 4s & 6s

batsman46 <- select(batsmanDF,batsman,ballsPlayed,fours,sixes,runs)
p1 <- batsmanFoursSixes(batsman46,"batsmanName")

### 5. Batsman dismissals

batsmanDismissals(batsmanDF,"batsmanName")

### 6. Runs vs Strike rate

batsmanRunsVsStrikeRate(batsmanDF,"batsmanName")

### 7. Batsman Moving Average

batsmanMovingAverage(batsmanDF,"batsmanName")

### 8. Batsman cumulative average

batsmanCumulativeAverageRuns(batsmanDF,"batsmanName")

### 9. Batsman cumulative strike rate

batsmanCumulativeStrikeRate(batsmanDF,"batsmanName")

### 10. Batsman runs against oppositions

batsmanRunsAgainstOpposition(batsmanDF,"batsmanName")

### 11. Batsman runs vs venue

batsmanRunsVenue(batsmanDF,"batsmanName")

### 12. Batsman runs predict

batsmanRunsPredict(batsmanDF,"batsmanName")

### 13.Bowler set up functions

setwd("../BattingBowlingDetails")
# IPL Team names
IPLTeamNames <- list("Chennai Super Kings","Deccan Chargers", "Delhi Daredevils","Kings Xi Punjab",
"Kochi Tuskers Kerala","Kolkata Knight Riders","Mumbai Indians","Pune Warriors",
"Rajasthan Royals","Royal Challengers Bangalore","Sunrisers Hyderabad","Gujarat Lions",
"Rising Pune Supergiants")

# Get the team indices of IPL teams for which the bowler as played
getTeamIndex_bowler <- function(bowler){
setwd("./data")
setwd("..")
teams_bowlers = list(csk_bowlers,dc_bowlers,dd_bowlers,kxip_bowlers,ktk_bowlers,kkr_bowlers,mi_bowlers,
pw_bowlers,rr_bowlers,rcb_bowlers,sh_bowlers,gl_bowlers,rps_bowlers)
b <- NULL
for (i in 1:length(teams_bowlers)){
a <- which(teams_bowlers[[i]] == bowler)
if(length(a) != 0){
b <- c(b,i)
}
}
b
}

# Get the list of the IPL team names from the indices passed
getTeams <- function(x){

l <- NULL
# Get the teams passed in as indexes
for (i in seq_along(x)){

l <- c(l, IPLTeamNames[[x[i]]])

}
l
}

# Get the team names
teamNames <- getTeams(i)

getIPLBowlerDF <- function(teamNames){
bowlerDF <- NULL

# Create a consolidated Data frame of batsman for all IPL teams played
for (i in seq_along(teamNames)){
df <- getBowlerWicketDetails(team=teamNames[i],name=IPLBowler,dir="./BattingBowlingDetails")
bowlerDF <- rbind(bowlerDF,df)

}
bowlerDF
}

### 14. Get the consolidated data frame for an IPL bowler

# Since an IPL bowler could have played in multiple teams we need to determine these teams and
# create a consolidated data frame for the analysis
# For example to check R Ashwin we need to do the following

IPLBowler = "R Ashwin"
#Check and get the team indices of IPL teams in which the batsman has played
i <- getTeamIndex(IPLBowler)

# Get the team names in which the IPL batsman has played
teamNames <- getTeams(i)
# Check if file exists in the directory. This check is necessary when moving between matchType

############## Create a consolidated IPL batsman dataframe for analysis
bowlerDF <- getIPLBowlerDF(teamNames)


### 15. Bowler Mean Economy rate

# For e.g. to get the details of R Ashwin do
#bowlerMeanEconomyRate(bowlerDF,"R Ashwin")
bowlerMeanEconomyRate(bowlerDF,"bowlerName")

### 16. Bowler mean runs conceded

bowlerMeanRunsConceded(bowlerDF,"bowlerName")

### 17. Bowler Moving Average

bowlerMovingAverage(bowlerDF,"bowlerName")

### 18. Bowler cumulative average wickets

bowlerCumulativeAvgWickets(bowlerDF,"bowlerName")

### 19. Bowler cumulative Economy Rate (ER)

bowlerCumulativeAvgEconRate(bowlerDF,"bowlerName")

### 20. Bowler wicket plot

bowlerWicketPlot(bowlerDF,"bowlerName")

### 21. Bowler wicket against opposition

bowlerWicketsAgainstOpposition(bowlerDF,"bowlerName")

### 22. Bowler wicket at cricket grounds

bowlerWicketsVenue(bowlerDF,"bowlerName")

### 23. Predict number of deliveries to wickets

setwd("./IPLMatches")
bowlerDF1 <- getDeliveryWickets(team="IPLTeam1",dir=".",name="bowlerName",save=FALSE)
bowlerWktsPredict(bowlerDF1,"bowlerName")

# Video presentation on Machine Learning, Data Science, NLP and Big Data – Part 4

In this presentation I demonstrate some Shiny R apps that I have developed. The demo also includes googleMotion Charts using googleVis

# Video presentation on Machine Learning, Data Science, NLP and Big Data – Part 3

This presentation includes the 3 part and touches upon Apache Edgent, NLP and Big Data with Spark.

# Beaten by sheer pace – Cricket analytics with yorkr

My ebook “Beaten by sheer pace – Cricket analytics with yorkr’  has been published in Leanpub.  You can now download the book (hot off the press!)  for all formats to your favorite device (mobile, iPad, tablet, Kindle)  from the Leanpub  “Beaten by sheer pace!”. The book has been published in the following formats namely

• EPUB (for iPad or tablets. Save the file cricketAnalyticsWithYorkr.epub to Google Drive/Dropbox and choose “Open in” iBooks for iPad)
• MOBI (for Kindle. For this format, I suggest that you download & install SendToKindle for PC/Mac. You can then right click the downloaded cricketAnalyticsWithYorkr.mobi and choose SendToKindle. You will need to login to your Kindle account)

From Leanpub
Leanpub uses a variable pricing model. I have priced the book attractively (I think!). You can choose a price between FREE (limited time offer!) to $4.99 . The link is “Beaten by sheer pace! This format works with all type Kindle device, Kindle app, Android tablet, iPad. Checkout my interactive Shiny apps GooglyPlus (plots & tables) and Googly (only plots) which can be used to analyze IPL players, teams and matches. # Introducing cricket package yorkr:Part 4-In the block hole! ## Introduction “The nitrogen in our DNA, the calcium in our teeth, the iron in our blood, the carbon in our apple pies were made in the interiors of collapsing stars. We are made of starstuff.” “If you wish to make an apple pie from scratch, you must first invent the universe.” “We are like butterflies who flutter for a day and think it is forever.” “The absence of evidence is not the evidence of absence.” “We are star stuff which has taken its destiny into its own hands.”  Cosmos - Carl Sagan This post is the 4th and possibly, the last part of my introduction, to my latest cricket package yorkr. This is the 4th part of the introduction, the 3 earlier ones were The 1st part included functions dealing with a specific match, the 2nd part dealt with functions between 2 opposing teams. The 3rd part dealt with functions between a team and all matches with all oppositions. This 4th part includes individual batting and bowling performances in ODI matches and deals with Class 4 functions. If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at$12.99 for the paperback and $4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr  and Beaten by sheer pace-Cricket analytics with yorkr  A must read for any cricket lover! Check it out!!

d $4.99/Rs 320 and$6.99/Rs448 respectively

This post has also been published at RPubs yorkr-Part4 and can also be downloaded as a PDF document from yorkr-Part4.pdf.

You can clone/fork the code for the package yorkr from Github at yorkr-package

Checkout my interactive Shiny apps GooglyPlus (plots & tables) and Googly (only plots) which can be used to analyze IPL players, teams and matches.

#### Batsman functions

1. batsmanRunsVsDeliveries
3. batsmanDismissals
4. batsmanRunsVsStrikeRate
5. batsmanMovingAverage
6. batsmanCumulativeAverageRuns
7. batsmanCumulativeStrikeRate
8. batsmanRunsAgainstOpposition
9. batsmanRunsVenue
10. batsmanRunsPredict

#### Bowler functions

1. bowlerMeanEconomyRate
2. bowlerMeanRunsConceded
3. bowlerMovingAverage
4. bowlerCumulativeAvgWickets
5. bowlerCumulativeAvgEconRate
6. bowlerWicketPlot
7. bowlerWicketsAgainstOpposition
8. bowlerWicketsVenue
9. bowlerWktsPredict

Note: The yorkr package in its current avatar only supports ODI, T20 and IPL T20 matches.

library(yorkr)
library(gridExtra)
library(rpart.plot)
library(dplyr)
library(ggplot2)
rm(list=ls())

## A. Batsman functions

### 1. Get Team Batting details

The function below gets the overall team batting details based on the RData file available in ODI matches. This is currently also available in Github at (https://github.com/tvganesh/yorkrData/tree/master/ODI/ODI-matches).  However you may have to do this as future matches are added! The batting details of the team in each match is created and a huge data frame is created by rbinding the individual dataframes. This can be saved as a RData file

setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches")
india_details <- getTeamBattingDetails("India",dir=".", save=TRUE)
dim(india_details)
## [1] 11085    15
sa_details <- getTeamBattingDetails("South Africa",dir=".",save=TRUE)
dim(sa_details)
## [1] 6375   15
nz_details <- getTeamBattingDetails("New Zealand",dir=".",save=TRUE)
dim(nz_details)
## [1] 6262   15
eng_details <- getTeamBattingDetails("England",dir=".",save=TRUE)
dim(eng_details)
## [1] 9001   15

### 2. Get batsman details

This function is used to get the individual batting record for a the specified batsmen of the country as in the functions below. For analyzing the batting performances the following cricketers have been chosen

1. Virat Kohli (Ind)
2. M S Dhoni (Ind)
3. AB De Villiers (SA)
4. Q De Kock (SA)
5. J Root (Eng)
6. M J Guptill (NZ)
setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches")
kohli <- getBatsmanDetails(team="India",name="Kohli",dir=".")
## [1] "./India-BattingDetails.RData"
dhoni <- getBatsmanDetails(team="India",name="Dhoni")
## [1] "./India-BattingDetails.RData"
devilliers <-  getBatsmanDetails(team="South Africa",name="Villiers",dir=".")
## [1] "./South Africa-BattingDetails.RData"
deKock <-  getBatsmanDetails(team="South Africa",name="Kock",dir=".")
## [1] "./South Africa-BattingDetails.RData"
root <-  getBatsmanDetails(team="England",name="Root",dir=".")
## [1] "./England-BattingDetails.RData"
guptill <-  getBatsmanDetails(team="New Zealand",name="Guptill",dir=".")
## [1] "./New Zealand-BattingDetails.RData"

### 3. Runs versus deliveries

Kohli, De Villiers and Guptill have a good cluster of points that head towards 150 runs at 150 deliveries.

p1 <-batsmanRunsVsDeliveries(kohli,"Kohli")
p2 <- batsmanRunsVsDeliveries(dhoni, "Dhoni")
p3 <- batsmanRunsVsDeliveries(devilliers,"De Villiers")
p4 <- batsmanRunsVsDeliveries(deKock,"Q de Kock")
p5 <- batsmanRunsVsDeliveries(root,"JE Root")
p6 <- batsmanRunsVsDeliveries(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 4. Batsman Total runs, Fours and Sixes

The plots below show the total runs, fours and sixes by the batsmen

kohli46 <- select(kohli,batsman,ballsPlayed,fours,sixes,runs)
dhoni46 <- select(dhoni,batsman,ballsPlayed,fours,sixes,runs)
devilliers46 <- select(devilliers,batsman,ballsPlayed,fours,sixes,runs)
deKock46 <- select(deKock,batsman,ballsPlayed,fours,sixes,runs)
root46 <- select(root,batsman,ballsPlayed,fours,sixes,runs)
guptill46 <- select(guptill,batsman,ballsPlayed,fours,sixes,runs)
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 5. Batsman dismissals

The type of dismissal for each batsman is shown below

p1 <-batsmanDismissals(kohli,"Kohli")
p2 <- batsmanDismissals(dhoni, "Dhoni")
p3 <- batsmanDismissals(devilliers, "De Villiers")
p4 <- batsmanDismissals(deKock,"Q de Kock")
p5 <- batsmanDismissals(root,"JE Root")
p6 <- batsmanDismissals(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 6. Runs versus Strike Rate

De villiers has the best strike rate among all as there are more points to the right side of the plot for the same runs. Kohli and Dhoni do well too. Q De Kock and Joe Root also have a very good spread of points though they have fewer innings.

p1 <-batsmanRunsVsStrikeRate(kohli,"Kohli")
p2 <- batsmanRunsVsStrikeRate(dhoni, "Dhoni")
p3 <- batsmanRunsVsStrikeRate(devilliers, "De Villiers")
p4 <- batsmanRunsVsStrikeRate(deKock,"Q de Kock")
p5 <- batsmanRunsVsStrikeRate(root,"JE Root")
p6 <- batsmanRunsVsStrikeRate(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 7. Batsman moving average

Kohli’s average is on a gentle increase from below 50 to around 60’s. Joe Root performance is impressive with his moving average of late tending towards the 70’s. Q De Kock seemed to have a slump around 2015 but his performance is on the increase. Devilliers consistently averages around 50. Dhoni also has been having a stable run in the last several years.

p1 <-batsmanMovingAverage(kohli,"Kohli")
p2 <- batsmanMovingAverage(dhoni, "Dhoni")
p3 <- batsmanMovingAverage(devilliers, "De Villiers")
p4 <- batsmanMovingAverage(deKock,"Q de Kock")
p5 <- batsmanMovingAverage(root,"JE Root")
p6 <- batsmanMovingAverage(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 8. Batsman cumulative average

The functions below provide the cumulative average of runs scored. As can be seen Kohli and Devilliers have a cumulative runs rate that averages around 48-50. Q De Kock seems to have had a rocky career with several highs and lows as the cumulative average oscillates between 45-40. Root steadily improves to a cumulative average of around 42-43 from his 50th innings

p1 <-batsmanCumulativeAverageRuns(kohli,"Kohli")
p2 <- batsmanCumulativeAverageRuns(dhoni, "Dhoni")
p3 <- batsmanCumulativeAverageRuns(devilliers, "De Villiers")
p4 <- batsmanCumulativeAverageRuns(deKock,"Q de Kock")
p5 <- batsmanCumulativeAverageRuns(root,"JE Root")
p6 <- batsmanCumulativeAverageRuns(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 9. Cumulative Average Strike Rate

The plots below show the cumulative average strike rate of the batsmen. Dhoni and Devilliers have the best cumulative average strike rate of 90%. The rest average around 80% strike rate. Guptill shows a slump towards the latter part of his career.

p1 <-batsmanCumulativeStrikeRate(kohli,"Kohli")
p2 <- batsmanCumulativeStrikeRate(dhoni, "Dhoni")
p3 <- batsmanCumulativeStrikeRate(devilliers, "De Villiers")
p4 <- batsmanCumulativeStrikeRate(deKock,"Q de Kock")
p5 <- batsmanCumulativeStrikeRate(root,"JE Root")
p6 <- batsmanCumulativeStrikeRate(guptill,"MJ Guptill")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 10. Batsman runs against opposition

Kohli’s best performances are against Australia, West Indies and Sri Lanka

batsmanRunsAgainstOpposition(kohli,"Kohli")

batsmanRunsAgainstOpposition(dhoni, "Dhoni")

Kohli’s best performances are against Australia, Pakistan and West Indies

batsmanRunsAgainstOpposition(devilliers, "De Villiers")

Quentin de Kock average almost 100 runs against India and 75 runs against England

batsmanRunsAgainstOpposition(deKock, "Q de Kock")

Root’s best performances are against South Africa, Sri Lanka and West Indies

batsmanRunsAgainstOpposition(root, "JE Root")

batsmanRunsAgainstOpposition(guptill, "MJ Guptill")

### 11. Runs at different venues

The plots below give the performances of the batsmen at different grounds.

batsmanRunsVenue(kohli,"Kohli")

batsmanRunsVenue(dhoni, "Dhoni")

batsmanRunsVenue(devilliers, "De Villiers")

batsmanRunsVenue(deKock, "Q de Kock")

batsmanRunsVenue(root, "JE Root")

batsmanRunsVenue(guptill, "MJ Guptill")

### 12. Predict number of runs to deliveries

The plots below use rpart classification tree to predict the number of deliveries required to score the runs in the leaf node. For e.g. Kohli takes 66 deliveries to score 64 runs and for higher number of deliveries scores around 115 runs. Devilliers needs

par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
batsmanRunsPredict(kohli,"Kohli")
batsmanRunsPredict(dhoni, "Dhoni")
batsmanRunsPredict(devilliers, "De Villiers")

par(mfrow=c(1,3))
par(mar=c(4,4,2,2))
batsmanRunsPredict(deKock,"Q de Kock")
batsmanRunsPredict(root,"JE Root")
batsmanRunsPredict(guptill,"MJ Guptill")

## B. Bowler functions

### 13. Get bowling details

The function below gets the overall team bowling details based on the RData file available in ODI matches. This is currently also available in Github at (https://github.com/tvganesh/yorkrData/tree/master/ODI/ODI-matches). The bowling details of the team in each match is created and a huge data frame is created by rbinding the individual dataframes. This can be saved as a RData file

setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches")
ind_bowling <- getTeamBowlingDetails("India",dir=".",save=TRUE)
dim(ind_bowling)
## [1] 7816   12
aus_bowling <- getTeamBowlingDetails("Australia",dir=".",save=TRUE)
dim(aus_bowling)
## [1] 9191   12
ban_bowling <- getTeamBowlingDetails("Bangladesh",dir=".",save=TRUE)
dim(ban_bowling)
## [1] 5665   12
sa_bowling <- getTeamBowlingDetails("South Africa",dir=".",save=TRUE)
dim(sa_bowling)
## [1] 3806   12
sl_bowling <- getTeamBowlingDetails("Sri Lanka",dir=".",save=TRUE)
dim(sl_bowling)
## [1] 3964   12

### 14. Get bowling details of the individual bowlers

This function is used to get the individual bowling record for a specified bowler of the country as in the functions below. For analyzing the bowling performances the following cricketers have been chosen

2. Ravichander Ashwin (Ind)
3. Mitchell Starc (Aus)
4. Shakib Al Hasan (Ban)
5. Ajantha Mendis (SL)
6. Dale Steyn (SA)
jadeja <- getBowlerWicketDetails(team="India",name="Jadeja",dir=".")
ashwin <- getBowlerWicketDetails(team="India",name="Ashwin",dir=".")
starc <-  getBowlerWicketDetails(team="Australia",name="Starc",dir=".")
mendis <-  getBowlerWicketDetails(team="Sri Lanka",name="Mendis",dir=".")
steyn <-  getBowlerWicketDetails(team="South Africa",name="Steyn",dir=".")

### 15. Bowler Mean Economy Rate

Shakib Al Hassan is expensive in the 1st 3 overs after which he is very economical with a economy rate of 3-4. Starc, Steyn average around a ER of 4.0

p1<-bowlerMeanEconomyRate(jadeja,"RA Jadeja")
p2<-bowlerMeanEconomyRate(ashwin, "R Ashwin")
p3<-bowlerMeanEconomyRate(starc, "MA Starc")
p4<-bowlerMeanEconomyRate(shakib, "Shakib Al Hasan")
p5<-bowlerMeanEconomyRate(mendis, "A Mendis")
p6<-bowlerMeanEconomyRate(steyn, "D Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 16. Bowler Mean Runs conceded

Ashwin is expensive around 6 & 7 overs

p1<-bowlerMeanRunsConceded(jadeja,"RA Jadeja")
p2<-bowlerMeanRunsConceded(ashwin, "R Ashwin")
p3<-bowlerMeanRunsConceded(starc, "M A Starc")
p4<-bowlerMeanRunsConceded(shakib, "Shakib Al Hasan")
p5<-bowlerMeanRunsConceded(mendis, "A Mendis")
p6<-bowlerMeanRunsConceded(steyn, "D Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 17. Bowler Moving average

RA jadeja and Mendis’ performance has dipped considerably, while Ashwin and Shakib have improving performances. Starc average around 4 wickets

p1<-bowlerMovingAverage(jadeja,"RA Jadeja")
p2<-bowlerMovingAverage(ashwin, "Ashwin")
p3<-bowlerMovingAverage(starc, "M A Starc")
p4<-bowlerMovingAverage(shakib, "Shakib Al Hasan")
p5<-bowlerMovingAverage(mendis, "Ajantha Mendis")
p6<-bowlerMovingAverage(steyn, "Dale Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 17. Bowler cumulative average wickets

Starc is clearly the most consistent performer with 3 wickets on an average over his career, while Jadeja averages around 2.0. Ashwin seems to have dropped from 2.4-2.0 wickets, while Mendis drops from high 3.5 to 2.2 wickets. The fractional wickets only show a tendency to take another wicket.

p1<-bowlerCumulativeAvgWickets(jadeja,"RA Jadeja")
p2<-bowlerCumulativeAvgWickets(ashwin, "Ashwin")
p3<-bowlerCumulativeAvgWickets(starc, "M A Starc")
p4<-bowlerCumulativeAvgWickets(shakib, "Shakib Al Hasan")
p5<-bowlerCumulativeAvgWickets(mendis, "Ajantha Mendis")
p6<-bowlerCumulativeAvgWickets(steyn, "Dale Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 18. Bowler cumulative Economy Rate (ER)

The plots below are interesting. All of the bowlers seem to average around 4.5 runs/over. RA Jadeja’s ER improves and heads to 4.5, Mendis is seen to getting more expensive as his career progresses. From a ER of 3.0 he increases towards 4.5

p1<-bowlerCumulativeAvgEconRate(jadeja,"RA Jadeja")
p2<-bowlerCumulativeAvgEconRate(ashwin, "Ashwin")
p3<-bowlerCumulativeAvgEconRate(starc, "M A Starc")
p4<-bowlerCumulativeAvgEconRate(shakib, "Shakib Al Hasan")
p5<-bowlerCumulativeAvgEconRate(mendis, "Ajantha Mendis")
p6<-bowlerCumulativeAvgEconRate(steyn, "Dale Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 19. Bowler wicket plot

The plot below gives the average wickets versus number of overs

p1<-bowlerWicketPlot(jadeja,"RA Jadeja")
p2<-bowlerWicketPlot(ashwin, "Ashwin")
p3<-bowlerWicketPlot(starc, "M A Starc")
p4<-bowlerWicketPlot(shakib, "Shakib Al Hasan")
p5<-bowlerWicketPlot(mendis, "Ajantha Mendis")
p6<-bowlerWicketPlot(steyn, "Dale Steyn")
grid.arrange(p1,p2,p3,p4,p5,p6, ncol=3)

### 20. Bowler wicket against opposition

#Jadeja's' best pertformance are against England, Pakistan and West Indies
bowlerWicketsAgainstOpposition(jadeja,"RA Jadeja")

#Ashwin's bets pertformance are against England, Pakistan and South Africa
bowlerWicketsAgainstOpposition(ashwin, "Ashwin")

#Starc has good performances against India, New Zealand, Pakistan, West Indies
bowlerWicketsAgainstOpposition(starc, "M A Starc")

bowlerWicketsAgainstOpposition(shakib,"Shakib Al Hasan")

bowlerWicketsAgainstOpposition(mendis, "Ajantha Mendis")

#Steyn has good performances against India, Sri Lanka, Pakistan, West Indies
bowlerWicketsAgainstOpposition(steyn, "Dale Steyn")

### 21. Bowler wicket at cricket grounds

bowlerWicketsVenue(jadeja,"RA Jadeja")

bowlerWicketsVenue(ashwin, "Ashwin")

bowlerWicketsVenue(starc, "M A Starc")
## Warning: Removed 2 rows containing missing values (geom_bar).

bowlerWicketsVenue(shakib,"Shakib Al Hasan")

bowlerWicketsVenue(mendis, "Ajantha Mendis")

bowlerWicketsVenue(steyn, "Dale Steyn")

### 22. Get Delivery wickets for bowlers

Thsi function creates a dataframe of deliveries and the wickets taken

setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches")
ashwin1 <- getDeliveryWickets(team="India",dir=".",name="Ashwin",save=FALSE)
starc1 <- getDeliveryWickets(team="Australia",dir=".",name="MA Starc",save=FALSE)
mendis1 <- getDeliveryWickets(team="Sri Lanka",dir=".",name="Mendis",save=FALSE)
steyn1 <- getDeliveryWickets(team="South Africa",dir=".",name="Steyn",save=FALSE)

### 23. Predict number of deliveries to wickets

#Jadeja and Ashwin need around 22 to 28 deliveries to make a break through
par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
bowlerWktsPredict(ashwin1,"RAshwin")

#Starc and Shakib provide an early breakthrough producing a wicket in around 16 balls. Starc's 2nd wicket comed around the 30th delivery
par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
bowlerWktsPredict(starc1,"MA Starc")
bowlerWktsPredict(shakib1,"Shakib Al Hasan")

#Steyn and Mendis take 20 deliveries to get their 1st wicket
par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
bowlerWktsPredict(mendis1,"A Mendis")
bowlerWktsPredict(steyn1,"DSteyn")

## Conclusion

This concludes the 4 part introduction to my new R cricket package yorkr for ODIs. I will be enhancing the package to handle Twenty20 and IPL matches soon. You can fork/clone the code from Github at yorkr.

The yaml data from Cricsheet have already beeen converted into R consumable dataframes. The converted data can be downloaded from Github at yorkrData. There are 3 folders – ODI matches, ODI matches between 2 teams (oppnAllMatches), ODI matches between a team and the rest of the world (all matches,all oppositions).

As I have already mentioned I have around 67 functions for analysis, however I am certain that the data has a lot more secrets waiting to be tapped. So please do go ahead and run any machine learning or statistical learning algorithms on them. If you do come up with interesting insights, I would appreciate if attribute the source to Cricsheet(http://cricsheet.org), and my package yorkr and my blog Giga thoughts*, besides dropping me a note.

Hope you have a great time with my yorkr package!

Also see

# Introducing cricket package yorkr: Part 2-Trapped leg before wicket!

“It was a puzzling thing. The truth knocks on the door and you say ‘Go away, I ’m looking for the truth,’ and so it goes away. Puzzling.”

“But even though Quality cannot be defined, you know what Quality is!”

“The Buddha, the Godhead, resides quite comfortably in the circuits of a digital computer or the gears of a cycle transmission as he does at the top of a mountain or in the petals of the flower. To think otherwise is to demean the Buddha – which is to demean oneself.”

                Zen and the Art of Motorcycle maintenance - Robert M Pirsig

## Introduction

If we were to to extend the last quote from Zen and the Art of Motorcycle Maintenance, by Robert M Pirsig, I think it would be fair to say that the Buddha also comfortably resides in the exquisite backhand cross-court return of Bjorn Borg, to the the graceful arc of the football in a Lionel Messi’s free kick to the smashing cover drive of Sunil Gavaskar.

In this post I continue to introduce my latest cricket package yorkr. This post is a continuation of my earlier post – Introducing cricket package yorkr-Part1:Beaten by sheer pace!. This post deals with Class 2 functions namely the performances of a team in all matches against a single opposition for e.g all matches of India-Australia, Pakistan-West Indies etc. You can clone/fork the code for my package yorkr from Github at yorkr

If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at $12.99 for the paperback and$4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr and Beaten by sheer pace-Cricket analytics with yorkr A must read for any cricket lover! Check it out!! Note 1: The package currently only supports ODI, T20s and IPL T20 matches. This post has also been published at RPubs yorkr-Part2 and can also be downloaded as a PDF document from yorkr-Part2.pdf Checkout my interactive Shiny apps GooglyPlus (plots & tables) and Googly (only plots) which can be used to analyze IPL players, teams and matches. The list of function in Class 2 are 1. teamBatsmenPartnershiOppnAllMatches() 2. teamBatsmenPartnershipOppnAllMatchesChart() 3. teamBatsmenVsBowlersOppnAllMatches() 4. teamBattingScorecardOppnAllMatches() 5. teamBowlingPerfOppnAllMatches() 6. teamBowlersWicketsOppnAllMatches() 7. teamBowlersVsBatsmenOppnAllMatches() 8. teamBowlersWicketKindOppnAllMatches() 9. teamBowlersWicketRunsOppnAllMatches() 10. plotWinLossBetweenTeams() ### 1. Install the package from CRAN if (!require("yorkr")) { install.packages("yorkr") library("yorkr") } library(plotly) rm(list=ls()) ### 2. Get data for all matches between 2 teams We can get all matches between any 2 teams using the function below. The dir parameter should point to the folder which RData files of the individual matches. This function creates a data frame of all the matches and also saves the dataframe as RData setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches") matches <- getAllMatchesBetweenTeams("Australia","India",dir=".") dim(matches) ## [1] 67428 25 I have however already saved the matches for all possible combination of opposing countries. The data for these matches for the individual teams/countries can be obtained from Github at in the folder ODI-allmatches-between-two-teams Note: The dataframe for the different head-to-head matches can be loaded directly into your code. The datframes are 15000+ rows x 25 columns. While I have 10 functions to process the details between teams, feel free to let loose any statistical or machine learning algorithms on the dataframe. So go ahead with any insights that can be gleaned from random forests, ridge regression,SVM classifiers and so on. If you do come up with something interesting, I would appreciate if you could drop me a note. Also please do attribute source to Cricsheet (http://cricsheet.org), the package york and my blog Giga thoughts ### 3. Save data for all matches between all combination of 2 teams This can be done locally using the function below. You could use this function to combine all matches between any 2 teams into a single dataframe and save it in the current folder. The current implementation expectes that the the RData files of individual matches are in ../data folder. Since I already have converted this I will not be running this again #saveAllMatchesBetweenTeams() ### 4. Load data directly for all matches between 2 teams As in my earlier post I pick all matches between 2 random teams. I load the data directly from the stored RData files. When we load the Rdata file a “matches” object will be created. This object can be stored for the apporpriate teams as below setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-allmatches-between-two-teams") load("India-Australia-allMatches.RData") aus_ind_matches <- matches dim(aus_ind_matches) ## [1] 21909 25 load("England-New Zealand-allMatches.RData") eng_nz_matches <- matches dim(eng_nz_matches) ## [1] 15343 25 load("Pakistan-South Africa-allMatches.RData") pak_sa_matches <- matches dim(pak_sa_matches) ## [1] 17083 25 load("Sri Lanka-West Indies-allMatches.RData") sl_wi_matches <- matches dim(sl_wi_matches) ## [1] 4869 25 load("Bangladesh-Ireland-allMatches.RData") ban_ire_matches <-matches dim(ban_ire_matches) ## [1] 1668 25 load("Kenya-Bermuda-allMatches.RData") ken_ber_matches <- matches dim(ken_ber_matches) ## [1] 1518 25 load("Scotland-Canada-allMatches.RData") sco_can_matches <-matches dim(sco_can_matches) ## [1] 1061 25 load("Netherlands-Afghanistan-allMatches.RData") nl_afg_matches <- matches dim(nl_afg_matches) ## [1] 402 25 ### 5. Team Batsmen partnership (all matches with opposition) This function will create a report of the batting partnerships in the teams. The report can be brief or detailed depending on the parameter ‘report’. The top batsmen in India-Australia clashes are Ricky Ponting from Australia and Mahendra Singh Dhoni of India. m<- teamBatsmenPartnershiOppnAllMatches(aus_ind_matches,'Australia',report="summary") m ## Source: local data frame [47 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 RT Ponting 876 ## 2 MEK Hussey 753 ## 3 GJ Bailey 614 ## 4 SR Watson 609 ## 5 MJ Clarke 607 ## 6 ML Hayden 573 ## 7 A Symonds 536 ## 8 AJ Finch 525 ## 9 SPD Smith 467 ## 10 DA Warner 391 ## .. ... ... m <-teamBatsmenPartnershiOppnAllMatches(aus_ind_matches,'India',report="summary") m ## Source: local data frame [44 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 MS Dhoni 1156 ## 2 RG Sharma 918 ## 3 SR Tendulkar 910 ## 4 V Kohli 902 ## 5 G Gambhir 536 ## 6 Yuvraj Singh 524 ## 7 SK Raina 509 ## 8 S Dhawan 471 ## 9 V Sehwag 289 ## 10 RV Uthappa 283 ## .. ... ... m <-teamBatsmenPartnershiOppnAllMatches(aus_ind_matches,'Australia',report="detailed") m <-teamBatsmenPartnershiOppnAllMatches(pak_sa_matches,'Pakistan',report="summary") m ## Source: local data frame [40 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 Misbah-ul-Haq 727 ## 2 Younis Khan 657 ## 3 Shahid Afridi 558 ## 4 Mohammad Yousuf 539 ## 5 Mohammad Hafeez 477 ## 6 Shoaib Malik 452 ## 7 Ahmed Shehzad 348 ## 8 Abdul Razzaq 246 ## 9 Kamran Akmal 241 ## 10 Umar Akmal 215 ## .. ... ... m <-teamBatsmenPartnershiOppnAllMatches(eng_nz_matches,'England',report="summary") m ## Source: local data frame [47 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 IR Bell 654 ## 2 JE Root 612 ## 3 PD Collingwood 514 ## 4 EJG Morgan 479 ## 5 AN Cook 464 ## 6 IJL Trott 362 ## 7 KP Pietersen 358 ## 8 JC Buttler 287 ## 9 OA Shah 274 ## 10 RS Bopara 222 ## .. ... ... m <-teamBatsmenPartnershiOppnAllMatches(sl_wi_matches,'Sri Lanka',report="summary") m[1:50,] ## Source: local data frame [50 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 DPMD Jayawardene 288 ## 2 KC Sangakkara 238 ## 3 TM Dilshan 224 ## 4 WU Tharanga 220 ## 5 AD Mathews 161 ## 6 ST Jayasuriya 160 ## 7 ML Udawatte 87 ## 8 HDRL Thirimanne 67 ## 9 MDKJ Perera 64 ## 10 CK Kapugedera 57 ## .. ... ... m <- teamBatsmenPartnershiOppnAllMatches(ban_ire_matches,"Ireland",report="summary") m ## Source: local data frame [16 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 WTS Porterfield 111 ## 2 KJ O'Brien 99 ## 3 NJ O'Brien 75 ## 4 GC Wilson 60 ## 5 AR White 38 ## 6 DT Johnston 36 ## 7 JP Bray 31 ## 8 JF Mooney 28 ## 9 AC Botha 23 ## 10 EC Joyce 16 ## 11 PR Stirling 15 ## 12 GH Dockrell 9 ## 13 WB Rankin 9 ## 14 D Langford-Smith 6 ## 15 EJG Morgan 5 ## 16 AR Cusack 0 ### 6. Team batsmen partnership (all matches with opposition) This is plotted graphically in the charts below teamBatsmenPartnershipOppnAllMatchesChart(aus_ind_matches,"India","Australia") teamBatsmenPartnershipOppnAllMatchesChart(pak_sa_matches,main="South Africa",opposition="Pakistan") m<- teamBatsmenPartnershipOppnAllMatchesChart(eng_nz_matches,"New Zealand",opposition="England",plot=FALSE) m[1:30,] ## batsman nonStriker runs ## 1 KS Williamson LRPL Taylor 354 ## 2 BB McCullum MJ Guptill 275 ## 3 LRPL Taylor KS Williamson 273 ## 4 MJ Guptill BB McCullum 227 ## 5 BB McCullum JD Ryder 212 ## 6 MJ Guptill KS Williamson 196 ## 7 KS Williamson MJ Guptill 179 ## 8 JD Ryder BB McCullum 175 ## 9 JDP Oram SB Styris 153 ## 10 LRPL Taylor GD Elliott 147 ## 11 GD Elliott LRPL Taylor 143 ## 12 LRPL Taylor MJ Guptill 140 ## 13 JM How BB McCullum 128 ## 14 MJ Guptill LRPL Taylor 125 ## 15 BB McCullum JM How 117 ## 16 BB McCullum LRPL Taylor 116 ## 17 SB Styris JDP Oram 100 ## 18 LRPL Taylor JM How 98 ## 19 JM How LRPL Taylor 98 ## 20 JDP Oram BB McCullum 84 ## 21 LRPL Taylor L Vincent 71 ## 22 JDP Oram DL Vettori 70 ## 23 LRPL Taylor BB McCullum 61 ## 24 SB Styris JM How 55 ## 25 DR Flynn SB Styris 54 ## 26 DL Vettori JDP Oram 53 ## 27 L Vincent LRPL Taylor 53 ## 28 MJ Santner LRPL Taylor 53 ## 29 SP Fleming L Vincent 52 ## 30 JM How SB Styris 50 teamBatsmenPartnershipOppnAllMatchesChart(sl_wi_matches,"Sri Lanka","West Indies") teamBatsmenPartnershipOppnAllMatchesChart(ban_ire_matches,"Bangladesh","Ireland") ### 7. Team batsmen versus bowler (all matches with opposition) The plots below provide information on how each of the top batsmen fared against the opposition bowlers teamBatsmenVsBowlersOppnAllMatches(aus_ind_matches,"India","Australia") teamBatsmenVsBowlersOppnAllMatches(pak_sa_matches,"South Africa","Pakistan",top=3) m <- teamBatsmenVsBowlersOppnAllMatches(eng_nz_matches,"England","New Zealnd",top=10,plot=FALSE) m ## Source: local data frame [157 x 3] ## Groups: batsman [1] ## ## batsman bowler runs ## (fctr) (fctr) (dbl) ## 1 IR Bell JEC Franklin 63 ## 2 IR Bell SE Bond 13 ## 3 IR Bell MR Gillespie 33 ## 4 IR Bell NJ Astle 0 ## 5 IR Bell JS Patel 20 ## 6 IR Bell DL Vettori 28 ## 7 IR Bell JDP Oram 48 ## 8 IR Bell SB Styris 12 ## 9 IR Bell KD Mills 124 ## 10 IR Bell TG Southee 84 ## .. ... ... ... teamBatsmenVsBowlersOppnAllMatches(sl_wi_matches,"Sri Lanka","West Indies") teamBatsmenVsBowlersOppnAllMatches(ban_ire_matches,"Bangladesh","Ireland") ### 8. Team batsmen versus bowler (all matches with opposition) The following tables gives the overall performances of the country’s batsmen against the opposition. For India-Australia matches Dhoni, Rohit Sharma and Tendulkar lead the way. For Australia it is Ricky Ponting, M Hussey and GJ Bailey. In South Africa- Pakistan matches it is AB Devilliers, Hashim Amla etc. a <-teamBattingScorecardOppnAllMatches(aus_ind_matches,main="India",opposition="Australia") ## Total= 8331 a ## Source: local data frame [44 x 5] ## ## batsman ballsPlayed fours sixes runs ## (fctr) (int) (int) (int) (dbl) ## 1 MS Dhoni 1406 78 22 1156 ## 2 RG Sharma 1015 73 24 918 ## 3 SR Tendulkar 1157 103 6 910 ## 4 V Kohli 961 87 6 902 ## 5 G Gambhir 677 44 2 536 ## 6 Yuvraj Singh 664 52 11 524 ## 7 SK Raina 536 43 11 509 ## 8 S Dhawan 470 55 6 471 ## 9 V Sehwag 305 42 4 289 ## 10 RV Uthappa 295 29 7 283 ## .. ... ... ... ... ... teamBattingScorecardOppnAllMatches(aus_ind_matches,"Australia","India") ## Total= 9995 ## Source: local data frame [47 x 5] ## ## batsman ballsPlayed fours sixes runs ## (fctr) (int) (int) (int) (dbl) ## 1 RT Ponting 1107 86 8 876 ## 2 MEK Hussey 816 56 5 753 ## 3 GJ Bailey 578 51 13 614 ## 4 SR Watson 653 81 10 609 ## 5 MJ Clarke 786 45 5 607 ## 6 ML Hayden 660 72 8 573 ## 7 A Symonds 543 43 15 536 ## 8 AJ Finch 617 52 9 525 ## 9 SPD Smith 431 44 7 467 ## 10 DA Warner 385 40 6 391 ## .. ... ... ... ... ... teamBattingScorecardOppnAllMatches(pak_sa_matches,"South Africa","Pakistan") ## Total= 6657 ## Source: local data frame [36 x 5] ## ## batsman ballsPlayed fours sixes runs ## (fctr) (int) (int) (int) (dbl) ## 1 AB de Villiers 1533 128 23 1423 ## 2 HM Amla 864 88 3 815 ## 3 GC Smith 726 68 3 597 ## 4 JH Kallis 710 40 8 543 ## 5 JP Duminy 620 35 3 481 ## 6 CA Ingram 388 32 1 305 ## 7 F du Plessis 363 30 4 278 ## 8 Q de Kock 336 28 2 270 ## 9 DA Miller 329 20 2 250 ## 10 HH Gibbs 252 33 2 228 ## .. ... ... ... ... ... teamBattingScorecardOppnAllMatches(sl_wi_matches,"West Indies","Sri Lanka") ## Total= 1800 ## Source: local data frame [36 x 5] ## ## batsman ballsPlayed fours sixes runs ## (fctr) (int) (int) (int) (dbl) ## 1 DM Bravo 353 20 6 265 ## 2 RR Sarwan 315 11 3 205 ## 3 MN Samuels 209 19 5 188 ## 4 CH Gayle 198 18 8 176 ## 5 S Chanderpaul 181 6 7 152 ## 6 AB Barath 162 9 2 125 ## 7 DJ Bravo 139 7 2 102 ## 8 CS Baugh 102 5 NA 78 ## 9 LMP Simmons 78 5 4 67 ## 10 JO Holder 33 5 3 55 ## .. ... ... ... ... ... teamBattingScorecardOppnAllMatches(eng_nz_matches,"England","New Zealand") ## Total= 6472 ## Source: local data frame [47 x 5] ## ## batsman ballsPlayed fours sixes runs ## (fctr) (int) (int) (int) (dbl) ## 1 IR Bell 871 74 7 654 ## 2 JE Root 651 54 5 612 ## 3 PD Collingwood 619 34 15 514 ## 4 EJG Morgan 445 35 22 479 ## 5 AN Cook 616 49 3 464 ## 6 IJL Trott 421 26 1 362 ## 7 KP Pietersen 481 30 6 358 ## 8 JC Buttler 199 28 11 287 ## 9 OA Shah 323 17 6 274 ## 10 RS Bopara 350 21 NA 222 ## .. ... ... ... ... ... teamBatsmenPartnershiOppnAllMatches(sco_can_matches,"Scotland","Canada") ## Source: local data frame [20 x 2] ## ## batsman totalRuns ## (fctr) (dbl) ## 1 CS MacLeod 177 ## 2 MW Machan 68 ## 3 CJO Smith 43 ## 4 FRJ Coleman 40 ## 5 RR Watson 14 ## 6 JH Stander 12 ## 7 MA Leask 12 ## 8 RML Taylor 10 ## 9 KJ Coetzer 8 ## 10 GM Hamilton 7 ## 11 RM Haq 7 ## 12 PL Mommsen 6 ## 13 CM Wright 5 ## 14 JD Nel 5 ## 15 MH Cross 4 ## 16 SM Sharif 4 ## 17 JAR Blain 2 ## 18 NFI McCallum 1 ## 19 RD Berrington 1 ## 20 NS Poonia 0 ### 9. Team performances of bowlers (all matches with opposition) Like the function above the following tables provide the top bowlers of the countries in the matches against the oppoition. In India-Australia matches Ishant Sharma leads, in Pakistan-South Africa matches Shahid Afridi tops and so on. teamBowlingPerfOppnAllMatches(aus_ind_matches,"India","Australia") ## Source: local data frame [36 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 I Sharma 44 1 739 20 ## 2 Harbhajan Singh 40 0 926 15 ## 3 RA Jadeja 39 0 867 14 ## 4 IK Pathan 42 1 702 11 ## 5 UT Yadav 37 2 606 10 ## 6 P Kumar 27 0 501 10 ## 7 Z Khan 33 1 500 10 ## 8 S Sreesanth 34 0 454 10 ## 9 R Ashwin 43 0 684 9 ## 10 R Vinay Kumar 31 1 380 9 ## .. ... ... ... ... ... teamBowlingPerfOppnAllMatches(pak_sa_matches,main="Pakistan",opposition="South Africa") ## Source: local data frame [24 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 Shahid Afridi 38 0 1053 17 ## 2 Saeed Ajmal 39 0 658 14 ## 3 Mohammad Hafeez 38 0 774 13 ## 4 Mohammad Irfan 29 0 467 13 ## 5 Iftikhar Anjum 29 1 257 12 ## 6 Wahab Riaz 31 0 534 11 ## 7 Junaid Khan 32 0 429 10 ## 8 Sohail Tanvir 26 1 409 9 ## 9 Shoaib Akhtar 22 1 313 9 ## 10 Umar Gul 25 2 365 7 ## .. ... ... ... ... ... teamBowlingPerfOppnAllMatches(eng_nz_matches,"New Zealand","England") ## Source: local data frame [33 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 TG Southee 40 0 684 19 ## 2 KD Mills 36 1 742 17 ## 3 DL Vettori 35 0 561 16 ## 4 MJ McClenaghan 34 0 515 14 ## 5 SE Bond 17 1 205 11 ## 6 GD Elliott 20 0 194 10 ## 7 JEC Franklin 24 0 418 7 ## 8 KS Williamson 21 1 225 7 ## 9 TA Boult 18 2 195 7 ## 10 NL McCullum 30 0 425 6 ## .. ... ... ... ... ... teamBowlingPerfOppnAllMatches(sl_wi_matches,"Sri Lanka","West Indies") ## Source: local data frame [24 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 SL Malinga 28 1 280 11 ## 2 BAW Mendis 15 0 267 8 ## 3 KMDN Kulasekara 13 1 185 7 ## 4 AD Mathews 14 0 191 6 ## 5 M Muralitharan 20 1 157 6 ## 6 MF Maharoof 9 2 14 6 ## 7 WPUJC Vaas 7 2 82 5 ## 8 RAS Lakmal 7 0 55 4 ## 9 ST Jayasuriya 1 0 38 4 ## 10 HMRKB Herath 10 1 124 3 ## .. ... ... ... ... ... teamBowlingPerfOppnAllMatches(ken_ber_matches,"Kenya","Bermuda") ## Source: local data frame [9 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 JK Kamande 16 0 122 5 ## 2 HA Varaiya 13 1 64 5 ## 3 AS Luseno 6 0 32 4 ## 4 PJ Ongondo 7 0 39 3 ## 5 TM Odoyo 7 0 36 3 ## 6 LN Onyango 7 0 37 2 ## 7 SO Tikolo 18 0 81 1 ## 8 NN Odhiambo 14 1 76 1 ## 9 CO Obuya 4 0 20 0 ### 10. Team bowler’s wickets (all matches with opposition) This provided a graphical plot of the tables above teamBowlersWicketsOppnAllMatches(aus_ind_matches,"India","Australia") teamBowlersWicketsOppnAllMatches(aus_ind_matches,"Australia","India") teamBowlersWicketsOppnAllMatches(pak_sa_matches,"South Africa","Pakistan",top=10) m <-teamBowlersWicketsOppnAllMatches(eng_nz_matches,"England","Zealand",plot=FALSE) m ## Source: local data frame [20 x 2] ## ## bowler wickets ## (fctr) (int) ## 1 JM Anderson 20 ## 2 SCJ Broad 13 ## 3 ST Finn 12 ## 4 PD Collingwood 11 ## 5 GP Swann 10 ## 6 RJ Sidebottom 8 ## 7 CR Woakes 8 ## 8 A Flintoff 7 ## 9 LE Plunkett 6 ## 10 AU Rashid 6 ## 11 BA Stokes 6 ## 12 MS Panesar 5 ## 13 LJ Wright 4 ## 14 TT Bresnan 4 ## 15 DJ Willey 4 ## 16 JC Tredwell 3 ## 17 CT Tremlett 2 ## 18 RS Bopara 2 ## 19 CJ Jordan 2 ## 20 J Lewis 1 teamBowlersWicketsOppnAllMatches(ban_ire_matches,"Bangladesh","Ireland",top=7) ### 11. Team bowler vs batsmen (all matches with opposition) These plots show how the bowlers fared against the batsmen. It shows which of the opposing teams batsmen were able to score the most runs teamBowlersVsBatsmenOppnAllMatches(aus_ind_matches,'India',"Australia",top=5) teamBowlersVsBatsmenOppnAllMatches(pak_sa_matches,"Pakistan","South Africa",top=3) teamBowlersVsBatsmenOppnAllMatches(eng_nz_matches,"England","New Zealand") teamBowlersVsBatsmenOppnAllMatches(eng_nz_matches,"New Zealand","England") ### 12. Team bowler’s wicket kind (caught,bowled,etc) (all matches with opposition) The charts below show the wicket kind taken by the bowler (caught, bowled, lbw etc) teamBowlersWicketKindOppnAllMatches(aus_ind_matches,"India","Australia",plot=TRUE) m <- teamBowlersWicketKindOppnAllMatches(aus_ind_matches,"Australia","India",plot=FALSE) m[1:30,] ## bowler wicketKind wicketPlayerOut runs ## 1 GD McGrath caught SR Tendulkar 69 ## 2 SR Watson caught D Mongia 532 ## 3 MG Johnson lbw V Sehwag 1020 ## 4 B Lee caught R Dravid 671 ## 5 B Lee bowled M Kaif 671 ## 6 NW Bracken caught SK Raina 429 ## 7 GD McGrath caught IK Pathan 69 ## 8 NW Bracken lbw MS Dhoni 429 ## 9 MG Johnson lbw SR Tendulkar 1020 ## 10 MG Johnson bowled G Gambhir 1020 ## 11 SR Clark caught SR Tendulkar 254 ## 12 JR Hopes caught Yuvraj Singh 346 ## 13 SR Clark lbw RV Uthappa 254 ## 14 GB Hogg caught R Dravid 427 ## 15 MJ Clarke run out IK Pathan 212 ## 16 MJ Clarke stumped Harbhajan Singh 212 ## 17 MJ Clarke bowled RR Powar 212 ## 18 GB Hogg caught Z Khan 427 ## 19 GB Hogg caught MS Dhoni 427 ## 20 B Lee lbw G Gambhir 671 ## 21 MG Johnson lbw RV Uthappa 1020 ## 22 B Lee caught R Dravid 671 ## 23 GB Hogg bowled SR Tendulkar 427 ## 24 B Lee caught MS Dhoni 671 ## 25 JR Hopes caught RG Sharma 346 ## 26 GB Hogg lbw IK Pathan 427 ## 27 MG Johnson bowled Yuvraj Singh 1020 ## 28 GB Hogg caught and bowled Z Khan 427 ## 29 SR Clark bowled S Sreesanth 254 ## 30 JR Hopes caught SC Ganguly 346 teamBowlersWicketKindOppnAllMatches(sl_wi_matches,"Sri Lanka",'West Indies',plot=TRUE) ### 13. Team bowler’s wicket taken and runs conceded (all matches with opposition) teamBowlersWicketRunsOppnAllMatches(aus_ind_matches,"India","Australia") m <-teamBowlersWicketRunsOppnAllMatches(pak_sa_matches,"Pakistan","South Africa",plot=FALSE) m[1:30,] ## Source: local data frame [30 x 5] ## ## bowler overs maidens runs wickets ## (fctr) (int) (int) (dbl) (dbl) ## 1 Umar Gul 25 2 365 7 ## 2 Iftikhar Anjum 29 1 257 12 ## 3 Yasir Arafat 5 0 33 1 ## 4 Abdul Razzaq 16 0 290 4 ## 5 Mohammad Hafeez 38 0 774 13 ## 6 Shahid Afridi 38 0 1053 17 ## 7 Shoaib Malik 18 0 219 4 ## 8 Sohail Tanvir 26 1 409 9 ## 9 Abdur Rehman 25 0 301 4 ## 10 Mohammad Asif 10 1 204 2 ## .. ... ... ... ... ... ### 14. Plot of wins vs losses between teams. setwd("C:/software/cricket-package/york-test/yorkrData/ODI/ODI-matches") plotWinLossBetweenTeams("India","Sri Lanka") plotWinLossBetweenTeams('Pakistan',"South Africa",".") plotWinLossBetweenTeams('England',"New Zealand",".") plotWinLossBetweenTeams("Australia","West Indies",".") plotWinLossBetweenTeams('Bangladesh',"Zimbabwe",".") plotWinLossBetweenTeams('Scotland',"Ireland",".") ## Conclusion This post included all functions for all matches between any 2 opposing countries. As before the data frames are already available. You can load the data and begin to use them. If more insights from the dataframe are possible do go ahead. But please do attribute the source to Cricheet (http://cricsheet.org), my package yorkr and my blog. Do give the functions a spin for yourself. There are 2 more posts required for the introduction of MY yorkr package.So, Hasta la vista, baby! I’ll be back! Also see You may also like # Cricket analytics with cricketr!!! My ebook “Cricket analytics with cricketr’ has been published in Leanpub. You can now download the book (hot off the press!) for all formats to your favorite device (mobile, iPad, tablet, Kindle) from the Leanpub “Cricket analytics with cricketr”. The book has been published in the following formats namely • PDF (for your computer) • EPUB (for iPad or tablets. Save the file cricketr.epub to Google Drive/Dropbox and choose “Open in” iBooks for iPad) • MOBI (for Kindle. For this format, I suggest that you download & install SendToKindle for PC/Mac. You can then right click the downloaded cricketr.mobi and choose SendToKindle. You will need to login to your Kindle account) From Leanpub Leanpub uses a variable pricing model. I have priced the book attractively (I think!) at$2.50 with a minimum price of $0.00 (FREE!!! limited time offer!). The link is “Cricket analytics with cricketr This format works with all type Kindle, Android tablet, iPad. From Amazon You can also download for Kindle. The price is$2.50 (Rs 169/-). Cricket analytics with cricketr.

I am including my preface in the book below

Preface
Cricket has been the “national passion” of India for decades. As a boy I was also held in thrall by a strong cricketing passion like many. Cricket is a truly fascinating game! I would catch the sporting action with my friends as we crowded around a transistor that brought us live, breathless radio commentary. We also spent many hours glued to live cricket action on the early black and white TVs. This used to be an experience of sorts, as every now and then a part of the body of the players, would detach itself and stretch to the sides. But it was enjoyable all the same.

Nowadays broadcast technology has improved so much and we get detailed visual analysis of the how each bowler varies the swing and length of the delivery. We are also able to see the strokes of batsman in slow motion.   Similarly computing technology has also advanced by leaps and bounds and we can analyze players in great detail with a few lines of code in languages like R, Python etc.

In 2015, I completed Machine Learning from Stanford at Coursera.  I was looking around for data to play around with, when it suddenly struck me that I could do some regression analysis of batting records.  In the subsequent months, I took the Data Science Specialization from John Hopkins University, which triggered more ideas in me. One thing led to another and I managed to put together an R package called ‘cricketr’.  I developed this package over 7 months adding and refining functions. Finally, I managed to submit the package to CRAN.  During the development of the package for different formats of the game I wrote a series of posts in my blog.

This book is a collection of those cricket related posts.  There are 6 posts based on my R package cricketr. I have also included 2 earlier posts based on R which I wrote before I created my R package. Finally, I also include another 2 cricket posts based on Machine Learning in which I used the language Octave.

My cricketr’ package is a first, for cricket analytics, howzzat!  and I am certain that it won’t be the last. Cricket is a wonderful pitch for statisticians, data scientists and machine learning experts. So you can expect some cool packages in the years to come.

I had a great time developing the package. I hope you have a wonderful time reading this book. Do remember to download from “Cricket analytics with cricketr

Feel free to get in touch with me anytime through email included below

Tinniam V Ganesh
tvganesh.85@gmail.com
January 28, 2016

# Introduction

This should be last in the series of posts based on my R package cricketr. That is, unless some bright idea comes trotting along and light bulbs go on around my head.

In this post cricketr adapts to the Twenty20 International format. Now cricketr can handle stats from all 3 formats of the game namely Test matches, ODIs and Twenty20 International from ESPN Cricinfo. You should be able to install the package from GitHub and use the many of the functions available in the package.

If you are passionate about cricket, and love analyzing cricket performances, then check out my 2 racy books on cricket! In my books, I perform detailed yet compact analysis of performances of both batsmen, bowlers besides evaluating team & match performances in Tests , ODIs, T20s & IPL. You can buy my books on cricket from Amazon at $12.99 for the paperback and$4.99/$6.99 respectively for the kindle versions. The books can be accessed at Cricket analytics with cricketr and Beaten by sheer pace-Cricket analytics with yorkr A must read for any cricket lover! Check it out!! You can also read this post at Rpubs as twenty20-cricketr. Download this report as a PDF file from twenty20-cricketr.pdf Do check out my interactive Shiny app implementation using the cricketr package – Sixer – R package cricketr’s new Shiny avatar Note: If you would like to do a similar analysis for a different set of batsman and bowlers, you can clone/download my skeleton cricketr template from Github (which is the R Markdown file I have used for the analysis below). You will only need to make appropriate changes for the players you are interested in. Just a familiarity with R and R Markdown only is needed. I have chosen the Top 4 batsmen and top 4 bowlers based on ICC rankings and/or number of matches played. Batsmen 1. Virat Kohli (Ind) 2. Faf du Plessis (SA) 3. A J Finch (Aus) 4. Brendon McCullum (Aus) Bowlers 1. Samuel Badree (WI) 2. Sunil Narine (WI) 3. Ravichander Ashwin (Ind) 4. Ajantha Mendis (SL) I have explained the plots and added my own observations. Please feel free to draw your conclusions! The data for a particular player can be obtained with the getPlayerData() function. To do you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Virat Kohli, Sunil Narine etc. This will bring up a page which have the profile number for the player e.g. for Virat Kohli this would be http://www.espncricinfo.com/india/content/player/253802.html. The package can be installed directly from CRAN if (!require("cricketr")){ install.packages("cricketr",lib = "c:/test") } library(cricketr) or from Github library(devtools) install_github("tvganesh/cricketr") library(cricketr) The data for a particular player can be obtained with the getPlayerData() function. To do you will need to go to ESPN CricInfo Player and type in the name of the player for e.g Virat Kohli, Sunil Narine etc. This will bring up a page which have the profile number for the player e.g. for Virat Kohli this would be http://www.espncricinfo.com/india/content/player/253802.html. Hence, Kohlis profile is 253802. This can be used to get the data for Virat Kohli as shown below kohli <- getPlayerDataTT(253802,dir="..",file="kohli.csv",type="batting") The analysis is included below ## Analyses of Batsmen The following plots gives the analysis of the 4 ODI batsmen 1. Virat Kohli (Ind) – Innings-26, Runs-972, Average-46.28,Strike Rate-131.70 2. Faf du Plessis (SA) – Innings-24, Runs-805, Average-42.36,Strike Rate-135.75 3. A J Finch (Aus) – Innings-22, Runs-756, Average-39.78,Strike Rate-152.41 4. Brendon McCullum (NZ) – Innings-70, Runs-2140, Average-35.66,Strike Rate-136.21 ## Plot of 4s, 6s and the scoring rate in ODIs The 3 charts below give the number of 1. 4s vs Runs scored 2. 6s vs Runs scored 3. Balls faced vs Runs scored A regression line is fitted in each of these plots for each of the ODI batsmen A. Virat Kohli – The 1st plot shows that Kohli approximately hits about 5 4’s on his way to the 50s – The 2nd box plot of no of 6s and runs shows the range of runs when Kohli scored 1,2 or 4 6s. The dark line in the box shows the average runs when he scored those number of 6s. So when he scored 1 6 the average runs he scored was 45 – The 3rd plot shows the number of runs scored against the balls faced. It can be seen when Kohli faced 50 balls he had scored around ~ 70 runs par(mfrow=c(1,3)) par(mar=c(4,4,2,2)) batsman4s("./kohli.csv","Kohli") batsman6s("./kohli.csv","Kohli") batsmanScoringRateODTT("./kohli.csv","Kohli") dev.off() ## null device ## 1 B. Faf du Plessis par(mfrow=c(1,3)) par(mar=c(4,4,2,2)) batsman4s("./plessis.csv","Du Plessis") batsman6s("./plessis.csv","Du Plessis") batsmanScoringRateODTT("./plessis.csv","Du Plessss") dev.off() ## null device ## 1 C. A J Finch par(mfrow=c(1,3)) par(mar=c(4,4,2,2)) batsman4s("./finch.csv","A J Finch") batsman6s("./finch.csv","A J Finch") batsmanScoringRateODTT("./finch.csv","A J Finch") dev.off() ## null device ## 1 D. Brendon McCullum par(mfrow=c(1,3)) par(mar=c(4,4,2,2)) batsman4s("./mccullum.csv","McCullum") batsman6s("./mccullum.csv","McCullum") batsmanScoringRateODTT("./mccullum.csv","McCullum") dev.off() ## null device ## 1 ## Relative Mean Strike Rate This plot shows the Mean Strike Rate of the batsman in each run range. It can be seen the A J Finch has the best strike rate followed by B McCullum. par(mar=c(4,4,2,2)) frames <- list("./kohli.csv","./plessis.csv","finch.csv","mccullum.csv") names <- list("Kohli","Du Plessis","Finch","McCullum") relativeBatsmanSRODTT(frames,names) ## Relative Runs Frequency Percentage The plot below provides the average runs scored in each run range 0-5,5-10,10-15 etc. Clearly Kohli has the most runs scored in most of the runs ranges. . This is also evident in the fact that Kohli has the highest average. He is followed by McCullum frames <- list("./kohli.csv","./plessis.csv","finch.csv","mccullum.csv") names <- list("Kohli","Du Plessis","Finch","McCullum") relativeRunsFreqPerfODTT(frames,names) ## Percent 4’s,6’s in total runs scored The plot below shows the percentage of runs scored by way of 4s and 6s for each batsman. Du Plessis has the highest percentage of 4s, McCullum has the highest 6s. Finch has the highest percentage of 4s & 6s – 25.37 + 15.64= 41.01% rames <- list("./kohli.csv","./plessis.csv","finch.csv","mccullum.csv") names <- list("Kohli","Du Plessis","Finch","McCullum") runs4s6s <-batsman4s6s(frames,names) print(runs4s6s) ## Kohli Du Plessis Finch McCullum ## Runs(1s,2s,3s) 64.29 64.55 58.99 61.45 ## 4s 27.78 24.38 25.37 22.87 ## 6s 7.94 11.07 15.64 15.69 ## 3D plot of Runs vs Balls Faced and Minutes at Crease The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A prediction plane is then fitted based on the Balls Faced and Minutes at Crease to give the runs scored par(mfrow=c(1,2)) par(mar=c(4,4,2,2)) battingPerf3d("./kohli.csv","Kohli") battingPerf3d("./plessis.csv","Du Plessis") dev.off() ## null device ## 1 par(mfrow=c(1,2)) par(mar=c(4,4,2,2)) battingPerf3d("./finch.csv","A J Finch") battingPerf3d("./mccullum.csv","McCullum") dev.off() ## null device ## 1 ## Predicting Runs given Balls Faced and Minutes at Crease A hypothetical Balls faced and Minutes at Crease is used to predict the runs scored by each batsman based on the computed prediction plane BF <- seq( 5, 70,length=10) Mins <- seq(5,70,length=10) newDF <- data.frame(BF,Mins) kohli <- batsmanRunsPredict("./kohli.csv","Kohli",newdataframe=newDF) plessis <- batsmanRunsPredict("./plessis.csv","Du Plessis",newdataframe=newDF) finch <- batsmanRunsPredict("./finch.csv","A J Finch",newdataframe=newDF) mccullum <- batsmanRunsPredict("./mccullum.csv","McCullum",newdataframe=newDF) The predicted runs is displayed. As can be seen Finch has the best overall strike rate followed by McCullum. batsmen <-cbind(round(kohli$Runs),round(plessis$Runs),round(finch$Runs),round(mccullum$Runs)) colnames(batsmen) <- c("Kohli","Du Plessis","Finch","McCullum") newDF <- data.frame(round(newDF$BF),round(newDF\$Mins))
colnames(newDF) <- c("BallsFaced","MinsAtCrease")
predictedRuns <- cbind(newDF,batsmen)
predictedRuns
##    BallsFaced MinsAtCrease Kohli Du Plessis Finch McCullum
## 1           5            5     2          1     5        3
## 2          12           12    12         10    22       16
## 3          19           19    22         19    40       28
## 4          27           27    31         28    57       41
## 5          34           34    41         37    74       54
## 6          41           41    51         47    91       66
## 7          48           48    60         56   108       79
## 8          56           56    70         65   125       91
## 9          63           63    79         74   142      104
## 10         70           70    89         84   159      117

## Highest runs likelihood

The plots below the runs likelihood of batsman. This uses K-Means Kohli has the highest likelihood of scoring runs 34.2% likely to score 66 runs. Du Plessis has 25% likelihood to score 53 runs, A. Virat Kohli

batsmanRunsLikelihood("./kohli.csv","Kohli")

## Summary of  Kohli 's runs scoring likelihood
## **************************************************
##
## There is a 23.08 % likelihood that Kohli  will make  10 Runs in  10 balls over 13  Minutes
## There is a 42.31 % likelihood that Kohli  will make  29 Runs in  23 balls over  30  Minutes
## There is a 34.62 % likelihood that Kohli  will make  66 Runs in  47 balls over 63  Minutes

B. Faf Du Plessis

batsmanRunsLikelihood("./plessis.csv","Du Plessis")

## Summary of  Du Plessis 's runs scoring likelihood
## **************************************************
##
## There is a 62.5 % likelihood that Du Plessis  will make  14 Runs in  11 balls over 19  Minutes
## There is a 25 % likelihood that Du Plessis  will make  53 Runs in  40 balls over  50  Minutes
## There is a 12.5 % likelihood that Du Plessis  will make  94 Runs in  61 balls over 90  Minutes

C. A J Finch

batsmanRunsLikelihood("./finch.csv","A J Finch")

## Summary of  A J Finch 's runs scoring likelihood
## **************************************************
##
## There is a 20 % likelihood that A J Finch  will make  95 Runs in  54 balls over 70  Minutes
## There is a 25 % likelihood that A J Finch  will make  42 Runs in  27 balls over  35  Minutes
## There is a 55 % likelihood that A J Finch  will make  8 Runs in  8 balls over 12  Minutes

D. Brendon McCullum

batsmanRunsLikelihood("./mccullum.csv","McCullum")

## Summary of  McCullum 's runs scoring likelihood
## **************************************************
##
## There is a 50.72 % likelihood that McCullum  will make  11 Runs in  10 balls over 13  Minutes
## There is a 28.99 % likelihood that McCullum  will make  36 Runs in  27 balls over  37  Minutes
## There is a 20.29 % likelihood that McCullum  will make  74 Runs in  48 balls over 70  Minutes

## Moving Average of runs over career

The moving average for the 4 batsmen indicate the following. It must be noted that there is not sufficient data yet on Twenty20 Internationals. Kpohli, Du Plessis and Finch average only 26 innings while McCullum has close to 70. So the moving average while an indication will regress towards the mean over time.

1. The moving average of Kohli and Du Plessis is on the way up.
2. McCullum has a consistent performance while Finch had a brief burst in 2013-2014
par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
batsmanMovingAverage("./kohli.csv","Kohli")
batsmanMovingAverage("./plessis.csv","Du Plessis")
batsmanMovingAverage("./finch.csv","A J Finch")
batsmanMovingAverage("./mccullum.csv","McCullum")

dev.off()
## null device
##           1

## Analysis of bowlers

1. Samuel Badree (WI) – Innings-22, Runs -464, Wickets – 31, Econ Rate : 5.39
2. Sunil Narine (WI)- Innings-31,Runs-666, Wickets – 38 , Econ Rate : 5.70
3. Ravichander Ashwin (Ind)- Innings-26, Runs- 732, Wickets – 25, Econ Rate : 7.32
4. Ajantha Mendis (SL)- Innings-39, Runs – 952,Wickets – 66, Econ Rate : 6.45

The plot shows the frequency with which the bowlers have taken 1,2,3 etc wickets. The most wickets taken is by Ajantha Mendis (6 wickets)

# Wicket Frequency percentage

This plot gives the percentage of wickets for each wickets (1,2,3…etc)

par(mfrow=c(1,4))
par(mar=c(4,4,2,2))
bowlerWktsFreqPercent("./mendis.csv","Mendis")
bowlerWktsFreqPercent("./narine.csv","Narine")
bowlerWktsFreqPercent("./ashwin.csv","Ashwin")

dev.off()
## null device
##           1

## Wickets Runs plot

The plot below gives a boxplot of the runs ranges for each of the wickets taken by the bowlers. The ends of the box indicate the 25% and 75% percentile of runs scored for the wickets taken and the dark balck line is the average runs conceded.

par(mfrow=c(1,4))
par(mar=c(4,4,2,2))
bowlerWktsRunsPlot("./mendis.csv","Mendis")
bowlerWktsRunsPlot("./narine.csv","Narine")
bowlerWktsRunsPlot("./ashwin.csv","Ashwin")

dev.off()
## null device
##           1

This plot below shows the average number of deliveries needed by the bowler to take the wickets (1,2,3 etc)

par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
bowlerWktRateTT("./mendis.csv","Mendis")

dev.off()
## null device
##           1
par(mfrow=c(1,2))
par(mar=c(4,4,2,2))
bowlerWktRateTT("./narine.csv","Narine")
bowlerWktRateTT("./ashwin.csv","Ashwin")

dev.off()
## null device
##           1

## Relative bowling performance

The plot below shows that Narine has the most wickets in the 2 -4 range followed by Mendis

frames <- list("./badree.csv","./mendis.csv","narine.csv","ashwin.csv")
relativeBowlingPerf(frames,names)

## Relative Economy Rate against wickets taken

The economy rate can be deduced as follows from the plot below. Narine has a good economy rate around 1 & 4 wickets, Ashwin around 2 wickets and Badree around 3. wickets

frames <- list("./badree.csv","./mendis.csv","narine.csv","ashwin.csv")
relativeBowlingERODTT(frames,names)

## Relative Wicket Rate

The relative wicket rate plots the mean number of deliveries needed to take the wickets namely (1,2,3,4). For e.g. Narine needed an average of 22 deliveries to take 1 wicket and 22.5,23.2, 24 deliveries to take 2,3 & 4 wickets respectively

frames <- list("./badree.csv","./mendis.csv","narine.csv","ashwin.csv")
relativeWktRateTT(frames,names)

Moving average of wickets over career

par(mfrow=c(2,2))
par(mar=c(4,4,2,2))
bowlerMovingAverage("./mendis.csv","Mendis")
bowlerMovingAverage("./narine.csv","Narine")
bowlerMovingAverage("./ashwin.csv","Ashwin")
## null device
##           1

# Key findings

Here are some key conclusions

Twenty 20 batsmen

1. Kohli has the a very consistent performance scoring high runs in the different run ranges. Kohli also has a 34.2% likelihood to score 6 runs. He is followed by McCullum for consisten performance
2. Finch has a best strike rate followed by McCullum.
3. Du Plessis has the highest percentage of 4s and McCullum has the percentage of 6s. Finch is superior in the percentage of runs scored in 4s and 6s
4. For a hypothetical balls faced and minutes at crease, Finch does best followed by McCullum
5. Kohli’s & Du Plessis Twenty20 career is on a upswing. Can they maintain the momentum. McCullum is consistent

Twenty20 bowlers

1. Narine has the highest wickets percentage for different wickets taken followed by Mendis
2. Mendis has taken 1,2,3,4,6 wickets in 24 deliveries
3. Narine has the lowest economy rate for 1 & 4 wickets, Ashwin for 2 wickets and Badree for 3 wickets. Mendis is comparatively expensive
4. Narine needed the least deliveries to get 1 (22.5) & 2 (23.2) wickets, Mendis needed 20.5 deliveries and Ashwin 19 deliveries for 4 wickets

Key takeaways 1. If all the above batsment and bowlers were in the same team we expect

1. Finch would be most useful when the run rate has to be greatly accelerated followed by McCullum
2. If the need is to consolidate, then Kohli is the best man for the job followed by McCullum
3. Overall McCullum is the best bet for Twenty20
4. When it comes to bowling Narine wins hands down as he has the most wickets, a good economy rate and a very good attack rate. So Narine is great bet for providing a vital breakthrough.

Also see my other posts in R

You may also like