Welcome to MLBase’s documentation!¶

Data Repetition¶

repeach(a, n)¶

Repeat each element in vector a for n times. Here n can be either a scalar or a vector with the same length as a.

using MLBase

repeach(1:3, 2) # --> [1, 1, 2, 2, 3, 3]
repeach(1:3, [3,2,1]) # --> [1, 1, 1, 2, 2, 3]

repeachcol(a, n)¶

Repeat each column in matrix a for n times. Here n can be either a scalar or a vector with length(n) == size(a,2).

repeachrow(a, n)¶

Repeat each row in matrix a for n times. Here n can be either a scalar or a vector with length(n) == size(a,1).

Label Processing¶

In machine learning, we often need to first attach each class with an integer label. This package provides a type LabelMap that captures the association between discrete values (e.g a finite set of strings) and integer labels.

Together with LabelMap, the package also provides a function labelmap to construct the map from a sequence of discrete values, and a function labelencode to map discrete values to integer labels.

julia> lm = labelmap(["a", "a", "b", "b", "c"])
LabelMap (with 3 labels):
[1] a
[2] b
[3] c

julia> labelencode(lm, "b")
2

julia> labelencode(lm, ["a", "c", "b"])
3-element Array{Int64,1}:
 1
 3
 2

Note that labelencode can be applied to either single value or an array.

The package also provides a function groupindices to group indices based on associated labels.

julia> groupindices(3, [1, 1, 1, 2, 2, 3, 2])
3-element Array{Array{Int64,1},1}:
 [1,2,3]
 [4,5,7]
 [6]

 # using lm as constructed above
julia> groupindices(lm, ["a", "a", "c", "b", "b"])
3-element Array{Array{Int64,1},1}:
 [1,2]
 [4,5]
 [3]

Classification Performance¶

correctrate(gt, pred)¶

Compute correct rate of predictions given by pred w.r.t. the ground truths given in gt.

errorrate(gt, pred)¶

Compute error rate of predictions given by pred w.r.t. the ground truths given in gt.

confusmat(k, gt, pred)¶

Compute the confusion matrix of the predictions given by pred w.r.t. the ground truths given in gt. Here, k is the number of classes.

It returns an integer matrix R of size (k, k), such that R(i, j) == countnz((gt .== i) & (pred .== j)).

Examples:

julia> gt = [1, 1, 1, 2, 2, 2, 3, 3];

julia> pred = [1, 1, 2, 2, 2, 3, 3, 3];

julia> C = confusmat(3, gt, pred)   # compute confusion matrix
3x3 Array{Int64,2}:
 2  1  0
 0  2  1
 0  0  2

julia> C ./ sum(C, 2)   # normalize per class
3x3 Array{Float64,2}:
 0.666667  0.333333  0.0
 0.0       0.666667  0.333333
 0.0       0.0       1.0

julia> trace(C) / length(gt)  # compute correct rate from confusion matrix
0.75

julia> correctrate(gt, pred)
0.75

Hit rate (for retrieval tasks)¶

hitrate(gt, ranklist, k)¶

Compute the hitrate of rank k for a ranked list of predictions given by ranklist w.r.t. the ground truths given in gt.

Particularly, if gt[i] is contained in ranklist[1:k, i], then the prediction for the i-th sample is said to be hit within rank ``k``. The hitrate of rank k is the fraction of predictions that hit within rank k.

hitrates(gt, ranklist, ks)¶

Compute hit-rates of multiple ranks (as given by a vector ks). It returns a vector of hitrates r, where r[i] corresponding to the rank ks[i].

Note that computing hit-rates for multiple ranks jointly is more efficient than computing them separately.

Receiver Operating Characteristics (ROC)¶

Receiver Operating Characteristics (ROC) is often used to measure the performance of a detector, thresholded classifier, or a verification algorithm.

immutable ROCNums{T<:Real}
    p::T    # positive in ground-truth
    n::T    # negative in ground-truth
    tp::T   # correct positive prediction
    tn::T   # correct negative prediction
    fp::T   # (incorrect) positive prediction when ground-truth is negative
    fn::T   # (incorrect) negative prediction when ground-truth is positive
end

Cross Validation Schemes¶

Kfold(n, k)¶

k-fold cross validation over a set of n samples, which are randomly partitioned into k disjoint validation sets of nearly the same sizes. This generates k training subsets of length about n*(1-1/k).

julia> collect(Kfold(10, 3))
3-element Array{Any,1}:
 [1,3,4,6,7,8,10]
 [2,5,7,8,9,10]
 [1,2,3,4,5,6,9]

StratifiedKfold(strata, k)¶

Like Kfold, but indexes in each strata (defined by unique values of an iterator strata) are distributed approximately equally across the k folds. Each strata should have at least k members.

julia> collect(StratifiedKfold([:a, :a, :a, :b, :b, :c, :c, :a, :b, :c], 3))
3-element Array{Any,1}:
 [1,2,4,6,8,9,10]
 [3,4,5,7,8,10]
 [1,2,3,5,6,7,9]

LOOCV(n)¶

Leave-one-out cross validation over a set of n samples.

julia> collect(LOOCV(4))
4-element Array{Any,1}:
 [2,3,4]
 [1,3,4]
 [1,2,4]
 [1,2,3]

RandomSub(n, sn, k)¶

Repetitively random subsampling. Particularly, this generates k subsets of length sn from a data set with n samples.

julia> collect(RandomSub(10, 5, 3))
3-element Array{Any,1}:
 [1,2,5,8,9]
 [2,5,7,8,10]
 [1,3,5,6,7]

StratifiedRandomSum(strata, sn, k)¶

Like RandomSub, but indexes in each strata (defined by unique values of an iterator strata) are distributed approximately equally across the k subsets. sn should be greater than the number of strata, so that each stratum can be represented in each subset.

julia> collect(StratifiedRandomSub([:a, :a, :a, :b, :b, :c, :c, :a, :b, :c], 7, 5))
5-element Array{Any,1}:
 [1,2,3,4,6,7,9]
 [1,3,4,6,8,9,10]
 [1,3,5,7,8,9,10]
 [1,2,4,7,8,9,10]
 [1,2,3,4,5,6,10]

Cross Validation Function¶

The package also provides a function cross_validate as below to run a cross validation procedure.

cross_validate(estfun, evalfun, n, gen)¶

Run a cross validation procedure.

Parameters:

estfun – The estimation function, which takes a vector of training indices as input and returns a learned model, as: model = estfun(train_inds) evalfun – The evaluation function, which takes a model and a vector of testing indices as input and returns a score that indicates the goodness of the model, as score = evalfun(model, test_inds) n – The total number of samples. gen – An iterable object that provides training indices, e.g., one of the cross validation schemes listed above.

Returns:

a vector of scores obtained in the multiple runs.

Example:

# A simple example to demonstrate the use of cross validation
#
# Here, we consider a simple model: using a mean vector to represent
# a set of samples. The goodness of the model is assessed in terms
# of the RMSE (root-mean-square-error) evaluated on the testing set
#

using MLBase

# functions
compute_center(X::Matrix{Float64}) = vec(mean(X, 2))

compute_rmse(c::Vector{Float64}, X::Matrix{Float64}) =
    sqrt(mean(sum(abs2(X .- c),1)))

# data
const n = 200
const data = [2., 3.] .+ randn(2, n)

# cross validation
scores = cross_validate(
    inds -> compute_center(data[:, inds]),        # training function
    (c, inds) -> compute_rmse(c, data[:, inds]),  # evaluation function
    n,              # total number of samples
    Kfold(n, 5))    # cross validation plan: 5-fold

# get the mean and std of the scores
(m, s) = mean_and_std(scores)

Please refer to examples/crossval.jl for the entire script.