filter#
Methods to identify which examples have label issues in a classification dataset.
The documentation below assumes a dataset with N
examples and K
classes.
This module considers two types of datasets:
standard (multiclass) classification where each example is labeled as belonging to exactly one of K classes (e.g.
labels = np.array([0,0,1,0,2,1])
)multilabel classification where each example can be labeled as belonging to multiple classes (e.g.
labels = [[1,2],[1],[0],[],...]
)
Functions:

Identifies potentially bad labels in a classification dataset using confident learning. 
A baseline approach that uses the confusion matrix of 


A simple baseline approach that considers 
 cleanlab.filter.find_label_issues(labels, pred_probs, *, return_indices_ranked_by=None, rank_by_kwargs=None, filter_by='prune_by_noise_rate', multi_label=False, frac_noise=1.0, num_to_remove_per_class=None, min_examples_per_class=1, confident_joint=None, n_jobs=None, verbose=False)[source]#
Identifies potentially bad labels in a classification dataset using confident learning.
Returns a boolean mask for the entire dataset where
True
represents an example identified with a label issue andFalse
represents an example that seems correctly labeled.Instead of a mask, you can obtain indices of the examples with label issues in your dataset (sorted by issue severity) by specifying the return_indices_ranked_by argument. This determines which label quality score is used to quantify severity, and is useful to view only the topJ most severe issues in your dataset.
The number of indices returned as issues is controlled by frac_noise: reduce its value to identify fewer label issues. If you aren’t sure, leave this set to 1.0.
Tip: if you encounter the error “pred_probs is not defined”, try setting
n_jobs=1
. Parameters:
labels (
np.ndarray
orlist
) – A discrete vector of noisy labels for a classification dataset, i.e. some labels may be erroneous. Format requirements: for dataset with K classes, each label must be integer in 0, 1, …, K1. For a standard (multiclass) classification dataset where each example is labeled with one class, labels should be 1D array of shape(N,)
, for example:labels = [1,0,2,1,1,0...]
. For a multilabel classification dataset where each example can belong to multiple (or no) classes, labels should be an iterable of iterables (e.g.List[List[int]]
) whose ith element corresponds to list of classes that ith example belongs to (e.g.labels = [[1,2],[1],[0],[],...]
).pred_probs (
np.ndarray
, optional) –An array of shape
(N, K)
of modelpredicted class probabilities,P(label=kx)
. Each row of this matrix corresponds to an example x and contains the modelpredicted probabilities that x belongs to each possible class, for each of the K classes. The columns must be ordered such that these probabilities correspond to class 0, 1, …, K1.Note: Returned label issues are most accurate when they are computed based on outofsample pred_probs from your model. To obtain outofsample predicted probabilities for every datapoint in your dataset, you can use crossvalidation. This is encouraged to get better results.
return_indices_ranked_by (
{None, 'self_confidence', 'normalized_margin', 'confidence_weighted_entropy'}
, defaultNone
) –Determines what is returned by this method: either a boolean mask or list of indices np.ndarray. If
None
, this function returns a boolean mask (True
if example at index is label error). If notNone
, this function returns a sorted array of indices of examples with label issues (instead of a boolean mask). Indices are sorted by label quality score which can be one of:'normalized_margin'
:normalized margin (p(label = k)  max(p(label != k)))
'self_confidence'
:[pred_probs[i][labels[i]] for i in label_issues_idx]
'confidence_weighted_entropy'
:entropy(pred_probs) / self_confidence
rank_by_kwargs (
dict
, optional) – Optional keyword arguments to pass into scoring functions for ranking by label quality score (seerank.get_label_quality_scores
).filter_by (
{'prune_by_class', 'prune_by_noise_rate', 'both', 'confident_learning', 'predicted_neq_given'}
, default'prune_by_noise_rate'
) –Method to determine which examples are flagged as having label issue, so you can filter/prune them from the dataset. Options:
'prune_by_noise_rate'
: filters examples with high probability of being mislabeled for every nondiagonal in the confident joint (see prune_counts_matrix in filter.py). These are the examples where (with high confidence) the given label is unlikely to match the predicted label for the example.'prune_by_class'
: filters the examples with smallest probability of belonging to their given class label for every class.'both'
: filters only those examples that would be filtered by both'prune_by_noise_rate'
and'prune_by_class'
.'confident_learning'
: filters the examples counted as part of the offdiagonals of the confident joint. These are the examples that are confidently predicted to be a different label than their given label.'predicted_neq_given'
: filters examples for which the predicted class (i.e. argmax of the predicted probabilities) does not match the given label.
multi_label (
bool
, optional) – IfTrue
, labels should be an iterable (e.g. list) of iterables, containing a list of class labels for each example, instead of just a single label. The multilabel setting supports classification tasks where an example can belong to more than 1 class or none of the classes (rather than exactly one class as in standard multiclass classification). Example of a multilabeled labels input:[[0,1], [1], [0,2], [0,1,2], [0], [1], [], ...]
. This says the first example in dataset belongs to both class 0 and class 1, according to its given label. Each row of pred_probs no longer needs to sum to 1 in multilabel settings, since one example can now belong to multiple classes simultaneously.frac_noise (
float
, default1.0
) –Used to only return the “top”
frac_noise * num_label_issues
. The choice of which “top” label issues to return is dependent on the filter_by method used. It works by reducing the size of the offdiagonals of the joint distribution of given labels and true labels proportionally by frac_noise prior to estimating label issues with each method. This parameter only applies for filter_by=both, filter_by=prune_by_class, and filter_by=prune_by_noise_rate methods and currently is unused by other methods. Whenfrac_noise=1.0
, return all “confident” estimated noise indices (recommended).frac_noise * number_of_mislabeled_examples_in_class_k. Note: specifying frac_noise is not yet supported if multi_label is True.
num_to_remove_per_class (
array_like
) –An iterable of length K, the number of classes. E.g. if K = 3,
num_to_remove_per_class=[5, 0, 1]
would return the indices of the 5 most likely mislabeled examples in class 0, and the most likely mislabeled example in class 2.Note
Only set this parameter if
filter_by='prune_by_class'
. You may use withfilter_by='prune_by_noise_rate'
, but ifnum_to_remove_per_class=k
, then either k1, k, or k+1 examples may be removed for any class due to rounding error. If you need exactly ‘k’ examples removed from every class, you should usefilter_by='prune_by_class'
.min_examples_per_class (
int
, default1
) – Minimum number of examples per class to avoid flagging as label issues. This is useful to avoid deleting too much data from one class when pruning noisy examples in datasets with rare classes.confident_joint (
np.ndarray
, optional) – An array of shape(K, K)
representing the confident joint, the matrix used for identifying label issues, which estimates a confident subset of the joint distribution of the noisy and true labels,P_{noisy label, true label}
. Entry(j, k)
in the matrix is the number of examples confidently counted into the pair of(noisy label=j, true label=k)
classes. The confident_joint can be computed usingcount.compute_confident_joint
. If not provided, it is computed from the given (noisy) labels and pred_probs. If multi_label is True, confident_joint should instead be a onevsrest array with shape(K, 2, 2)
as returned bycount.compute_confident_joint
function.n_jobs (optional) – Number of processing threads used by multiprocessing. Default
None
sets to the number of cores on your CPU. Set this to 1 to disable parallel processing (if its causing issues). Windows users may see a speedup withn_jobs=1
.verbose (optional) – If
True
, prints when multiprocessing happens.
 Return type:
ndarray
 Returns:
label_issues (
np.ndarray
) – If return_indices_ranked_by left unspecified, returns a boolean mask for the entire dataset whereTrue
represents a label issue andFalse
represents an example that is accurately labeled with high confidence. If return_indices_ranked_by is specified, returns a shorter array of indices of examples identified to have label issues (i.e. those indices where the mask would beTrue
), sorted by likelihood that the corresponding label is correct.Note
Obtain the indices of examples with label issues in your dataset by setting return_indices_ranked_by.
 cleanlab.filter.find_label_issues_using_argmax_confusion_matrix(labels, pred_probs, *, calibrate=True, filter_by='prune_by_noise_rate')[source]#
A baseline approach that uses the confusion matrix of
argmax(pred_probs)
and labels as the confident joint and then uses cleanlab (confident learning) to find the label issues using this matrix.The only difference between this and
find_label_issues
is that it uses the confusion matrix based on the argmax and given label instead of using the confident joint fromcount.compute_confident_joint
. Parameters:
labels (
np.ndarray
) – An array of shape(N,)
of noisy labels, i.e. some labels may be erroneous. Elements must be in the set 0, 1, …, K1, where K is the number of classes.pred_probs (
np.ndarray
) – An array of shape(N, K)
of modelpredicted probabilities,P(label=kx)
. Each row of this matrix corresponds to an example x and contains the modelpredicted probabilities that x belongs to each possible class, for each of the K classes. The columns must be ordered such that these probabilities correspond to class 0, 1, …, K1. pred_probs should have been computed using 3 (or higher) fold crossvalidation.calibrate (
bool
, defaultTrue
) – Set toTrue
to calibrate the confusion matrix created bypred != given labels
. This calibration adjusts the confusion matrix / confident joint so that the prior (given noisy labels) is correct based on the original labels.filter_by (
str
, default'prune_by_noise_rate'
) – See filter_by argument offind_label_issues
.
 Return type:
ndarray
 Returns:
label_issues_mask (
np.ndarray
) – A boolean mask for the entire dataset whereTrue
represents a label issue andFalse
represents an example that is accurately labeled with high confidence.
Note
Multilabel classification is not supported in this method.
 cleanlab.filter.find_predicted_neq_given(labels, pred_probs, *, multi_label=False)[source]#
A simple baseline approach that considers
argmax(pred_probs) != labels
as the examples with label issues. Parameters:
labels (
np.ndarray
orlist
) – Labels in the same format expected by thefind_label_issues
function.pred_probs (
np.ndarray
) – Predictedprobabilities in the same format expected by thefind_label_issues
function.multi_label (
bool
, optional) – Whether each example may have multiple labels or not (see documentation for thefind_label_issues
function).
 Return type:
ndarray
 Returns:
label_issues_mask (
np.ndarray
) – A boolean mask for the entire dataset whereTrue
represents a label issue andFalse
represents an example that is accurately labeled with high confidence.