rank#
Methods to rank and score images in an object detection dataset (object detection data), based on how likely they are to contain label errors.
Functions:

Computes a label quality score for each image of the 

Convert label quality scores to a list of indices of images with issues sorted from most to least severe cut off at threshold. 

Returns an array of overlooked box scores for each image. 

Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the edges of this box are not badly located. 

Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the class label for this box was not accidentally swapped with another class. 

Aggregates all perbox scores within an image to return a single quality score for the image rather than for individual boxes within it. 
 cleanlab.object_detection.rank.get_label_quality_scores(labels, predictions, *, aggregation_weights=None, overlapping_label_check=True, verbose=True)[source]#
Computes a label quality score for each image of the
N
images in the dataset.For object detection datasets, the label quality score for an image estimates how likely it has been correctly labeled. Lower scores indicate images whose annotation is more likely imperfect. Annotators may have mislabeled an image because they:
overlooked an object (missing annotated bounding box),
chose the wrong class label for an annotated box in the correct location,
imperfectly annotated the location/edges of a bounding box.
Any of these annotation errors should lead to an image with a lower label quality score. This quality score is between 0 and 1.
1  clean label (given label is likely correct).
0  dirty label (given label is likely incorrect).
 Parameters:
labels (
List
[Dict
[str
,Any
]]) – A list ofN
dictionaries such thatlabels[i]
contains the given labels for the ith image. Refer to documentation for this argument infind_label_issues
for further details.predictions (
List
[ndarray
]) – A list ofN
np.ndarray
such thatpredictions[i]
corresponds to the model predictions for the ith image. Refer to documentation for this argument infind_label_issues
for further details.verbose (
bool
, default= True
) – Set toFalse
to suppress all print statements.aggregation_weights (
Optional
[Dict
[str
,float
]]) –Optional dictionary to specify weights for aggregating quality scores for subtype of label issue into an overall label quality score for the image. Its keys are: “overlooked”, “swap”, “badloc”, and values should be nonnegative weights that sum to 1. Increase one of these weights to prioritize images with bounding boxes that were either: missing in the annotations (overlooked object), annotated with the wrong class label (class for the object should be swapped to another class), or annotated in a suboptimal location (badly located).
swapped examples, bad location examples, and overlooked examples. It is important to ensure that the weights are nonnegative values and that their sum equals 1.0.
overlapping_label_check (
bool
, default= True
) – If True, boxes annotated with more than one class label have their swap score penalized. Set this to False if you are not concerned when two very similar boxes exist with different class labels in the given annotations.
 Return type:
ndarray
 Returns:
label_quality_scores
– Array of shape(N, )
of scores between 0 and 1, one per image in the object detection dataset. Lower scores indicate images that are more likely mislabeled.
 cleanlab.object_detection.rank.issues_from_scores(label_quality_scores, *, threshold=0.1)[source]#
Convert label quality scores to a list of indices of images with issues sorted from most to least severe cut off at threshold.
Returns the list of indices of images with issues sorted from most to least severe cut off at threshold.
 Parameters:
label_quality_scores (
ndarray
) – Array of shape(N, )
of scores between 0 and 1, one per image in the object detection dataset. Lower scores indicate images are more likely to contain a label issue.threshold (
float
) – Label quality scores above the threshold are not considered to be label issues. The corresponding examples’ indices are omitted from the returned array.
 Return type:
ndarray
 Returns:
issue_indices
– Array of issue indices sorted from most to least severe who’s label quality scores fall below the threshold if one is provided.
 cleanlab.object_detection.rank.compute_overlooked_box_scores(*, labels=None, predictions=None, alpha=None, high_probability_threshold=None, auxiliary_inputs=None)[source]#
Returns an array of overlooked box scores for each image. This is a helper method mostly for advanced users.
An overlooked box error is when an image contains an object that is one of the given classes but there is no annotated bounding box around it. Score per highconfidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.
Each image has
L
annotated bounding boxes andM
predicted bounding boxes. A score is calculated for each predicted box in each of theN
images in dataset.Note:
M
andL
can be a different values for each image, as the number of annotated and predicted boxes varies. Parameters:
labels (
Optional
[List
[Dict
[str
,Any
]]]) – A list ofN
dictionaries such thatlabels[i]
contains the given labels for the ith image. Refer to documentation for this argument infind_label_issues
for further details.predictions (
Optional
[List
[ndarray
]]) – A list ofN
np.ndarray
such thatpredictions[i]
corresponds to the model predictions for the ith image. Refer to documentation for this argument infind_label_issues
for further details.alpha (
Optional
[float
]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.high_probability_threshold (
Optional
[float
]) – Optional probability threshold that determines which predicted boxes are considered highconfidence when computing overlooked scores. If not provided, a good default is used.auxiliary_inputs (
Optional
[List
[TypeVar
(AuxiliaryTypesDict
)]]) –Optional list of
N
dictionaries containing keys for subparts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the ith image, auxiliary_inputs[i] should contain following keys: pred_labels: np.ndarray
Array of predicted classes for ith image of shape
(M,)
.
 pred_label_probs: np.ndarray
Array of predicted class probabilities for ith image of shape
(M,)
.
 pred_bboxes: np.ndarray
Array of predicted bounding boxes for ith image of shape
(M, 4)
.
 lab_labels: np.ndarray
Array of given label classed for ith image of shape
(L,)
.
 lab_bboxes: np.ndarray
Array of given label bounding boxes for ith image of shape
(L, 4)
.
 similarity_matrix: np.ndarray
Similarity matrix between labels and predictions ith image.
 min_possible_similarity: float
Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.
 Return type:
List
[ndarray
] Returns:
scores_overlooked
– A list ofN
numpy arrays where scores_overlooked[i] is an array of sizeM
of overlooked scores per predicted box for the ith image.
 cleanlab.object_detection.rank.compute_badloc_box_scores(*, labels=None, predictions=None, alpha=None, low_probability_threshold=None, auxiliary_inputs=None)[source]#
Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the edges of this box are not badly located. This is a helper method mostly for advanced users.
A badly located box error is when a box has the correct label but incorrect coordinates so it does not correctly encapsulate the entire object it is for. Score per highconfidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.
Each image has
L
annotated bounding boxes andM
predicted bounding boxes. A score is calculated for each predicted box in each of theN
images in dataset.Note:
M
andL
can be a different values for each image, as the number of annotated and predicted boxes varies. Parameters:
labels (
Optional
[List
[Dict
[str
,Any
]]]) – A list ofN
dictionaries such thatlabels[i]
contains the given labels for the ith image. Refer to documentation for this argument infind_label_issues
for further details.predictions (
Optional
[List
[ndarray
]]) – A list ofN
np.ndarray
such thatpredictions[i]
corresponds to the model predictions for the ith image. Refer to documentation for this argument infind_label_issues
for further details.alpha (
Optional
[float
]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.low_probability_threshold (
Optional
[float
]) – Optional minimum probability threshold that determines which predicted boxes are considered when computing badly located scores. If not provided, a good default is used.auxiliary_inputs (
Optional
[List
[TypeVar
(AuxiliaryTypesDict
)]]) –Optional list of
N
dictionaries containing keys for subparts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the ith image, auxiliary_inputs[i] should contain following keys: pred_labels: np.ndarray
Array of predicted classes for ith image of shape
(M,)
.
 pred_label_probs: np.ndarray
Array of predicted class probabilities for ith image of shape
(M,)
.
 pred_bboxes: np.ndarray
Array of predicted bounding boxes for ith image of shape
(M, 4)
.
 lab_labels: np.ndarray
Array of given label classed for ith image of shape
(L,)
.
 lab_bboxes: np.ndarray
Array of given label bounding boxes for ith image of shape
(L, 4)
.
 similarity_matrix: np.ndarray
Similarity matrix between labels and predictions ith image.
 min_possible_similarity: float
Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.
 Return type:
List
[ndarray
] Returns:
scores_badloc
– A list ofN
numpy arrays where scores_badloc[i] is an array of sizeL
badly located scores per annotated box for the ith image.
 cleanlab.object_detection.rank.compute_swap_box_scores(*, labels=None, predictions=None, alpha=None, high_probability_threshold=None, overlapping_label_check=True, auxiliary_inputs=None)[source]#
Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the class label for this box was not accidentally swapped with another class. This is a helper method mostly for advanced users.
A swapped box error occurs when a bounding box should be labeled as a class different to what the current label is. Score per highconfidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.
Each image has
L
annotated bounding boxes andM
predicted bounding boxes. A score is calculated for each predicted box in each of theN
images in dataset.Note:
M
andL
can be a different values for each image, as the number of annotated and predicted boxes varies. Parameters:
labels (
Optional
[List
[Dict
[str
,Any
]]]) – A list ofN
dictionaries such thatlabels[i]
contains the given labels for the ith image. Refer to documentation for this argument infind_label_issues
for further details.predictions (
Optional
[List
[ndarray
]]) – A list ofN
np.ndarray
such thatpredictions[i]
corresponds to the model predictions for the ith image. Refer to documentation for this argument infind_label_issues
for further details.alpha (
Optional
[float
]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.high_probability_threshold (
Optional
[float
]) – Optional probability threshold that determines which predicted boxes are considered highconfidence when computing overlooked scores. If not provided, a good default is used.overlapping_label_check (
bool
, default= True
) – If True, boxes annotated with more than one class label have their swap score penalized. Set this to False if you are not concerned when two very similar boxes exist with different class labels in the given annotations.auxiliary_inputs (
Optional
[List
[TypeVar
(AuxiliaryTypesDict
)]]) –Optional list of
N
dictionaries containing keys for subparts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the ith image, auxiliary_inputs[i] should contain following keys: pred_labels: np.ndarray
Array of predicted classes for ith image of shape
(M,)
.
 pred_label_probs: np.ndarray
Array of predicted class probabilities for ith image of shape
(M,)
.
 pred_bboxes: np.ndarray
Array of predicted bounding boxes for ith image of shape
(M, 4)
.
 lab_labels: np.ndarray
Array of given label classed for ith image of shape
(L,)
.
 lab_bboxes: np.ndarray
Array of given label bounding boxes for ith image of shape
(L, 4)
.
 similarity_matrix: np.ndarray
Similarity matrix between labels and predictions ith image.
 min_possible_similarity: float
Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.
 Return type:
List
[ndarray
] Returns:
scores_swap
– A list ofN
numpy arrays where scores_swap[i] is an array of sizeL
swap scores per annotated box for the ith image.
 cleanlab.object_detection.rank.pool_box_scores_per_image(box_scores, *, temperature=None)[source]#
Aggregates all perbox scores within an image to return a single quality score for the image rather than for individual boxes within it. This is a helper method mostly for advanced users to be used with the outputs of
object_detection.rank.compute_overlooked_box_scores
,object_detection.rank.compute_badloc_box_scores
, andobject_detection.rank.compute_swap_box_scores
.Score per image is between 0 and 1, with lower values indicating we are more confident image contains an error.
 Parameters:
box_scores (
List
[ndarray
]) – A list ofN
numpy arrays where box_scores[i] is an array of badly located scores per box for the ith image.temperature (
Optional
[float
]) – Optional temperature of the softmin function where a lower value suggests softmin acts closer to min. If not provided, a good default is used.
 Return type:
ndarray
 Returns:
image_scores
– An array of sizeN
whereimage_scores[i]
represents the score for the ith image.