rank

Methods to rank and score images in an object detection dataset (object detection data), based on how likely they are to contain label errors.

Functions:

get_label_quality_scores(labels, predictions, *)	Computes a label quality score for each image of the N images in the dataset.
issues_from_scores(label_quality_scores, *)	Convert label quality scores to a list of indices of images with issues sorted from most to least severe cut off at threshold.
compute_overlooked_box_scores(*[, labels, ...])	Returns an array of overlooked box scores for each image.
compute_badloc_box_scores(*[, labels, ...])	Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the edges of this box are not badly located.
compute_swap_box_scores(*[, labels, ...])	Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the class label for this box was not accidentally swapped with another class.
pool_box_scores_per_image(box_scores, *[, ...])	Aggregates all per-box scores within an image to return a single quality score for the image rather than for individual boxes within it.

cleanlab.object_detection.rank.get_label_quality_scores(labels, predictions, *, aggregation_weights=None, overlapping_label_check=True, verbose=True)

Computes a label quality score for each image of the N images in the dataset.

For object detection datasets, the label quality score for an image estimates how likely it has been correctly labeled. Lower scores indicate images whose annotation is more likely imperfect. Annotators may have mislabeled an image because they:

• overlooked an object (missing annotated bounding box),

• chose the wrong class label for an annotated box in the correct location,

• imperfectly annotated the location/edges of a bounding box.

Any of these annotation errors should lead to an image with a lower label quality score. This quality score is between 0 and 1.

• 1 - clean label (given label is likely correct).

• 0 - dirty label (given label is likely incorrect).

Parameters:

• labels (List[Dict[str, Any]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• predictions (List[ndarray]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• verbose (bool, default = True) – Set to False to suppress all print statements.

• aggregation_weights (Optional[Dict[str, float]]) –

  Optional dictionary to specify weights for aggregating quality scores for subtype of label issue into an overall label quality score for the image. Its keys are: “overlooked”, “swap”, “badloc”, and values should be nonnegative weights that sum to 1. Increase one of these weights to prioritize images with bounding boxes that were either: missing in the annotations (overlooked object), annotated with the wrong class label (class for the object should be swapped to another class), or annotated in a suboptimal location (badly located).

  swapped examples, bad location examples, and overlooked examples. It is important to ensure that the weights are non-negative values and that their sum equals 1.0.

• overlapping_label_check (bool, default = True) – If True, boxes annotated with more than one class label have their swap score penalized. Set this to False if you are not concerned when two very similar boxes exist with different class labels in the given annotations.

Return type:

ndarray

Returns:

label_quality_scores – Array of shape (N, ) of scores between 0 and 1, one per image in the object detection dataset. Lower scores indicate images that are more likely mislabeled.

cleanlab.object_detection.rank.issues_from_scores(label_quality_scores, *, threshold=0.1)

Convert label quality scores to a list of indices of images with issues sorted from most to least severe cut off at threshold.

Returns the list of indices of images with issues sorted from most to least severe cut off at threshold.

Parameters:

• label_quality_scores (ndarray) – Array of shape (N, ) of scores between 0 and 1, one per image in the object detection dataset. Lower scores indicate images are more likely to contain a label issue.

• threshold (float) – Label quality scores above the threshold are not considered to be label issues. The corresponding examples’ indices are omitted from the returned array.

Return type:

ndarray

Returns:

issue_indices – Array of issue indices sorted from most to least severe who’s label quality scores fall below the threshold if one is provided.

cleanlab.object_detection.rank.compute_overlooked_box_scores(*, labels=None, predictions=None, alpha=None, high_probability_threshold=None, auxiliary_inputs=None)

Returns an array of overlooked box scores for each image. This is a helper method mostly for advanced users.

An overlooked box error is when an image contains an object that is one of the given classes but there is no annotated bounding box around it. Score per high-confidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.

Each image has L annotated bounding boxes and M predicted bounding boxes. A score is calculated for each predicted box in each of the N images in dataset.

Note: M and L can be a different values for each image, as the number of annotated and predicted boxes varies.

Parameters:

• labels (Optional[List[Dict[str, Any]]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• predictions (Optional[List[ndarray]]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• alpha (Optional[float]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.

• high_probability_threshold (Optional[float]) – Optional probability threshold that determines which predicted boxes are considered high-confidence when computing overlooked scores. If not provided, a good default is used.

• auxiliary_inputs (Optional[List[TypeVar(AuxiliaryTypesDict)]]) –

  Optional list of N dictionaries containing keys for sub-parts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the i-th image, auxiliary_inputs[i] should contain following keys:

  • pred_labels: np.ndarray

    Array of predicted classes for i-th image of shape (M,).

  • pred_label_probs: np.ndarray

    Array of predicted class probabilities for i-th image of shape (M,).

  • pred_bboxes: np.ndarray

    Array of predicted bounding boxes for i-th image of shape (M, 4).

  • lab_labels: np.ndarray

    Array of given label classed for i-th image of shape (L,).

  • lab_bboxes: np.ndarray

    Array of given label bounding boxes for i-th image of shape (L, 4).

  • similarity_matrix: np.ndarray

    Similarity matrix between labels and predictions i-th image.

  • min_possible_similarity: float

    Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.

Return type:

List[ndarray]

Returns:

scores_overlooked – A list of N numpy arrays where scores_overlooked[i] is an array of size M of overlooked scores per predicted box for the i-th image.

cleanlab.object_detection.rank.compute_badloc_box_scores(*, labels=None, predictions=None, alpha=None, low_probability_threshold=None, auxiliary_inputs=None)

Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the edges of this box are not badly located. This is a helper method mostly for advanced users.

A badly located box error is when a box has the correct label but incorrect coordinates so it does not correctly encapsulate the entire object it is for. Score per high-confidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.

Each image has L annotated bounding boxes and M predicted bounding boxes. A score is calculated for each predicted box in each of the N images in dataset.

Note: M and L can be a different values for each image, as the number of annotated and predicted boxes varies.

Parameters:

• labels (Optional[List[Dict[str, Any]]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• predictions (Optional[List[ndarray]]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• alpha (Optional[float]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.

• low_probability_threshold (Optional[float]) – Optional minimum probability threshold that determines which predicted boxes are considered when computing badly located scores. If not provided, a good default is used.

• auxiliary_inputs (Optional[List[TypeVar(AuxiliaryTypesDict)]]) –

  Optional list of N dictionaries containing keys for sub-parts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the i-th image, auxiliary_inputs[i] should contain following keys:

  • pred_labels: np.ndarray

    Array of predicted classes for i-th image of shape (M,).

  • pred_label_probs: np.ndarray

    Array of predicted class probabilities for i-th image of shape (M,).

  • pred_bboxes: np.ndarray

    Array of predicted bounding boxes for i-th image of shape (M, 4).

  • lab_labels: np.ndarray

    Array of given label classed for i-th image of shape (L,).

  • lab_bboxes: np.ndarray

    Array of given label bounding boxes for i-th image of shape (L, 4).

  • similarity_matrix: np.ndarray

    Similarity matrix between labels and predictions i-th image.

  • min_possible_similarity: float

    Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.

Return type:

List[ndarray]

Returns:

scores_badloc – A list of N numpy arrays where scores_badloc[i] is an array of size L badly located scores per annotated box for the i-th image.

cleanlab.object_detection.rank.compute_swap_box_scores(*, labels=None, predictions=None, alpha=None, high_probability_threshold=None, overlapping_label_check=True, auxiliary_inputs=None)

Returns a numeric score for each annotated bounding box in each image, estimating the likelihood that the class label for this box was not accidentally swapped with another class. This is a helper method mostly for advanced users.

A swapped box error occurs when a bounding box should be labeled as a class different to what the current label is. Score per high-confidence predicted bounding box is between 0 and 1, with lower values indicating boxes we are more confident were overlooked in the given label.

Each image has L annotated bounding boxes and M predicted bounding boxes. A score is calculated for each predicted box in each of the N images in dataset.

Note: M and L can be a different values for each image, as the number of annotated and predicted boxes varies.

Parameters:

• labels (Optional[List[Dict[str, Any]]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• predictions (Optional[List[ndarray]]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in find_label_issues for further details.

• alpha (Optional[float]) – Optional weighting between IoU and Euclidean distance when calculating similarity between predicted and annotated boxes. High alpha means weighting IoU more heavily over Euclidean distance. If no alpha is provided, a good default is used.

• high_probability_threshold (Optional[float]) – Optional probability threshold that determines which predicted boxes are considered high-confidence when computing overlooked scores. If not provided, a good default is used.

• overlapping_label_check (bool, default = True) – If True, boxes annotated with more than one class label have their swap score penalized. Set this to False if you are not concerned when two very similar boxes exist with different class labels in the given annotations.

• auxiliary_inputs (Optional[List[TypeVar(AuxiliaryTypesDict)]]) –

  Optional list of N dictionaries containing keys for sub-parts of label and prediction per image. Useful to minimize computation when computing multiple box scores for a single set of images. For the i-th image, auxiliary_inputs[i] should contain following keys:

  • pred_labels: np.ndarray

    Array of predicted classes for i-th image of shape (M,).

  • pred_label_probs: np.ndarray

    Array of predicted class probabilities for i-th image of shape (M,).

  • pred_bboxes: np.ndarray

    Array of predicted bounding boxes for i-th image of shape (M, 4).

  • lab_labels: np.ndarray

    Array of given label classed for i-th image of shape (L,).

  • lab_bboxes: np.ndarray

    Array of given label bounding boxes for i-th image of shape (L, 4).

  • similarity_matrix: np.ndarray

    Similarity matrix between labels and predictions i-th image.

  • min_possible_similarity: float

    Minimum possible similarity value greater than 0 between labels and predictions for the entire dataset.

Return type:

List[ndarray]

Returns:

scores_swap – A list of N numpy arrays where scores_swap[i] is an array of size L swap scores per annotated box for the i-th image.

cleanlab.object_detection.rank.pool_box_scores_per_image(box_scores, *, temperature=None)

Aggregates all per-box scores within an image to return a single quality score for the image rather than for individual boxes within it. This is a helper method mostly for advanced users to be used with the outputs of object_detection.rank.compute_overlooked_box_scores, object_detection.rank.compute_badloc_box_scores, and object_detection.rank.compute_swap_box_scores.

Score per image is between 0 and 1, with lower values indicating we are more confident image contains an error.

Parameters:

• box_scores (List[ndarray]) – A list of N numpy arrays where box_scores[i] is an array of badly located scores per box for the i-th image.

• temperature (Optional[float]) – Optional temperature of the softmin function where a lower value suggests softmin acts closer to min. If not provided, a good default is used.

Return type:

ndarray

Returns:

image_scores – An array of size N where image_scores[i] represents the score for the i-th image.