summary#

Methods to display examples and their label issues in an object detection dataset. Here each image can have multiple objects, each with its own bounding box and class label.

Functions:

`object_counts_per_image`([labels, ...])	Return the number of annotated and predicted objects for each image in the dataset.
`bounding_box_size_distribution`([labels, ...])	Return the distribution over sizes of annotated and predicted bounding boxes across the dataset, broken down by each class.
`class_label_distribution`([labels, ...])	Returns the distribution of class labels associated with all annotated bounding boxes (or predicted bounding boxes) in the dataset.
`get_sorted_bbox_count_idxs`(labels, predictions)	Returns a tuple of idxs and bounding box counts of images sorted from highest to lowest number of bounding boxes.
`plot_class_size_distributions`(labels, ...[, ...])	Plots the size distributions for bounding boxes for each class.
`plot_class_distribution`(labels, predictions)	Plots the distribution of class labels associated with all annotated bounding boxes and predicted bounding boxes in the dataset.
`visualize`(image, *[, label, prediction, ...])	Display the annotated bounding boxes (given labels) and predicted bounding boxes (model predictions) for a particular image.
`get_average_per_class_confusion_matrix`(...)	Compute a confusion matrix dictionary for each class containing the average number of True Positive, False Positive, and False Negative detections from the object detection model across a range of Intersection over Union thresholds.
`calculate_per_class_metrics`(labels, predictions)	Calculate the object detection model's precision, recall, and F1 score for each class in the dataset.

cleanlab.object_detection.summary.object_counts_per_image(labels=None, predictions=None, *, auxiliary_inputs=None)[source]#

Return the number of annotated and predicted objects for each image in the dataset.

This method can help you discover images with abnormally many/few object annotations.

Parameters:

labels –
Annotated boxes and class labels in the original dataset, which may contain some errors. This is a list of N dictionaries such that labels[i] contains the given labels for the i-th image in the following format: {'bboxes': np.ndarray((L,4)), 'labels': np.ndarray((L,)), 'image_name': str} where L is the number of annotated bounding boxes for the i-th image and bboxes[l] is a bounding box of coordinates in [x1,y1,x2,y2] format with given class label labels[j]. image_name is an optional part of the labels that can be used to later refer to specific images.

Note: Here, (x1,y1) corresponds to the top-left and (x2,y2) corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>, Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>].

For more information on proper labels formatting, check out the MMDetection library.
predictions –
Predictions output by a trained object detection model. For the most accurate results, predictions should be out-of-sample to avoid overfitting, eg. obtained via cross-validation. This is a list of N np.ndarray such that predictions[i] corresponds to the model prediction for the i-th image. For each possible class k in 0, 1, …, K-1: predictions[i][k] is a np.ndarray of shape (M,5), where M is the number of predicted bounding boxes for class k. Here the five columns correspond to [x1,y1,x2,y2,pred_prob], where [x1,y1,x2,y2] are coordinates of the bounding box predicted by the model and pred_prob is the model’s confidence in the predicted class label for this bounding box.

Note: Here, (x1,y1) corresponds to the top-left and (x2,y2) corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>, Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>]. The last column, pred_prob, represents the predicted probability that the bounding box contains an object of the class k.

For more information see the MMDetection package for an example object detection library that outputs predictions in the correct format.
auxiliary_inputs (optional) – Auxiliary inputs to be used in the computation of counts. The auxiliary_inputs can be computed using rank._get_valid_inputs_for_compute_scores. It is internally computed from the given labels and predictions.

Return type:

Tuple[List, List]

Returns:

object_counts (Tuple[List, List]) – A tuple containing two lists. The first is an array of shape (N,) containing the number of annotated objects for each image in the dataset. The second is an array of shape (N,) containing the number of predicted objects for each image in the dataset.

cleanlab.object_detection.summary.bounding_box_size_distribution(labels=None, predictions=None, *, auxiliary_inputs=None, class_names=None, sort=False)[source]#

Return the distribution over sizes of annotated and predicted bounding boxes across the dataset, broken down by each class.

This method can help you find annotated/predicted boxes for a particular class that are abnormally big/small.

Parameters:

labels – Annotated boxes and class labels in the original dataset, which may contain some errors. Refer to documentation for this argument in object_counts_per_image for further details.
predictions – Predictions output by a trained object detection model. Refer to documentation for this argument in object_counts_per_image for further details.
auxiliary_inputs (optional) – Auxiliary inputs to be used in the computation of counts. Refer to documentation for this argument in object_counts_per_image for further details.
class_names (optional) – A dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}. You can use this argument to control the classes for which the size distribution is computed.
sort (bool) – If True, the returned dictionaries are sorted by the number of instances of each class in the dataset in descending order.

Return type:

Tuple[Dict[Any, List], Dict[Any, List]]

Returns:

bbox_sizes (Tuple[Dict[Any, List], Dict[Any, List]]) – A tuple containing two dictionaries. Each maps each class label to a list of the sizes of annotated bounding boxes for that class in the label and prediction datasets, respectively.

cleanlab.object_detection.summary.class_label_distribution(labels=None, predictions=None, *, auxiliary_inputs=None, class_names=None)[source]#

Returns the distribution of class labels associated with all annotated bounding boxes (or predicted bounding boxes) in the dataset.

This method can help you understand which classes are: rare or over/under-predicted by the model overall.

Parameters:

labels – Annotated boxes and class labels in the original dataset, which may contain some errors. Refer to documentation for this argument in object_counts_per_image for further details.
predictions – Predictions output by a trained object detection model. Refer to documentation for this argument in object_counts_per_image for further details.
auxiliary_inputs (optional) – Auxiliary inputs to be used in the computation of counts. Refer to documentation for this argument in object_counts_per_image for further details.
class_names (optional) – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}.

Return type:

Tuple[Dict[Any, float], Dict[Any, float]]

Returns:

class_distribution (Tuple[Dict[Any, float], Dict[Any, float]]) – A tuple containing two dictionaries. The first is a dictionary mapping each class label to its frequency in the dataset annotations. The second is a dictionary mapping each class label to its frequency in the model predictions across all images in the dataset.

cleanlab.object_detection.summary.get_sorted_bbox_count_idxs(labels, predictions)[source]#

Returns a tuple of idxs and bounding box counts of images sorted from highest to lowest number of bounding boxes.

This plot can help you discover images with abnormally many/few object annotations.

Parameters:

labels – Annotated boxes and class labels in the original dataset, which may contain some errors. Refer to documentation for this argument in object_counts_per_image for further details.
predictions – Predictions output by a trained object detection model. Refer to documentation for this argument in object_counts_per_image for further details.

Returns:

sorted_idxs (List[Tuple[int, int]], List[Tuple[int, int]]) – A tuple containing two lists. The first is an array of shape (N,) containing the number of annotated objects for each image in the dataset. The second is an array of shape (N,) containing the number of predicted objects for each image in the dataset.

cleanlab.object_detection.summary.plot_class_size_distributions(labels, predictions, class_names=None, class_to_show=10, **kwargs)[source]#

Plots the size distributions for bounding boxes for each class.

This plot can help you find annotated/predicted boxes for a particular class that are abnormally big/small.

Parameters:

labels – Annotated boxes and class labels in the original dataset, which may contain some errors. Refer to documentation for this argument in object_counts_per_image for further details.
predictions – Predictions output by a trained object detection model. Refer to documentation for this argument in object_counts_per_image for further details.
class_names (optional) – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}. You can use this argument to control the classes for which the size distribution is plotted.
class_to_show (optional) – The number of classes to show in the plots. Classes over class_to_show are hidden. If this argument is provided, then the classes are sorted by the number of instances in the dataset. Defaults to MAX_CLASS_TO_SHOW which is set to 10.
kwargs – Additional keyword arguments to pass to plt.show() (matplotlib.pyplot.show).

cleanlab.object_detection.summary.plot_class_distribution(labels, predictions, class_names=None, **kwargs)[source]#

Plots the distribution of class labels associated with all annotated bounding boxes and predicted bounding boxes in the dataset.

This plot can help you understand which classes are rare or over/under-predicted by the model overall.

Parameters:

labels – Annotated boxes and class labels in the original dataset, which may contain some errors. Refer to documentation for this argument in object_counts_per_image for further details.
predictions – Predictions output by a trained object detection model. Refer to documentation for this argument in object_counts_per_image for further details.
class_names (optional) – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}.
kwargs – Additional keyword arguments to pass to plt.show() (matplotlib.pyplot.show).

cleanlab.object_detection.summary.visualize(image, *, label=None, prediction=None, prediction_threshold=None, overlay=True, class_names=None, figsize=None, save_path=None, **kwargs)[source]#

Display the annotated bounding boxes (given labels) and predicted bounding boxes (model predictions) for a particular image. Given labels are shown in red, model predictions in blue.

Parameters:

image (Union[str, ndarray, TypeVar(Image)]) – Image object loaded into memory or full path to the image file. If path is provided, image is loaded into memory.
label (Optional[Dict[str, Any]]) –
The given label for a single image in the format {'bboxes': np.ndarray((L,4)), 'labels': np.ndarray((L,))} where L is the number of bounding boxes for the i-th image and bboxes[j] is in the format [x1,y1,x2,y2] with given label labels[j].

Note: Here, (x1,y1) corresponds to the top-left and (x2,y2) corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>, Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>].
prediction (Optional[ndarray]) –
A prediction for a single image in the format np.ndarray((K,)) and prediction[k] is of shape np.ndarray(N,5) where M is the number of predicted bounding boxes for class k and the five columns correspond to [x,y,x,y,pred_prob] where [x1,y1,x2,y2] are the bounding box coordinates predicted by the model and pred_prob is the model’s confidence in predictions[i].

Note: Here, (x1,y1) corresponds to the top-left and (x2,y2) corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>, Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>]. The last column, pred_prob, represents the predicted probability that the bounding box contains an object of the class k.
prediction_threshold (Optional[float]) – All model-predicted bounding boxes with confidence (pred_prob) below this threshold are omitted from the visualization.
overlay (bool) – If True, display a single image with given labels and predictions overlaid. If False, display two images (side by side) with the left image showing the model predictions and the right image showing the given label.
class_names (Optional[Dict[Any, Any]]) – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}.
save_path (Optional[str]) – Path to save figure at. If a path is provided, the figure is saved. To save in a specific image format, add desired file extension to the end of save_path. Allowed file extensions are: ‘png’, ‘pdf’, ‘ps’, ‘eps’, and ‘svg’.
figsize (Optional[Tuple[int, int]]) – Optional figure size for plotting the image. Corresponds to matplotlib.figure.figsize.
kwargs – Additional keyword arguments to pass to plt.show() (matplotlib.pyplot.show).

Return type:

None

cleanlab.object_detection.summary.get_average_per_class_confusion_matrix(labels, predictions, num_procs=1, class_names=None)[source]#

Compute a confusion matrix dictionary for each class containing the average number of True Positive, False Positive, and False Negative detections from the object detection model across a range of Intersection over Union thresholds.

At each IoU threshold, the metrics are calculated as follows: - True Positive (TP): Instances where the model correctly identifies the class with IoU above the threshold. - False Positive (FP): Instances where the model predicts the class, but IoU is below the threshold. - False Negative (FN): Instances where the ground truth class is not predicted by the model.

The average confusion matrix provides insights into the model strengths and potential biases.

Note: lower TP at certain IoU thresholds does not necessarily imply that everything else is FP, instead it indicates that, at those specific IoU thresholds, the model is not performing as well in terms of correctly identifying class instances. The other metrics (FP and FN) provide additional information about the model’s behavior.

Note: Since we average over many IoU thresholds, ‘TP’, ‘FP’, and ‘FN’ may contain float values representing the average across these thresholds.

Parameters:

labels (List[Dict[str, Any]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in object_detection.filter.find_label_issues for further details.
predictions (List[ndarray]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in object_detection.filter.find_label_issues for further details.
num_procs (int) – Number of processes for parallelization. Default is 1.
class_names (Optional[Dict[Any, Any]]) – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}

Return type:

Dict[Union[int, str], Dict[str, float]]

Returns:

avg_metrics (dict) – A distionary containing the average confusion matrix.

The default range of Intersection over Union thresholds is from 0.5 to 0.95 with a step size of 0.05.

cleanlab.object_detection.summary.calculate_per_class_metrics(labels, predictions, num_procs=1, class_names=None)[source]#

Calculate the object detection model’s precision, recall, and F1 score for each class in the dataset.

These metrics can help you identify model strengths and weaknesses, and provide reference statistics for model evaluation and comparisons.

Parameters:

labels (List[Dict[str, Any]]) – A list of N dictionaries such that labels[i] contains the given labels for the i-th image. Refer to documentation for this argument in object_detection.filter.find_label_issues for further details.
predictions (List[ndarray]) – A list of N np.ndarray such that predictions[i] corresponds to the model predictions for the i-th image. Refer to documentation for this argument in object_detection.filter.find_label_issues for further details.
num_procs (int) – Number of processes for parallelization. Default is 1.
class_names – Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format {"integer-label": "original-class-name"}

Return type:

Dict[Union[int, str], Dict[str, float]]

Returns:

per_class_metrics (dict) – A dictionary containing per-class metrics computed from the object detection model’s average confusion matrix values across a range of Intersection over Union thresholds.

The default range of Intersection over Union thresholds is from 0.5 to 0.95 with a step size of 0.05.