underperforming_group#

Classes:

UnderperformingGroupIssueManager(datalab[, ...])

Manages issues related to underperforming group examples.

class cleanlab.datalab.internal.issue_manager.underperforming_group.UnderperformingGroupIssueManager(datalab, metric=None, threshold=0.1, k=10, clustering_kwargs={}, min_cluster_samples=5, **_)[source]#

Bases: IssueManager

Manages issues related to underperforming group examples.

Note: The min_cluster_samples argument should not be confused with the min_samples argument of sklearn.cluster.DBSCAN.

Examples

>>> from cleanlab import Datalab
>>> import numpy as np
>>> X = np.random.normal(size=(50, 2))
>>> y = np.random.randint(2, size=50)
>>> pred_probs = X / X.sum(axis=1, keepdims=True)
>>> data = {"X": X, "y": y}
>>> lab = Datalab(data, label_name="y")
>>> issue_types={"underperforming_group": {"clustering_kwargs": {"eps": 0.5}}}
>>> lab.find_issues(pred_probs=pred_probs, features=X, issue_types=issue_types)

Attributes:

`description`	Short text that summarizes the type of issues handled by this IssueManager.
`issue_name`	Returns a key that is used to store issue summary results about the assigned Lab.
`verbosity_levels`	A dictionary of verbosity levels and their corresponding dictionaries of report items to print.
`OUTLIER_CLUSTER_LABELS`	Specifies labels considered as outliers by the clustering algorithm.
`NO_UNDERPERFORMING_CLUSTER_ID`	Constant to signify absence of any underperforming cluster.
`issue_score_key`	Returns a key that is used to store issue score results about the assigned Lab.
`info`
`issues`
`summary`

Methods:

`find_issues`(pred_probs[, features, cluster_ids])	Finds occurrences of this particular issue in the dataset.
`set_knn_graph`(features, find_issues_kwargs)	rtype: `csr_matrix`
`perform_clustering`(knn_graph)	Perform clustering of datapoints using a knn graph as distance matrix.
`filter_cluster_ids`(cluster_ids)	Remove outlier clusters and return IDs of clusters with at least self.min_cluster_samples number of datapoints.
`get_worst_cluster`(cluster_ids, ...)	Get ID and quality score of underperforming cluster.
`collect_info`(knn_graph, n_clusters, ...)	Collects data for the info attribute of the Datalab.
`make_summary`(score)	Construct a summary dataframe.
`report`(issues, summary, info[, ...])	Compose a report of the issues found by this IssueManager.

description: ClassVar[str]#

Short text that summarizes the type of issues handled by this IssueManager.

issue_name: ClassVar[str] = 'underperforming_group'#: Returns a key that is used to store issue summary results about the assigned Lab.

verbosity_levels: ClassVar[Dict[int, List[str]]]#

A dictionary of verbosity levels and their corresponding dictionaries of report items to print.

Example

>>> verbosity_levels = {
...     0: [],
...     1: ["some_info_key"],
...     2: ["additional_info_key"],
... }

OUTLIER_CLUSTER_LABELS: ClassVar[Tuple[int]] = (-1,)#: Specifies labels considered as outliers by the clustering algorithm.

NO_UNDERPERFORMING_CLUSTER_ID: ClassVar[int] = -2#: Constant to signify absence of any underperforming cluster.

find_issues(pred_probs, features=None, cluster_ids=None, **kwargs)[source]#

Finds occurrences of this particular issue in the dataset.

Computes the issues and summary dataframes. Calls collect_info to compute the info dict.

Return type:: None

set_knn_graph(features, find_issues_kwargs)[source]#

Return type:: csr_matrix

perform_clustering(knn_graph)[source]#

Perform clustering of datapoints using a knn graph as distance matrix.

Return type:: ndarray[Any, dtype[int64]]

Args:: knn_graph (csr_matrix): Sparse Distance Matrix.
Returns:: cluster_ids (npt.NDArray[np.int_]): Cluster IDs for each datapoint.

filter_cluster_ids(cluster_ids)[source]#

Remove outlier clusters and return IDs of clusters with at least self.min_cluster_samples number of datapoints.

Return type:: ndarray[Any, dtype[int64]]

Args:: cluster_ids (npt.NDArray[np.int_]): Cluster IDs for each datapoint.
Returns:: unique_cluster_ids (npt.NDArray[np.int_]): List of unique cluster IDs after removing outlier clusters and clusters with less than self.min_cluster_samples number of datapoints.

get_worst_cluster(cluster_ids, unique_cluster_ids, labels, pred_probs)[source]#

Get ID and quality score of underperforming cluster.

Return type:: Tuple[int, float]

Args:: cluster_ids (npt.NDArray[np.int_]): _description_ unique_cluster_ids (npt.NDArray[np.int_]): _description_ labels (npt.NDArray): _description_ pred_probs (npt.NDArray): _description_
Returns:: Tuple[int, float]: (Underperforming Cluster ID, Cluster Quality Score)

collect_info(knn_graph, n_clusters, cluster_ids, performed_clustering, worst_cluster_id)[source]#: Collects data for the info attribute of the Datalab. :rtype: Dict[str, Any]

Note

This method is called by find_issues() after find_issues() has set the issues and summary dataframes as instance attributes.

issue_score_key: ClassVar[str] = 'underperforming_group_score'#: Returns a key that is used to store issue score results about the assigned Lab.

classmethod make_summary(score)#

Construct a summary dataframe.

Parameters:: score (float) – The overall score for this issue.
Return type:: DataFrame
Returns:: summary – A summary dataframe.

classmethod report(issues, summary, info, num_examples=5, verbosity=0, include_description=False, info_to_omit=None)#

Compose a report of the issues found by this IssueManager.

Parameters:

issues (DataFrame) –

An issues dataframe.

Example

>>> import pandas as pd
>>> issues = pd.DataFrame(
...     {
...         "is_X_issue": [True, False, True],
...         "X_score": [0.2, 0.9, 0.4],
...     },
... )

summary (DataFrame) –

The summary dataframe.

Example

>>> summary = pd.DataFrame(
...     {
...         "issue_type": ["X"],
...         "score": [0.5],
...     },
... )

info (Dict[str, Any]) –

The info dict.

Example

>>> info = {
...     "A": "val_A",
...     "B": ["val_B1", "val_B2"],
... }

num_examples (int) – The number of examples to print.
verbosity (int) – The verbosity level of the report.
include_description (bool) – Whether to include a description of the issue in the report.

Return type:

str

Returns:

report_str – A string containing the report.

info: Dict[str, Any]#

issues: pd.DataFrame#

summary: pd.DataFrame#