issue_finder

Note

This module is not intended to be used directly by users. It is used by the cleanlab.datalab.datalab module. Specifically, it is used by the Datalab.find_issues method.

Module for the IssueFinder class, which is responsible for configuring, creating and running issue managers.

It determines which types of issues to look for, instatiates the IssueManagers via a factory, run the issue managers (IssueManager.find_issues), and collects the results to DataIssues.

Note

This module is not intended to be used directly. Instead, use the public-facing Datalab.find_issues method.

Classes:

IssueFinder(datalab, task[, verbosity])

The IssueFinder class is responsible for managing the process of identifying issues in the dataset by handling the creation and execution of relevant IssueManagers.

class cleanlab.datalab.internal.issue_finder.IssueFinder(datalab, task, verbosity=1)

Bases: object

The IssueFinder class is responsible for managing the process of identifying issues in the dataset by handling the creation and execution of relevant IssueManagers. It serves as a coordinator or helper class for the Datalab class to encapsulate the specific behavior of the issue finding process.

At a high level, the IssueFinder is responsible for:

• Determining which types of issues to look for.

• Instantiating the appropriate IssueManagers using a factory.

• Running the IssueManagers’ find_issues methods.

• Collecting the results into a DataIssues instance.

Parameters:

• datalab (Datalab) – The Datalab instance associated with this IssueFinder.

• task (str) – The type of machine learning task that the dataset is used for.

• verbosity (int) – Controls the verbosity of the output during the issue finding process.

Note

This class is not intended to be used directly. Instead, use the Datalab.find_issues method which internally utilizes an IssueFinder instance.

Methods:

find_issues(*[, pred_probs, features, ...])	Checks the dataset for all sorts of common issues in real-world data (in both labels and feature values).
get_available_issue_types(**kwargs)	Returns a dictionary of issue types that can be used in Datalab.find_issues method.

find_issues(*, pred_probs=None, features=None, knn_graph=None, issue_types=None)

Checks the dataset for all sorts of common issues in real-world data (in both labels and feature values).

You can use Datalab to find issues in your data, utilizing any model you have already trained. This method only interacts with your model via its predictions or embeddings (and other functions thereof). The more of these inputs you provide, the more types of issues Datalab can detect in your dataset/labels. If you provide a subset of these inputs, Datalab will output what insights it can based on the limited information from your model.

Note

This method is not intended to be used directly. Instead, use the Datalab.find_issues method.

Note

The issues are saved in the self.datalab.data_issues.issues attribute, but are not returned.

Parameters:

• pred_probs (Optional[ndarray]) –

  Out-of-sample predicted class probabilities made by the model for every example in the dataset. To best detect label issues, provide this input obtained from the most accurate model you can produce.

  If provided for classification, this must be a 2D array with shape (num_examples, K) where K is the number of classes in the dataset. If provided for regression, this must be a 1D array with shape (num_examples,).

• features (Optional[np.ndarray]) –

  Feature embeddings (vector representations) of every example in the dataset.

  If provided, this must be a 2D array with shape (num_examples, num_features).

• knn_graph (Optional[csr_matrix]) –

  Sparse matrix representing distances between examples in the dataset in a k nearest neighbor graph.

  For details, refer to the documentation of the same argument in Datalab.find_issues

• issue_types (Optional[Dict[str, Any]]) –

  Collection specifying which types of issues to consider in audit and any non-default parameter settings to use. If unspecified, a default set of issue types and recommended parameter settings is considered.

  This is a dictionary of dictionaries, where the keys are the issue types of interest and the values are dictionaries of parameter values that control how each type of issue is detected (only for advanced users). More specifically, the values are constructor keyword arguments passed to the corresponding IssueManager, which is responsible for detecting the particular issue type.

  See also

  IssueManager

Return type:

None

get_available_issue_types(**kwargs)

Returns a dictionary of issue types that can be used in Datalab.find_issues method.