Welcome > Set up classification > Classification benchmarks and optimization with the Result Matrix > Add classification benchmark exceptions

Add classification benchmark exceptions

Once a project has two or more classes, the Benchmark Exceptions icon Classification Benchmark Exceptions icon becomes available on the Classification Benchmark toolbar.

Tip To ensure that your classification benchmark results are accurate and up-to-date with your project, the best practice is that you train your project for classification before generating any classification benchmark results.

If you use content classification, you must have recognition results before starting the benchmark. When a classification benchmark using all settings is calculated, the documents in the training set or the selected Benchmark Set are classified using your current settings. When a benchmark for a specific classifier is run, all other classification settings are ignored so you see only the classifier results.

The training set or benchmark set results are compared to the Assigned Class in the document set, and marks mismatches as invalid. Sometimes, the mismatch between the classification and the Assigned Class of a document results in documents being marked invalid, even when they are correctly classified. You can add benchmark exceptions that set invalid results to valid or unclassified. The Classification Benchmark window Result Matrix will then provide more accurate statistics.

For example, several documents have an Assigned Class for a parent class that contains inherited child classes. When the benchmark is performed, the classification result returned is for one of the child classes rather than the parent class. These documents are marked invalid in the Classification Benchmark window Result Matrix, even though the classification result is valid. As a result, even if they are correctly classified as the inherited child class, they are displayed as invalid in the classification benchmark results. You can add an exception so that the invalid child result is valid, but only in the Classification Benchmark window and the Result Matrix.

Important Specific exceptions take precedence over general exceptions. For example, if you have a rule to make "All inherited classes" unclassified, and an exception to make a specific inherited class (Invoice_TypeB) valid, all documents except Invoice_TypeB document are unclassified. Documents classified as Invoice_TypeB are valid.

Also use exceptions when inheritance and nested classes are not the issue. For example, an organization has two classes with similar settings at the same level. Due to service level agreements, you do not want to count misclassified documents between the two classes as false positives. For technical and business reasons, these two classes are commonly mixed up. One class contains draft documents; the other contains final certified versions of the same documents. The only difference between the two documents is that the final certified documents have a signature and certification stamp. These items are difficult to detect, so as a result, all documents classified as either of these two classes are uncertain. To use the Result Matrix to gather statistics, add exceptions to thoroughly test benchmark results.

As each exception is added, the Result Matrix graph, the Document Evaluation results, and the Project Statistic results are automatically updated. Similarly, if a class is deleted from the project, any exceptions using that class are also removed.

You can add benchmark exceptions by following these steps:

  1. Open the Classification Benchmark window if it is not already open.
  2. Generate a benchmark for all classification settings or a specific classifier for your classification training set or a benchmark set.

    The selected document set is classified and the benchmark results are displayed in the Result Matrix.

  3. In the Classification Benchmark window toolbar, click Benchmark Exceptions Classification Benchmark Exceptions icon.

    The Classification Benchmark Exceptions window is displayed.

  4. In the Benchmark Exceptions window, click Add Exception Add icon.
  5. From the Desired Result list, select a class.
  6. From the Classification Result list, select a class.
  7. From the Exception Status list, select a value.
  8. Optionally, add a Comment.

    Although optional, the value you enter in this step is displayed as a tool-tip on the Result Matrix for any affected cells.

  9. Click OK.

    The Classification Benchmark Exceptions window closes and the benchmark results in the Result Matrix are updated automatically.