What Happens to an Inspection AI When NG Images Slip into Training Data Meant to Be Good-Only?

When you build a visual inspection AI, most methods rest on one assumption: the training data is all good parts. On a real production floor, however, labelling mistakes and missed inspections mean that a handful of images of defective parts can slip into the training folder. When that assumption breaks, how badly does the model break? SoftPatch (NeurIPS 2022) is a paper that examines both how it breaks and how to prevent it.

One extra step that closes a weakness in PatchCore

The foundation is PatchCore, a widely used method. It divides training images into small regions (patches), stores their features in a dictionary called a "memory bank", and at inspection time uses the distance between each patch of the test image and its nearest feature in the dictionary as the anomaly score. If no similar feature exists in the dictionary, the patch is judged "never seen before, therefore anomalous".

The problem arises when defective patches get into this dictionary. If a similar defect then appears at inspection time, it matches the dictionary and is wrongly judged "seen before, therefore normal". Worse, PatchCore builds its dictionary by selecting representative points that cover the feature space evenly, so it tends to pick up unusual features, that is, outlying defective patches, preferentially.

What SoftPatch adds is one extra step: removing suspicious patches, patch by patch, before the dictionary is built. Because the judgement is made per patch rather than discarding whole images, the normal parts of a defective image are still put to use. The judgement uses LOF (Local Outlier Factor), a method that measures how much a patch "stands out" from the density ratio to its surrounding patches. In addition, to guard against confusing defects that removal did not catch, each retained feature is given a weight according to its degree of outlierness (a soft weight), which is multiplied into the distance calculation at inspection time.

As Figure 1 shows, SoftPatch differs from PatchCore in two respects: removal before the dictionary is built, and the weights attached to the dictionary.

flowchart LR
  subgraph PC["PatchCore (conventional)"]
    direction TB
    A1["Patch features of training images<br/>(including defective patches that slipped in)"] --> A2["Select representative points to build the dictionary"]
    A2 --> A3["Dictionary (memory bank)<br/>defective patches may remain"]
    A3 --> A4["At inspection: distance to nearest feature = anomaly score"]
  end
  subgraph SP["SoftPatch"]
    direction TB
    B1["Patch features of training images<br/>(including defective patches that slipped in)"] --> B2["Measure outlierness with LOF at each position<br/>and remove highly outlying patches"]
    B2 --> B3["Select representative points from the rest to build the dictionary"]
    B3 --> B4["Dictionary + a weight per feature based on outlierness<br/>(soft weight)"]
    B4 --> B5["At inspection: weight of nearest feature × distance = anomaly score"]
  end

  PC -->|Improved| SP

Figure 1: Comparison of the processing (prepared from the description in the paper)

The paper's own overview diagram is shown in Figure 2.

Overview of SoftPatch. During training, likely defective patches are identified at each position of the feature map using LOF or similar, the most outlying are removed, representative points are then selected, and the dictionary and soft weights are stored in the memory bank. At inspection, the nearest-neighbour distance is weighted by the soft weight to output an anomaly score and an anomaly map

Figure 2 Source: Jiang et al., "SoftPatch: Unsupervised Anomaly Detection with Noisy Data", NeurIPS 2022, Figure 2. arXiv:2403.14233 (arXiv non-exclusive license. Reproduced unaltered as a quotation. Product images in the figure are from MVTec AD)

The basis for "never let NG images in", and its conditions

What this paper exposes is that the impact of NG contamination changes greatly with the evaluation conditions. It mixes defective images into the training data and compares a condition in which those same defective images also remain in the test set (Overlap, the worst case) with one in which they are excluded from the test set (No overlap, the milder case).

Under Overlap with a 10% contamination rate, PatchCore's image AUROC (1.0 is perfect, 0.5 is a random guess) falls from 0.992 to 0.683. SoftPatch, by contrast, is almost unaffected and holds 0.982. When the contamination rate is raised to 15%, the authors report that PatchCore's decline under Overlap reaches up to 40%. The main figures are summarised in Table 1.

Table 1: From Tables 1 and 2 of the original paper

Overlap, however, is an extreme condition. Under No overlap, PatchCore's decline stays small even as the contamination rate rises. The reality is this: "A few NG images slipping in does not cause immediate collapse. But when the same defective images appear in both training and test data, the NG that has entered the dictionary bites back." This is the basis for "do not put NG images into the dictionary as they are".

In the ablation study, most of the improvement comes from the LOF removal itself; the additional effect of the soft weight goes up or down depending on the metric, and the authors themselves describe it as "minor". Also, under the ideal condition with no NG at all in the training data, SoftPatch falls slightly behind PatchCore in image AUROC. This appears to be the price of setting the removal threshold somewhat conservatively.

Practical implications and limitations

Most real production floors are likely to sit somewhere between the worst-case Overlap and No overlap. The threshold and the number of LOF neighbours are fixed values, and the authors acknowledge that the optimal values may change with the target and the contamination rate. The contamination rates tested also go only up to 15%; situations in which NG images approach a majority are outside its scope.

Rather than relying on visual checks to remove every NG image from the training data, building automatic patch-level removal into the inspection pipeline is a more realistic safeguard. Processing time is also almost the same as PatchCore, so the barrier to adoption is not high. Moreover, on the real-world dataset BTAD, SoftPatch outperformed PatchCore even in categories where the training images already contained scratches, confirming an effect that is not limited to synthetic experiments. It is worth being clear, however, that SoftPatch is not a method that actively makes use of NG images; it is strictly a method for removing them.

Source

Jiang, Liu, Wang, Nie, Wu, Liu, Wang, Zheng. "SoftPatch: Unsupervised Anomaly Detection with Noisy Data." NeurIPS 2022. arXiv:2403.14233