How Are Thresholds Set in AI Visual Inspection? A Free Web App That Shows the Trade-Off of "Zero Misses" in Numbers
kotai2003Choose "zero misses," and Recall (detection rate) reaches 100%. At the same time, false detections rise to 53 and Precision drops to 65%. TOMOMI RESEARCH, INC. has released a free educational demo app that lets you explore, by moving sliders in your browser, how the anomaly score, the threshold, and evaluation metrics relate to one another in AI visual inspection. This article shows what you can see with the app, along with actual figures.
URL: https://anomalydetectiondemo.streamlit.app/
(Free, no installation, no login, no account registration)
The AI outputs a score, not "OK / NG"
AI visual inspection does not answer "OK" or "NG" from the start. In practice, it takes two steps.
- For each product, it turns "how different this is from the good parts it learned" into a number. This is the anomaly score (0.0 to 1.0). The more a product looks like a good part, the smaller the score; the more it looks like a defect, the larger.
- Anything whose anomaly score exceeds a certain boundary value is judged NG. This boundary is the threshold.
The rule is simple: if anomaly score ≥ threshold, the item is judged NG. In other words, all the AI outputs is a continuous score. It is people who draw the line that says "beyond here is NG."
This structure has a consequence. The question "What is the accuracy?" only has a definite answer once you decide which threshold you are talking about. Accuracy is not a fixed value determined by the detector alone; it only has meaning when paired with a threshold.
The initial view of Tab 1. On the left, a histogram of the anomaly score distribution; on the right, a scatter plot with one dot per sample. The dotted line in the center is the threshold.
Why does the AI learn "only good parts"?
At the top of the app screen, we display a line from Leo Tolstoy's Anna Karenina.
Happy families are all alike; every unhappy family is unhappy in its own way.
Good parts all look alike, but every defect is different. Scratches, chips, stains, deformation, foreign matter: it is not realistic to enumerate every type of defect and train on all of them. It is more straightforward to learn only "what a good part looks like" and measure the distance from it. That is the idea behind anomaly detection trained on good parts. In this app, we call this framework "the Anna Karenina principle in anomaly detection" (it is a framework for understanding, not an academically established theorem).
"If it learns from good parts, do you not need NG images?" A question the terminology invites
The term "trained on good parts" can sound as if NG images are not needed at all. Precisely, it means NG images are not used when building the model. It does not mean NG samples are unnecessary for performance testing. NG samples are needed both to decide where to place the threshold and to measure how many defects are missed at that threshold.
What happens when you move the slider
Now to the main topic. The app's default setting uses well-separated data with AUC ≈ 0.97 (the closer to 1, the better the detector). In this state, let's move only the threshold.
With the threshold lowered to 0.30. The NG region (gray band) widens greatly and sweeps in good parts as well.
| Threshold | False detections | Missed defects | What happens on the shop floor |
|---|---|---|---|
| 0.30 (low) | 67 | 0 | No defects escape, but too many good parts are stopped |
| 0.44 (middle) | 11 | 12 | Balanced |
| 0.60 (high) | 0 | 62 | Good parts are not stopped, but defects are missed |
* Values are for the demo's default setting (simulated data generated within the app).
A false detection is the error of "judging a good part NG"; a missed defect is the error of "judging a defect OK." Reading the table from top to bottom, you can see that when one decreases, the other increases. As long as the score distributions of good parts and defects overlap even slightly, no threshold can bring both false detections and missed defects to zero.
With the threshold raised to 0.60. The NG region shrinks: good parts are no longer stopped, but a large number of defects slip through.
"Zero misses" comes with a bill
A common quality requirement is "we want zero missed defects." The app has a threshold setting mode called "Zero misses first," so let's actually select it.
- 0 missed defects, Recall (detection rate) 100%: the requirement is met
- However, false detections increase to 53
- Precision drops to 65%
* These are also values on simulated data.
Precision of 65% means that, of the items judged "NG," 65% were actually defects. The rest are good parts that were stopped, and every one of them requires re-inspection work.
The option of zero misses certainly exists. What does not exist is zero misses without a trade-off. Safety is never free.
Tab 3 with "Zero misses first" selected. The red dot on the curve (the threshold in use) moves; Recall reaches 100% while Precision drops.
That is why the threshold is set by quality policy
Putting it all together, the threshold turns out to be not a mathematically correct answer but an operating condition. The reason is that the losses from the two errors are not symmetric.
- Missed defects: defective products escape to downstream processes or the market, leading to recalls and loss of trust
- False detections: good parts are stopped, causing re-inspection work and lower yield. They also strain the throughput of the inspection line
Which one you want to avoid more depends on the product's application, the configuration of downstream processes, production volume, and your quality policy. Even with the same detector, the right threshold differs from site to site.
In the app, you can choose how to set the threshold from five modes and compare the results.
| Method | Approach |
|---|---|
| Manual | Use the slider value as is |
| F1 max | The point with the best balance between Recall and Precision |
| Zero misses first | Top priority on not missing a single defect |
| Cap on false detection rate | Set a cap such as "stop no more than X% of good parts," and find the best point within it |
| Minimize shop-floor cost | Enter the loss weights for one missed defect and one false detection, and find the point with the smallest loss |
We would especially like to draw your attention to "Minimize shop-floor cost." Entering the weights for one missed defect and one false detection is, put the other way around, nothing less than stating in numbers how your company weighs the two errors against each other. Being able to decide these numbers internally is the real preparation for introducing AI inspection.
Caution when reading a scorecard as a single number
Tab 2 shows the results for the same threshold as a confusion matrix (four categories: TN / FP / FN / TP) and five evaluation metrics: Recall (detection rate), Specificity, Precision, Accuracy, and F1 Score. When you move the threshold, the two that trade off directly are FP (false detections) and FN (missed defects). Once you understand the four categories together with TN and TP, you can see that every evaluation metric is defined by a combination of these four.
The confusion matrix in Tab 2. Each of the four categories includes an explanation in Japanese.
The widely used F1 Score is the harmonic mean of Recall and Precision. It is high only when both are high; if one is low, it is pulled strongly toward the lower value rather than the average. For example, with Precision 1.0 and Recall 0.1, the ordinary average is 0.55, but F1 is only about 0.18. That is why F1 stays low both for "zero misses but full of false detections" and for "zero false detections but full of misses." However, the point where F1 is highest is not necessarily the best for your company, as the previous section explained.
In this app, metrics that cannot be calculated because the denominator is 0 are shown as "—" rather than "0." This design prevents confusing "a score of 0" with "cannot be calculated."
What this app is: learning material for building knowledge
The trade-off between false detections and missed defects, the meaning of Precision and Recall, the fact that F1 is a harmonic mean, how to read ROC and PR curves: these are prerequisite knowledge you should understand if you are putting AI visual inspection into mass production, and they are statistics and machine learning knowledge in their own right. We built this app not as a tool that makes this knowledge unnecessary, but as learning material for acquiring it.
Thresholds and evaluation metrics are hard to work with because they are dynamic concepts, where results change together as values change, yet they are explained with static text and formulas. They are learned faster by moving the values and watching the results change than by reading. The operation is simple, but the content is real-world practice itself. The "📖 How to use" tab in the app embeds an operating manual, which can also be opened in a separate window, so you can refer to it on a dual-monitor setup while you operate the app.
Few Japanese-language learning materials teach the evaluation metrics of AI visual inspection in this form. We have made the app free and registration-free so that it can serve as shared learning material for in-house training and for engineers' self-study.
Upcoming plans
We plan to add the following features.
- The small-sample problem: visualize, with repeated sampling and box plots, how much evaluation values and thresholds fluctuate with 10 versus 100 good-part images
- Separating the training, threshold tuning and independent evaluation phases: visualize why the same images must not be reused
- Mass-production simulation: estimate daily false detections, re-inspection work, and throughput overruns from production volume and defect rate
- CSV data import: run Tabs 1 to 3 with your own anomaly score CSV
Summary
The AI only outputs a score called the anomaly score. What draws the OK/NG line is the threshold, an operating condition decided by people. Move the threshold, and false detections and missed defects swap like a seesaw; there is no point at which both are zero.
That is why deciding the threshold is, more than a technical optimization, a quality-policy decision about which failure you want to avoid more. We hope you will use this app as a common foundation for discussing that decision with the same terms and the same metrics.
It takes 5 minutes. Just open it in your browser and move the slider left and right.
▶ AI Anomaly Detection: Thresholds and Evaluation Metrics
— https://anomalydetectiondemo.streamlit.app/
Free, no installation, no login, no account registration. Works on PCs and tablets.
Notes
- This app is a free educational simulator and is separate from our products and services.
- All data shown is simulated data generated within the app. It contains no data from real projects and no customer data. All counts and ratios in this article are values on simulated data at the demo's default settings and do not represent the performance of any specific product or inspection system.
- Features listed in the roadmap are planned additions, and their release dates have not been determined.
