RobustMAD: How Far Can We Trust the "Small AI" We Would Hand Factory Inspection To?

"Is there a defect in this image?" Ask that, and the AI correctly finds the scratch on the cable. But simply rephrase it as "Please confirm that there is no defect," and it answers "No problem" for the very same image. In inspection, this kind of reversal leads straight to a missed defect, in other words, an incident. How well can a small inspection AI that can be installed in a factory withstand this kind of pressure? RobustMAD is the first work to measure this systematically.

Why pack an inspection AI that "answers in words" into 4B parameters?

Industrial inspection is shifting from AI that looks at an image and returns only an "anomaly score" to AI that can explain in words and hold a dialogue. Its appeal is that it can tell you "what this part is, where and how it is damaged, and why that is a problem."

But high-performing AI of this kind is large and runs only in the cloud. Relying on the cloud in a factory brings three costs that the shop floor cannot accept. (1) A communication delay of 1 to 2 seconds, which is fatal for a line carrying 200 parts per minute. (2) The risk of information leakage from sending inspection images outside the company. (3) Pay-per-use billing every time it is used.

That is why expectations are high for small models (MSLMs) of 4 billion parameters (4B) or fewer that run on a terminal beside the line. Their weights are a few GB, fit on an inspection PC, and answer immediately without a single byte of data leaving the factory. Until now, however, there has been no yardstick for measuring their "robustness on the shop floor."

A surprisingly strong showing, but the trick behind the "88% average"

Of the six 4B-class small AIs evaluated by RobustMAD, the top model, Qwen3-VL-4B, reached 88.31% accuracy on multiple-choice questions. It outperformed GPT-5 Nano, used for comparison, and the larger Phi-4 (6B).

How can it win despite being small? Because another model built on the same language model performs much worse, the authors attribute the gap to "the design of the vision side": a mechanism that feeds multiple visual layers into the language model to tighten the correspondence between image and text (DeepStack), and distillation, in which the model learns by imitating a strong teacher AI. The difference lies not in size but in "how it is trained." That said, the opponents are a 6B model and efficiency-focused commercial small models, and none of the MSLMs reach Gemini 3 Flash, the top model among those used for comparison. This is not a giant-slaying story.

There is a trick behind the 88%. That figure is an average pushed up by easy questions such as "naming the object" (97.6%). On the task that matters most for safety, "finding the defect," even the best model, Qwen3-VL-4B, drops to about 77%.

graph TD
    ALL["Multiple choice overall: accuracy 88.31%"]
    A["Relatively easy task: object naming 97.6%"]
    B["Safety-critical task: defect detection approx. 77%"]

    ALL --> A
    ALL --> B

    style ALL fill:#dbeafe,stroke:#2563eb,stroke-width:2px
    style A fill:#dcfce7,stroke:#16a34a,stroke-width:1px
    style B fill:#fee2e2,stroke:#dc2626,stroke-width:2px

The 88% average is pushed up by easy tasks; the defect detection that really matters is about 77% (Qwen3-VL-4B, multiple choice).

Three pitfalls

RobustMAD exposes three failures that standard tests miss, through the way its questions are constructed.

First, fragile visual understanding. On specialized parts such as cables, transistors, and circuit boards, defect detection from a single image drops sharply. Even Qwen3-VL-4B falls from around 80% on general objects to close to 60%. It can fluently state names and uses, yet overlooks subtle scratches on the actual item and confidently declares "no problem" without noticing. It is like a specialist shop clerk who can recite the catalog but cannot spot a defect in the real product. When image quality degrades through blur or darkness, its already weak detection is eroded further.

Second, shallow answers. Remove the answer choices and require free-form responses, and guessing no longer works, so the shallowness of understanding shows. The share of answers that cleared the passing line (scoring was done by a separate AI, which agreed with human evaluation more than 90% of the time) was 72.8% even for the best model, meaning about a quarter failed. Style scores close to full marks, but accuracy and completeness of content are low.

Quoted from Figure 4 of the original paper. Pass rate for free-form answers (share with score of 3 or higher).

Third, hallucination on unanswerable questions or questions with broken premises. Ask about an attribute that does not exist (such as the voltage rating of a part that should not be there), and instead of saying "I don't know," the model confidently fabricates an answer. The "judgment flips when rephrased" example at the start is also of this kind. It answers a neutral question correctly but is pulled along by phrasing that contains the answer, much like a witness who goes along with a leading question. This is not intent, but a "habit of matching the phrasing" ingrained through training.

Quoted from Figure 1 of the original paper (responses in English). To the neutral question it says "defect present"; when prompted to confirm, it hallucinates "no defect."

What it takes to bring this into the factory

The promise is real, but it is orders of magnitude away from safety standards. Automotive safety parts require "a few per million," effectively zero. 88% is equivalent to "missing 12 out of every 100," a gap of 3 to 4 orders of magnitude. For now, then, the realistic use is as an assistant, not as a replacement for people.

The authors stress that these failures "cannot be fixed by prompt engineering." Workers on the floor will not rephrase their questions optimally, and a safety system cannot be allowed to depend on clever phrasing. The key lies not in prompts but in improvements to training and architecture. Put the other way, Qwen3-VL-4B's strong showing also suggests that this robustness is not an unreachable goal for a 4B model.

Source

• RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants
• Anushiya Arunan, Xin Li, Yan Qin et al. (2026), Transactions on Machine Learning Research (TMLR)
• https://openreview.net/forum?id=skrA9UYNIZ
• Code: https://github.com/en-research/RobustMAD / Project: https://robustmad.github.io/
• Figures quoted (Figures 1 and 4): Anushiya Arunan et al., "RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants", TMLR 2026 (OpenReview id=skrA9UYNIZ). Licensed under CC BY 4.0.
• Cable image in Figure 1: MVTec AD (Bergmann et al., 2019), CC BY-NC-SA 4.0 (non-commercial).

Papers to read next

• MMAD: A Comprehensive Benchmark for MLLMs in Industrial Anomaly Detection (Jiang et al., 2025): the immediately preceding benchmark that this paper builds beyond. RobustMAD fills its gap of being centered on multiple choice and unable to measure robustness.
  https://arxiv.org/abs/2410.09453
• DeepStack (Meng et al., 2024, NeurIPS 2024): the core technique behind "why Qwen3-VL-4B is strong." A mechanism that fuses visual tokens into the language model.
• Mitigating Hallucination via Robust Instruction Tuning (Liu et al., 2024, ICLR 2024): a remedy for the third failure (hallucination). Suppresses fabrication with explicit negative data.