The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering
Summary: arXiv:2603.25197v2 Announce Type: replace
Abstract: As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis.
We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing:
- Domain Knowledge
- Standards Expertise
- Operational Experience
- Contextual Understanding
- Judgment
We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. This concept is pivotal in understanding the limitations of AI in safety engineering.
To provide a deeper understanding of the implications of AI assistance, we formalize four canonical human-AI collaboration structures and derive closed-form performance bounds. Our research demonstrates that the competence shadow can compound multiplicatively, leading to a degradation of safety analysis quality that far exceeds naive additive estimates. This finding indicates that the integration of AI into safety workflows is not merely a matter of selecting the right software but involves a complex interplay of design considerations.
The central finding of our research is that AI assistance in safety engineering is fundamentally a collaboration design problem, rather than a software procurement decision. The same AI tool can either degrade or enhance analysis quality, heavily influenced by how it is utilized within the workflow. Consequently, we derive non-degradation conditions for shadow-resistant workflows, suggesting that a shift in focus is needed—from tool qualification to workflow qualification for ensuring trustworthy Physical AI applications.
In conclusion, our investigation into the competence shadow offers significant insights into the role of AI in safety engineering. As industries increasingly rely on AI for critical safety analyses, understanding the nuances of human-AI collaboration becomes essential. By recognizing the potential blind spots introduced by AI and addressing them through thoughtful workflow design, practitioners can enhance the safety and reliability of Physical AI systems.