LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspection
Summary: arXiv:2604.05371v1 Announce Type: new
Abstract: The deployment of lightweight segmentation models on drones for autonomous power line inspection presents a critical challenge: maintaining reliable performance under real-world conditions that differ from training data. Although compact architectures such as U-Net enable real-time onboard inference, their segmentation outputs can degrade unpredictably in adverse environments, raising safety concerns.
In this work, we study the feasibility of using a large language model (LLM) as a semantic judge to assess the reliability of power line segmentation results produced by drone-mounted models. Rather than introducing a new inspection system, we formalize a watchdog scenario in which an offboard LLM evaluates segmentation overlays and examine whether such a judge can be trusted to behave consistently and perceptually coherently.
Methodology
To address these challenges, we design two evaluation protocols that analyze the judge’s repeatability and sensitivity:
- Repeatability Assessment: We assess repeatability by repeatedly querying the LLM with identical inputs and fixed prompts, measuring the stability of its quality scores and confidence estimates.
- Perceptual Sensitivity Evaluation: We evaluate perceptual sensitivity by introducing controlled visual corruptions (fog, rain, snow, shadow, and sunflare) and analyzing how the judge’s outputs respond to progressive degradation in segmentation quality.
Results
Our results show that the LLM produces highly consistent categorical judgments under identical conditions while exhibiting appropriate declines in confidence as visual reliability deteriorates. The findings indicate that:
- The judge remains responsive to perceptual cues such as missing or misidentified power lines, even under challenging conditions.
- The LLM’s ability to maintain consistency in outputs under identical prompts reinforces its potential as a reliable semantic judge.
- It successfully adapts its confidence levels in accordance with the quality of segmentation presented, highlighting its utility in safety-critical applications.
Conclusion
These findings suggest that, when carefully constrained, an LLM can serve as a reliable semantic judge for monitoring segmentation quality in safety-critical aerial inspection tasks. This research paves the way for enhanced safety protocols and more dependable aerial inspection processes, ultimately contributing to the efficiency of infrastructure maintenance and monitoring.