Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Using a Large Language Model
Summary: arXiv:2604.11287v1 Announce Type: new
Abstract:
Background: Large language models (LLMs) have been explored as tools for generating personalized exercise prescriptions, yet the consistency of outputs under identical conditions remains insufficiently examined.
Objective: This study evaluated the intra-model consistency of LLM-generated exercise prescriptions using a repeated generation design.
Methods: Six clinical scenarios were used to generate exercise prescriptions using Gemini 2.5 Flash (20 outputs per scenario; total n = 120). Consistency was assessed across three dimensions:
- Semantic consistency using SBERT-based cosine similarity,
- Structural consistency based on the FITT principle using an AI-as-a-judge approach, and
- Safety expression consistency, including inclusion rates and sentence-level quantification.
Results: Semantic similarity was high across scenarios (mean cosine similarity: 0.879-0.939), with greater consistency in clinically constrained cases. Frequency showed consistent patterns, whereas variability was observed in quantitative components, particularly exercise intensity. Unclassifiable intensity expressions were observed in 10-25% of resistance training outputs. Safety-related expressions were included in 100% of outputs; however, safety sentence counts varied significantly across scenarios (H=86.18, p less than 0.001), with clinical cases generating more safety expressions than healthy adult cases.
Conclusions: LLM-generated exercise prescriptions demonstrated high semantic consistency but showed variability in key quantitative components. Reliability depends substantially on prompt structure, and additional structural constraints and expert validation are needed before clinical deployment.