Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling
Summary: arXiv:2604.05345v1 Announce Type: new
Abstract
In today’s artificial intelligence driven world, modern systems communicate with people from diverse backgrounds and skill levels. For human-machine interaction to be meaningful, systems must be aware of context and user expertise. This study proposes an agentic AI profiler that classifies natural language responses into four levels: Novice, Basic, Advanced, and Expert. The system uses a modular layered architecture built on LLaMA v3.1 (8B), with components for text preprocessing, scoring, aggregation, and classification.
Evaluation Phases
The evaluation of the proposed system was conducted in two main phases:
- Static Phase: This phase utilized pre-recorded transcripts from 82 participants to assess the accuracy of the profiler in a controlled environment.
- Dynamic Phase: In this phase, 402 live interviews were conducted by an agentic AI interviewer, allowing for real-time assessment of expertise after each response.
Results
In both evaluation phases, participant self-ratings were compared with the predictions made by the AI profiler. The findings revealed a high level of agreement between the self-assessments and the profiler evaluations:
- In the static phase, the accuracy of the profiler’s evaluations was significant, showcasing its potential for reliable expertise assessment.
- In the dynamic phase, expertise was continuously evaluated, providing more immediate feedback on participant responses.
Key Findings
Across various domains, an impressive 83% to 97% of profiler evaluations aligned with participant self-assessments. However, some discrepancies were noted, attributed to:
- Self-Rating Bias: Participants may have over- or under-rated their expertise levels.
- Unclear Responses: Some responses lacked the clarity needed for accurate assessment.
- Misinterpretation of Nuanced Expertise: The language model occasionally struggled with interpreting subtle distinctions in expertise levels.
Conclusion
The proposed agentic AI profiler demonstrates a promising approach to enhancing human-machine interaction by accurately assessing user expertise in real-time. Its modular architecture not only improves classification accuracy but also allows for adaptability across multiple domains. As the technology continues to develop, further refinements are expected, particularly in addressing the noted discrepancies in self-assessment alignment. This system has the potential to revolutionize how AI systems interact with users, making communication more effective and contextually aware.