From Domain Understanding to Design Readiness: A Playbook for GenAI-Supported Learning in Software Engineering
Summary: arXiv:2604.00120v1 Announce Type: cross
Abstract: Software engineering courses often require rapid upskilling in supporting knowledge areas such as domain understanding and modeling methods. We report an experience from a two-week milestone in a master’s course where 29 students used a customized ChatGPT (GPT-3.5) tutor grounded in a curated course knowledge base to learn cryptocurrency-finance basics and Domain-Driven Design (DDD).
In this innovative educational approach, we logged all interactions between the students and the AI tutor, focusing on a comprehensive evaluation of learning outcomes. A random sample of 34.5% of prompt-answer pairs (60 out of 174) was assessed using a five-dimension rubric, which included:
- Accuracy
- Relevance
- Pedagogical Value
- Cognitive Load
- Supportiveness
Additionally, we collected pre- and post-experience self-efficacy data to understand the impact of the AI tool on students’ confidence and understanding of the subject matter.
Key Findings
The analysis of the logged interactions revealed promising results:
- Accuracy: The overall accuracy of the AI responses averaged 98.9%, with no factual errors reported and only two minor inaccuracies found in the sample.
- Relevance: Responses maintained a high relevance score of 92.2%, indicating that the AI was effectively addressing the students’ queries.
- Pedagogical Value: The pedagogical value of the interactions was rated at 89.4%, reflecting the educational effectiveness of the AI tutor.
- Cognitive Load: Responses showed an appropriate cognitive load level of 82.78%, suggesting that the complexity of the information was manageable for students.
- Supportiveness: However, supportiveness was rated relatively low at 37.78%, indicating there is room for improvement in this area.
Self-Efficacy Gains
Students reported significant pre- to post-experience self-efficacy gains related to their domain learning and application of DDD principles. This highlights the potential of GenAI-assisted learning tools in enhancing learner confidence and competence in complex subjects.
Teaching Practices
Based on our observations, we distilled seventeen concrete teaching practices that can enhance the effectiveness of GenAI in educational contexts. These practices encompass:
- Setting expected granularity in responses
- Constraining verbosity to maintain clarity
- Curating guardrail examples for better understanding
- Adding small credit incentives aligned with a simple quality rubric
Conclusion
In conclusion, our findings suggest that GenAI-supported learning can effectively complement traditional instruction in domain understanding and modeling tasks within software engineering education. While the results are encouraging, there remains ample opportunity to enhance the tone and follow-up structure of AI responses, ensuring a more supportive learning environment for students.