With the rapid development of GPT-based models, educational chatbots are no longer limited to scripted dialogues. They can now support open-ended interaction and inquiry-based learning. The researchers directly compared:

• Lesson-plan-driven chatbots — systems that guide students step-by-step through a predefined curriculum.

• Ask-Me-Anything (AMA) chatbots — flexible tools where students lead the conversation by asking their own questions.

The experiment was conducted within an undergraduate UNIX Shell scripting course, a core subject in computer science education.

The study involved 55 undergraduate students who used several chatbot versions to prepare for quizzes and exams. After each interaction, participants completed surveys evaluating usability, learning experience, and overall satisfaction.

As described in the system architecture diagram on page 3, the OS315 chatbot used a modular setup: instructors defined lesson plans that were integrated into an AI assistant through Flowise and the OpenAI API, ensuring structured and controlled learning interactions.

Results indicate that students generally preferred the structured learning approach.

• The lesson-plan-driven chatbot achieved Net Promoter Scores (NPS) as high as +72.

• The AMA chatbot achieved approximately +35.

Although both tools received positive feedback, students reported that structured chatbots were more effective for exam preparation because they actively guided the learning process and asked targeted questions.

Interestingly, most participants favored a hybrid model combining human instruction with chatbot support rather than replacing instructors entirely.

Student Feedback

Participants highlighted several benefits:

• Faster identification of syntax mistakes in UNIX commands

• Interactive quiz-style learning

• Immediate explanations and feedback

However, the most common criticism concerned response speed and interface design, suggesting that technical improvements could further enhance learning outcomes.s

When the same chatbot architecture was applied to a Data Visualization course, student satisfaction dropped significantly. This suggests that AI-driven structured chatbots may be particularly effective in technical subjects with clearly defined rules, such as programming.

The authors emphasize that, despite high engagement levels, there is still limited evidence that chatbots directly improve measurable learning outcomes, an important area for future research.

The findings highlight the growing role of AI assistants in higher education. As technology costs continue to decline, chatbot-based learning tools could become scalable and cost-effective solutions for supporting programming education and assisting instructors.