AI Teaching Assistants

Back to All Actions

What is the Action?

Use AI assisted learning tools (chatbots) to provide supplementary help as programming tutors in computing modules.

Quick Facts to Support this Action

Female programming students rate AI tools higher in effectiveness compared to male students.

Example 1 – Chatbot for Python Practice

Study How Can Chatbots Improve the Gender Gap in Information Systems?

Development and features of AI tutor:

A chatbot was built in Python, using a JavaScript framework, Bolt and AWS Lambda designed for a Slack workspace. MySQL was used for the database. The chatbot was made available to students in Slack for the duration of the module.

Students could do four things with the chatbot:

➤ Drill

allowed students to select a category they wanted to learn more about and were then led through questions with instant feedback provided.

➤ Exam Review

allowed students to practice for their exams by giving them a choice of specific concepts and number of questions.

➤ The Reference Guide Function

for participants to browse through selected Python concepts and their descriptions.

➤ The Project’s Function

provided students with projects to practice with, e.g., to create a password generator or an online portfolio, and allowed doing that with different levels of difficulty.

Recruitment of students:

200 students in an introductory computer programming course were recruited and offered extra credit for taking part in the study.

Evaluation Approach:

At the beginning and at the end of the semester participants did a survey that assessed self-efficacy (using scale developed by Payne et al., 2018, and at the end it also assessed self-consciousness and social anxiety (using scale by Fenigstein et al., 1975), as well as perceived level of chatbot usage (as opposed to factual recorded usage) on a 5-point Likert scale. Chatbot usage was also recorded through the logs, calculating the number of messages and the type of function used the most.

Findings:

The more female participants used the chatbot, the more their self-efficacy increased, which was the opposite to the male participants, whose self-efficacy decreased with the more frequent use of the chatbot. Overall men increased in their self-efficacy levels more than women over the course of the module, irrespective of the chatbot use. The researchers concluded that as the use of chatbots benefits women specifically and can be a promising tool to address the gender confidence gap in computing.

Tutor for Coding Concepts

Study: A Chatbot tutor can lessen the gender confidence gap in information systems learning – by Leavitt et al. 2022

Chatbot for the Slack platform

Slack was the primary communication medium in the programming course, so the chatbot was developed to be integrated with it. It was built using Google’s DialogFlow, MySQL for the database. It was called Skylar to sound and look gender neutral. The chatbot’s aim was to help students learn coding principles, and to help them practice with theory and hands-on tools in their preferred coding environment. a There were additional resources such as YouTube videos and W3Schools lessons provided within the database.

Similarly to the functionality from Example one, the chatbot had several functions:

➤ Learn

providing students with theory of queried coding concepts, helpful resources, and help them test their knowledge with a few questions.

➤ Drill

allowed students to choose one of 17 topics and related questions for practice, as well as receive feedback.

➤ Review

helped them prepare for their assessments.

➤ Practice Program

offered a list of mini-projects students could complete on various levels, in preferred coding environment.

➤ Small Talk Ability

designed to improve the experience of users and make it more fun, the chatbot was for instance able to tell jokes.

Recruitment of participants:

All students in an introductory computing classes (in this case approx.. 150 students) were given the pre-test survey and an opportunity to use the chatbot over 7 weeks during their term.

 

Evaluation Approach:

Students participated in a pre- and post-test survey. The survey assessed the creative self-efficacy (CreaSE), as well as student programming knowledge (an example of a survey is available here (link) in the resources section of TechMate). Demographics were measured in the post-test. The post-test included a self-reported measure of usage of the chatbot.

 

Findings:

The study confirmed some of the previous research that women tend to be less confident in computing classes, despite their equal performance levels in them. New fundings suggest that chatbots can boost the overall confidence in the topic, and the increase in confidence levels is stronger for women than for men.

Berthelsen, R., Chandler, C., Dean, L., Keith, M., Schuetzler, R., Kettles, D. and Anderson, G., 2024. How Can Chatbots Improve the Gender Gap in Information Systems?

Fenigstein, A., Scheier, M. F., & Buss, A. H. (1975). Public and private self-consciousness: Assessment and theory. Journal of Consulting and Clinical Psychology, 43(4), 522–527. https://doi.org/10.1037/h0076760.

Leavitt, C., Corbin, M., Keith, M., Schuetzler, R., Anderson, G. and Kettles, D., 2022. A Chatbot Tutor Can Lessen the Gender Confidence Gap in Information Systems Learning.

Maurat, J.I.C., Isip, E.V. and Lumabas, A.G.R., 2024, October. A Comparative Study of Gender Differences in the Utilization and Effectiveness of AI-Assisted Learning Tools in Programming Among University Students. In Proceedings of the 2024 International Conference on Artificial Intelligence and Teacher Education (pp. 30-34).

Payne, K.C., J Keith, M., Babb, J. and N Spruill, A., 2018. Development and Validation of the Information Systems Creative-Self-Efficacy Scale.

KEEP ACCORDIAN COLLAPSED

Do not delete. This is important to keep the accordian items closed on initial load. Keep this at the top of the accordian stack.

The accordian will auto-open the first item on load. This item has custom css to keep it hidden. This means the user cannot see that it is open thus manually overriding the auto-open.

AI Chatbot Study - Student Survey Questions

See full list of survey questions from the study A Chatbot tutor can lessen the gender confidence gap in information systems learning – by Leavitt et al. 2022, using a 7-point Likert scale.

A sample extract (questions coded by the auhors) can include the following:

AFF1: I believe that I would have the determination necessary to creatively solve IS problems.

AFF2: I believe that creative IS problem solving would be very interesting to me.

AFF3: I believe that I would be enthusiastic about creative IS problem solving.

BUS1: I believe that I have the domain knowledge necessary to identify the root cause of an IS problem.

BUS2: I believe that I have the expertise necessary to solve complex IS problems.

BUS3: I believe that I have enough knowledge about business processes to create better solutions.

INT1: I believe that I have the ability to understand an IS problem from multiple angles.

INT2: I believe that I have the ability to think unconventionally to find IS solutions.

Competence survey questions 

C1 Which of the following are characteristics of a strongly typed programming language? Mark all that apply.

– In strongly typed languages, data types must be declared for each variable defined.

– Variables are declared without defining which data type the variable is in strongly typed languages.

– When using a strongly typed language, only the keywords const and let can be used to create variables.

– Errors are less likely to occur during compile time when using a strongly typed language as opposed to a loosely typed  language.

C2 What is the difference between a function and a method in programming?

C3 You’re given a list of textbooks with their respective prices and need to add up the total cost of all the textbooks. Which of the following would be the BEST statement to use?

  • If/then statement
  • Switch statement
  • For loop
  • While loop