The purpose of this assessment is to analyze evidence of student learning in order to help answer the research question, “does giving students guidance and encouragement in using generative AI (e.g., ChatGPT, Gemini, Claude, etc.) as a complement to their established learning resources and activities in the course help them improve their learning outcomes performance in their economics principles courses (ECN211 and ECN212)”.
Many students struggle to complete the open-ended, closed book assessment quizzes in the economic principles courses at EMCC; these assessments require students to recall, apply, and analyze economic concepts and models, and some struggle with them. My hypothesis is that guiding students in the appropriate use of generative AI applications (in a more structured, thoughtful way then they may have used them before) will lead to improved learning outcomes (as measured by embedded graded assessments).
During the fall 2025 semester, I have created an AI guidance sheet (see attached) that I have implemented into my ECN211 in-person, ECN212 in-person, and ECN212 online (second eight week section only, but not my first eight week ECN211 online class, as I’ll explain later). In addition, I review key elements of that guidance with students during our study review sessions in the in-person sections of the course to show them in real time how this would work in practice.
In this assessment, I will look for evidence of changes in student learning outcomes on the standard Module 1 Assessment 2 quiz embedded in my courses, which is designed to test students’ ability to “discuss foundational economic principles”, a program learning outcome (PLO) in the Associates of Arts, with an Emphasis in Economics degree, and is assessed in both ECN211 and ECN212. This assessment is taken relatively early in the course, generally within the first couple of weeks of the semester, and is generally the first assessment some students struggle mastering.
For each comparison as outlined below, I will analyze changes in “below expectations”, “meets expectations”, and “exceeds expectations” to see if there has been improvement in student performance between the two periods.
Learning Outcomes Assessment
Fall 2024 to Fall 2025 In-person Sections Comparisons
The AI guidance instruction was provided for students in the fall 2025 in-person sections, but not in the fall 2024 sections, so it provides a kind of natural experiment.
Fall 2025 First Eight Weeks vs. Second Eight Weeks Online Comparisons
The AI guidance instruction was provided for students in the second eight week fall 2025 online section, but not in the first eight week fall 2025 section, so it provides a kind of natural experiment.
Learning outcome measured
Students will be able to discuss foundational economic principles.
Section Comparisons
Fall 2024 to Fall 2025 In-person Section Comparisons
I looked for changes in “below expectations”, “meets expectations”, and “exceeds expectations” as measured by my Module 1 Assessment 2 in each in-person section between my fall 2024 students and fall 2025 students. This guidance was not available in fall 2024 but it was in fall 2025, so it provides a kind of natural experiment.
Semester
Sample (n)
Below Expectations
Meets Expectations
Exceeds Expectations
Fall 2024
49
3 (6.1%)
17 (34.7%)
29 (59.2%)
Fall 2025
51
5 (9.8%)
14 (27.5%)
32 (62.7%)
Change
2
2 (1.7%)
-3 (-9.2%)
3 (3.5%)
Table 1: Performance on M1 Assessment 2 in In-person sections, Fall 2024 to Fall 2025
Fall 2025 First Eight Weeks vs. Second Eight Weeks Online Comparisons
I looked for changes in “below expectations”, “meets expectations”, and “exceeds expectations” as measured by my Module 1 Assessment 2 in each eight week online section (first eight week vs. second eight week). As I did not have this guidance in the first eight weeks section but I did in the second eight weeks, it provides a kind of natural experiment.
Semester
Sample (n)
Below Expectations
Meets Expectations
Exceeds Expectations
First Eight Weeks
21
6 (28.6%)
4 (19.0%)
11 (52.4%)
Second Eight Weeks
25
4 (16.0%)
8 (32.0%)
13 (52.0%)
Change
4
-2 (-12.6%)
4 (13%)
2 (-.4%)
Table 2: Performance on M1 Assessment 2 in Online sections, First Eight Weeks vs. Second Eight Weeks, Fall 2025
Data Analysis
My data results are mixed, but generally can be viewed as positive (there were generally better outcomes after the AI intervention than before it). However the effect size was relatively small and could be the result of “noise” (natural variation in student performance between sections and between semesters) than of a true “signal” of the intervention working.
For my in-person students (comparing Fall 2024 to Fall 2025), the evidence of improvement were that 3 more students “exceeded expectations” than they did before, and 3 fewer students “met expectations”, suggesting that there were some redistribution of students away from simplifying meeting expectations at the base line level to exceeding expectations (See Table 1). Likewise for my online students, more students exceeded expectations and met expectations while fewer students were below expectations, an unequivocally better result (Table 2).
The slightly disappointing news for the in-person student perspective is that more students “did not meet expectations” in the fall 2025 semester than in Fall 2024 (see Table 1), though this was partly the result that there were simply more students sampled in fall 2025 than the earlier semester.
Overall the results show a moderately positive result for the intervention sections relative to the pre-intervention cases. Results, however, should be interpreted cautiously. The results were not substantially different as to rule out ordinary variation that might take place between sections and between semesters in the absence of any particular interventions. There could also have been subtle changes in the way the classes were taught or other learning resources provided that could explain the differences as well.