The goal is to explore whether Video Lessons (VLs) or Journal Entries (JEs) have a greater impact on student withdrawal rates and exam scores.
MAT151 is a building block for students on the STEM path and having a stronger foundation should lead to better outcomes in higher level courses.
VLs, but no JEs, were completed by three sections of MAT151 in Fall 2019. JEs, but no VLs, are being completed by three sections of MAT151 in Spring 2020. There are similarities and differences across semesters, so a comparison of withdrawal rates and exam scores will be done on an exploratory basis. The objective is to determine any changes in withdrawal rates and exam scores that could potentially be due to VLs or JEs.
Withdrawal rates will be explored using Kaplan Meier Curves and Log Rank Tests. Exam scores will be explored using Histograms and Wilcoxon Rank Sum Tests. The results are forthingcoming, as the current semester is in progress. This space will be updated.
The purpose of this CATS is to explore student withdrawal rates and exam scores from six MAT151 courses across two semesters to determine whether Video Lessons or Journal Entries appear to lead to greater student course retention and improved learning.
See comments for update to close the loop.
Comments
VLs vs. JEs: The results from the exploratory anaylsis are in!
Background: This analysis compares performance metrics through Exam 1 between Video Lessons (VLs) used in three classes in Semester 4196 and Journal Entries (JEs) used in three classes in Semester 4202. Exam 1 scores are presented graphically using histograms. Continuation rates through the Exam 1 date are displayed graphically using Kaplan-Meier survival curves. Comparison of continuation rates between the two assessment techniques is performed using a Fisher's exact test for overall continuation rates and a log-rank test for the continuation curves. Comparison of Exam 1 scores between the two assessment techniques is performed using a Wilcoxon rank sum test and a t-test.
Results: Exam 1 scores were obtained for 87 out of 90 students using VLs, with 3 students withdrawing from the course prior to Exam 1. Using JEs, Exam 1 scores were obtained for 72 out of 79 students, with 7 students withdrawing prior to taking Exam 1. Exam 1 was taken on the 40th day of the semester in all 6 classes. The overall continuation rate on the Exam 1 date was 97% when using VLs versus 91% using JEs. The difference in the overall continuation rate is not statistically significant at the 0.05 significance level (p=0.1917) using Fisher's exact test. The continuation curves are also not statistically significantly different using the log-rank test (p=0.1273). The mean (SD) of Exam 1 scores using VLs is 63.1 (19.5) compared to 58.2 (21.3) using JEs. While Exam 1 scores are 4.9 points higher on average using VLs than JEs, the difference is not statistically significant at the 0.05 significance level using either the Wilcoxon rank sum test (p=0.1595) or t-test (p=0.1324).
Summary: In summary, while there were no statistically significant differences between performance metrics using VLs and using JEs, students using VLs were continuing in the course at a higher rate than students using JEs and demonstrated a nearly 5-point improvement in Exam 1 scores compared to JEs.
Graphs: The summary graphs can be found here.
Thanks for taking time to submit this CATS. I have found that students are not fond of writing a journal because it requires writing, and they are in a "math" class. So, I definitely am not surprised by seeing more student withdraw from the course. But, I have also found that their conceptual understanding is higher, and this is more difficult to measure. Did your exam from last semester align to the VL questions? Did your exam this semester align to the JE content?
In my opinion, and from my experience, students from this semester will be more prepared for calculus I, II, III, differential equations, chemistry, and physics since most of these faculty use journals. What I am curious to see is if your students who journal can explain the problems at a higher level than those who use video lessons? Do the students' writing skills improve as compared to those who only use VL? Is there a way to incorporate both VL and JE?
I have also found that journaling at the lower level mathematics may not be as beneficial, but starting them in College Algebra is great to help them be sucessful in most of their STEM courses. While I would want to have a journal in courses like MAT141 or MAT121, I don't know that I would. This is a great start to having conversations with math faculty about journals. I know that most of the calc folks use journals and won't look back. It took me 15 years to finally try them in my classroom. Again, thank you! This is great information to have.
In reply to Thanks for taking time to submit this CATS I have found by Becky Baranowski
It was my assessment of the written answers on the final exam from last semester that prompted, in part, my exporatory analysis. My exams from both semesters, I believe, had parts that would align with VLs and parts that would align with JEs. I've thought about somehow repurposing the notes as perhaps a guide for JEs when watching VLs, but at this point, it's just a thought, and I like the creative nature of freeform JEs. Some of the journals I've seen are amazing! Thank you for your thought provoking questions and kind remarks.
Hi Carrie,
Thank you for posting your CATS' results!
Although you stated there was not a significant difference, I thought this was an interesting finding. With our classes being in an online format, the VL and JE are some great tools to measure in their relationship with our students' retention.
Thank you for sharing your CATS, and the great work you are doing in your courses!
Catherine
In reply to Hi Carrie Thank you for posting your CATS results by Catherine Cochran
Thank you for your kind words. I was a bit bummed that JEs had a slightly higher drop rate than VLs, but I'm not able to tell now whether the longer term drops rates would have continued this pattern given COVID-19. I hope to hear whether the upstream instructors notice a difference with MAT151 students who journaled.