Comprehensive Assessment Tracking System

Video Lessons vs. Journal Entries

Saturday, January 11, 2020 to Friday, May 8, 2020
What is the Need/Assessment?: 

The goal is to explore whether Video Lessons (VLs) or Journal Entries (JEs) have a greater impact on student withdrawal rates and exam scores.

Describe the necessity for this change: 

MAT151 is a building block for students on the STEM path and having a stronger foundation should lead to better outcomes in higher level courses.

Describe what will be (or was) implemented to affect change: 

VLs, but no JEs, were completed by three sections of MAT151 in Fall 2019.  JEs, but no VLs, are being completed by three sections of MAT151 in Spring 2020.  There are similarities and differences across semesters, so a comparison of withdrawal rates and exam scores will be done on an exploratory basis.  The objective is to determine any changes in withdrawal rates and exam scores that could potentially be due to VLs or JEs.

Interpret, compare and describe the results of the change: 

Withdrawal rates will be explored using Kaplan Meier Curves and Log Rank Tests.  Exam scores will be explored using Histograms and Wilcoxon Rank Sum Tests.  The results are forthingcoming, as the current semester is in progress.  This space will be updated.

After analyzing the information, what are the next steps?: 

Based on the results of this exploratory study, and in conjunction with student feedback, my plan is to implement either VLs or JEs in future courses depending on which type of assessment appears to lead to greater student course retention and improved learning.

Abstract: 

The purpose of this CATS is to explore student withdrawal rates and exam scores from six MAT151 courses across two semesters to determine whether Video Lessons or Journal Entries appear to lead to greater student course retention and improved learning.  

See comments for update to close the loop.  

Division/Department: 
Completed Full Cycle: 
Yes
Course Number: 
MAT151

Comments

CAR2199376's picture
up
0 users have voted.

VLs vs. JEs: The results from the exploratory anaylsis are in!

Background: This analysis compares performance metrics through Exam 1 between Video Lessons (VLs) used in three classes in Semester 4196 and Journal Entries (JEs) used in three classes in Semester 4202.  Exam 1 scores are presented graphically using histograms.  Continuation rates through the Exam 1 date are displayed graphically using Kaplan-Meier survival curves.  Comparison of continuation rates between the two assessment techniques is performed using a Fisher's exact test for overall continuation rates and a log-rank test for the continuation curves.  Comparison of Exam 1 scores between the two assessment techniques is performed using a Wilcoxon rank sum test and a t-test.

Results: Exam 1 scores were obtained for 87 out of 90 students using VLs, with 3 students withdrawing from the course prior to Exam 1.  Using JEs, Exam 1 scores were obtained for 72 out of 79 students, with 7 students withdrawing prior to taking Exam 1.  Exam 1 was taken on the 40th day of the semester in all 6 classes.  The overall continuation rate on the Exam 1 date was 97% when using VLs versus 91% using JEs.  The difference in the overall continuation rate is not statistically significant at the 0.05 significance level (p=0.1917) using Fisher's exact test.   The continuation curves are also not statistically significantly different using the log-rank test (p=0.1273).  The mean (SD) of Exam 1 scores using VLs is 63.1 (19.5) compared to 58.2 (21.3) using JEs.  While Exam 1 scores are 4.9 points higher on average using VLs than JEs, the difference is not statistically significant at the 0.05 significance level using either the Wilcoxon rank sum test (p=0.1595) or t-test (p=0.1324). 

Summary: In summary, while there were no statistically significant differences between performance metrics using VLs and using JEs, students using VLs were continuing in the course at a higher rate than students using JEs and demonstrated a nearly 5-point improvement in Exam 1 scores compared to JEs.

Graphs: The summary graphs can be found here.

REBZS76641's picture
up
0 users have voted.

Thanks for taking time to submit this CATS.  I have found that students are not fond of writing a journal because it requires writing, and they are in a "math" class.  So, I definitely am not surprised by seeing more student withdraw from the course.  But, I have also found that their conceptual understanding is higher, and this is more difficult to measure.  Did your exam from last semester align to the VL questions?  Did your exam this semester align to the JE content?

In my opinion, and from my experience, students from this semester will be more prepared for calculus I, II, III, differential equations, chemistry, and physics since most of these faculty use journals.  What I am curious to see is if your students who journal can explain the problems at a higher level than those who use video lessons?  Do the students' writing skills improve as compared to those who only use VL?  Is there a way to incorporate both VL and JE?  

I have also found that journaling at the lower level mathematics may not be as beneficial, but starting them in College Algebra is great to help them be sucessful in most of their STEM courses.  While I would want to have a journal in courses like MAT141 or MAT121, I don't know that I would.  This is a great start to having conversations with math faculty about journals.  I know that most of the calc folks use journals and won't look back.  It took me 15 years to finally try them in my classroom. Again, thank you!  This is great information to have.