Data study · October 2026
AI and student learning
- Question
Do coursework grades show what students remember, and what does an open AI chatbot do to the gap?
- Headline finding
71.7% of the 4,649 students scored lower on the final exam than on their coursework. Students who studied in the most weeks of the term scored 5.9 points higher on the exam than the least consistent quarter, even after accounting for coursework grade and total online activity. Both figures are stored Open University results, and the code behind them was not kept.
- My recommendation
The company should sell a hint-first tutor to schools, with weekly review sets built in. Schools should grade short weekly quizzes on old material and approve one guardrailed tutor.
- Tools
- JavaScript on Node for the derived rates
- D3.js for the first dashboard
- React and SVG for the charts on this page
The question
- The client
Two clients. One is a school or college deciding what to grade. The other is an edtech company that sells homework answers, with Chegg as the case. Chegg had 4.7 million subscribers in Q1 2024 and 3.2 million a year later.
- The decision they face
The school has to choose what counts toward a grade and what to report beside GPA. The company has to choose what to sell now that 54% of US teens say they use an AI chatbot for schoolwork.
The data, 8 tables
oulad_summary.csv6 rows
The size of the Open University sample and its totals. It covers 4,649 students in 2 STEM courses across 6 course runs, with the average gap between final exam and coursework and the share of students who scored below coursework on the exam.
- Where it came from
- Computed from the Open University Learning Analytics Dataset (OULAD), which covers UK distance learning.
- Why I chose it
- OULAD holds coursework scores and a cumulative final exam for the same students, along with their click logs. That lets a grade sit beside what the student kept.
- What it cannot tell us
- Be rerun. The raw OULAD files and the code that produced these rows are not in this repo, so the figures stand as stored results. The data was also collected before ChatGPT existed.
oulad_quartiles.csv8 rows
Students split into 4 equal groups by weekly consistency and again by deadline cramming. Each row has the group’s coursework average, final exam average, exam pass rate and exam difference after controls.
- Where it came from
- Computed from OULAD. Weekly consistency is the share of course weeks before the exam with any study activity. Deadline cramming is the share of a student’s clicks in the few days before each coursework deadline, divided by the share of course days those windows cover.
- Why I chose it
- It puts a student’s study pattern beside both scores, the coursework average and the exam.
- What it cannot tell us
- Show cause. Students who study every week may differ in ways the data can’t see. The raw data and the code behind the group means and the adjusted differences are not in this repo.
oulad_gradeband.csv20 rows
Final exam average for each consistency group inside 5 coursework grade bands, with the number of students in each cell.
- Where it came from
- Computed from OULAD.
- Why I chose it
- A reader can compare students with similar coursework grades without trusting the regression.
- What it cannot tell us
- Hold steady in the small cells. The smallest has 69 students. The raw data and the code that produced these rows are not in this repo.
oulad_model.csv4 rows
One regression of final exam score on weekly consistency, deadline cramming, coursework average and the log of total study clicks. It has course-run fixed effects and standard errors that do not assume equal variance across students. Each row holds one term with its estimate and p-value.
- Where it came from
- Computed from OULAD.
- Why I chose it
- It tests consistency and cramming in the same model, with coursework grade and total clicks held equal.
- What it cannot tell us
- Be refit or audited. The repo holds 4 coefficients with no code and no raw data behind them. P-values were stored to four decimal places, so 2 of them read 0.0.
wharton_rct.csv2 rows
Practice and exam score changes against a control group for two AI arms, an open chatbot and a tutor with guardrails. The students were in high school math classes in Turkey (Bastani et al., PNAS 2025).
- Where it came from
- Proceedings of the National Academy of Sciences.
- Why I chose it
- It is a field experiment with a control group, and it tested students on an exam with the AI taken away.
- What it cannot tell us
- Speak for other subjects or school systems. The tutor arm’s exam change is stored as 0 because the paper reports it as about the same as control, so that cell is a reading of the paper’s wording.
pew_ai_use.csv3 rows
The share of US teens aged 13 to 17 who used ChatGPT for schoolwork in 2023 and 2024, and the share who use any AI chatbot for schoolwork in 2025.
- Where it came from
- Pew Research Center survey releases.
- Why I chose it
- It sizes how many students now do schoolwork with a chatbot at hand.
- What it cannot tell us
- Give a trend. Pew asked about ChatGPT in the first two surveys and about any AI chatbot in the third, so the last figure answers a broader question.
national_trends.csv5 rows
Start and end values for US high school GPA and for ACT and NAEP grade 12 test scores.
- Where it came from
- Two NAEP publications, the High School Transcript Study and the Nation’s Report Card, plus ACT’s grade inflation report.
- Why I chose it
- It checks whether grades and test scores moved apart outside one university.
- What it cannot tell us
- Say anything about AI. The GPA series end in 2019 and 2021, before ChatGPT, and the 2024 NAEP seniors had a freshman year disrupted by the pandemic. Each series has two points, so the path between them is unknown.
edtech_signals.csv4 rows
Chegg’s subscriber counts for Q1 2024 and Q1 2025 with the year-over-year change, and the share of students with Khanmigo access who use it regularly.
- Where it came from
- Chegg’s quarterly results and a Khan Academy statement, both taken from news coverage.
- Why I chose it
- It gives the company track two market readings, one for a paid answer service and one for a guardrailed tutor.
- What it cannot tell us
- Explain why subscribers left. The Khanmigo figure is Khan Academy’s own statement, and the table holds no definition of regular use.
The process
Sample
The Open University work used the 2 OULAD courses with a scored final exam, across 6 course runs. Students with almost no platform activity were removed, which left 4,649. The cutoff for that filter was not recorded.
Measures
Weekly consistency is the share of course weeks before the exam with any study activity. Deadline cramming compares a student’s clicks in the few days before each coursework deadline with the share of course days those windows cover. Each measure splits the students into 4 equal groups.
Model
One regression holds consistency, cramming, coursework average and the log of total study clicks, with course-run fixed effects and standard errors that do not assume equal variance. R² is 0.485. The raw files and the code for this step were not saved, so derive.mjs reads the stored coefficients and cannot refit them.
Pivot
oulad_gradeband.csv is long, with one row per grade band per consistency group. I pivoted it to one row per band and added the gap between the most and least consistent groups.
Types
Every value in oulad_summary.csv is stored as a decimal, so the student count reads 4649.0. P-values are stored to four places, and 2 of them read 0.0. The code turns the counts into whole numbers and prints those p-values as <0.001.
Check
The stored tables cannot be rerun, so I checked them against each other. Group sizes add up to 4,649 in both splits and in the grade band table. The student-weighted gap of the groups lands within 0.03 points of the stored average, and each group’s exam average rebuilt from its band rows lands within 0.05 points.
Sources
The other 4 tables were compiled from published reports. They carry no source URL and no retrieval date, and only national_trends.csv names a source on each row. Chegg reported a 31% fall in subscribers. The two rounded counts give 31.9%, and the rounding allows anything from 30.1% to 33.7%.
One grade band before and after the pivot.
coursework_band,consistency_quartile,exam_avg,students,band_order
80 to 90,Q1 least,70.3,213,4
80 to 90,Q2,73.8,275,4
80 to 90,Q3,75.2,368,4
80 to 90,Q4 most,76.0,391,4{
"band": "80 to 90",
"students": 1247,
"q1": 70.3,
"q2": 73.8,
"q3": 75.2,
"q4": 76,
"spread": 5.7
}The 80 to 90 band holds 1,247 students. The spread is the most consistent group’s exam average minus the least consistent group’s.
The code for the key steps
export function bandWide(oulad_gradeband) {
const keys = { "Q1 least": "q1", Q2: "q2", Q3: "q3", "Q4 most": "q4" };
const by = {};
oulad_gradeband.forEach((r) => {
by[r.coursework_band] = by[r.coursework_band] || { band: r.coursework_band, order: Number(r.band_order), students: 0 };
by[r.coursework_band][keys[r.consistency_quartile]] = Number(r.exam_avg);
by[r.coursework_band].students += Number(r.students);
});
return Object.values(by)
.sort((a, b) => a.order - b.order)
.map((d) => ({ ...d, spread: d.q4 - d.q1, in_order: d.q1 < d.q2 && d.q2 < d.q3 && d.q3 < d.q4 }));
}export function gapCheck(oulad_summary, oulad_quartiles) {
const stored = Number(oulad_summary.find((r) => r.metric === "avg_gap").value);
const students = (r) => Number(r.students);
const splits = [...new Set(oulad_quartiles.map((r) => r.grouping))].map((split) => {
const rows = oulad_quartiles.filter((r) => r.grouping === split);
const weighted = sum(rows, (r) => Number(r.gap) * students(r)) / sum(rows, students);
return { split, weighted, stored, diff: Math.abs(weighted - stored) };
});
const rows = oulad_quartiles.map((r) => {
const from_means = Number(r.exam_avg) - Number(r.coursework_avg);
return { split: r.grouping, group: r.quartile, stored: Number(r.gap), from_means, diff: Math.abs(Number(r.gap) - from_means) };
});
return { splits, rows, worst_split_diff: Math.max(...splits.map((s) => s.diff)), worst_row_diff: Math.max(...rows.map((r) => r.diff)) };
}Dead ends
Cramming
Split on cramming alone, the heaviest crammers scored 2.6 points lower after controls. With consistency in the same model, one standard deviation more cramming goes with 0.51 fewer exam points, and the p-value is 0.076. That is above 0.05, so no option here rests on cramming.
The lowest grade band
In the “Under 60” coursework band the pattern is gone. The most consistent group averaged 45.5 on the exam and the least consistent group averaged 45.9. The gap favors the consistent group in 4 of 5 bands.
AI in the national series
The national table cannot carry a claim about AI. In the federal transcript study GPA rose from 2.68 in 1990 to 3.11 in 2019, and that series ends before ChatGPT. NAEP’s 2024 seniors also had a pandemic freshman year.
The 3 findings that change the decision
Finding 1
71.7%
Students whose final exam came in below their coursework average
71.7% of the 4,649 students scored lower on the final exam than on their coursework, and the average gap was 9.7 points.
Dataset oulad_summary.csv
Query derive.mjs, summary()Every consistency group scored at least 8.6 points lower on the exam than on coursework
Coursework average minus final exam average for 4 equal groups of students. Q1 studied in the fewest weeks of the term and Q4 in the most. These are stored results from OULAD.
Points below coursework on the final exam
Source: oulad_quartiles.csv, oulad_summary.csv, derive.mjs, groups().
Show the numbers as a table
Consistency group Points below coursework on the final exam Students Coursework average Final exam average Q1 least 11.3 1,165 68.6 57.2 Q2 9.6 1,160 72.6 63.0 Q3 9.4 1,160 77.2 67.7 Q4 most 8.6 1,164 81.3 72.7 Finding 2
5.9 points
Exam edge for the most consistent group over the least, after accounting for grade and activity
After accounting for coursework grade and total online activity within each course, the most consistent quarter of students scored 5.9 points higher on the final exam than the least consistent quarter.
Dataset oulad_quartiles.csv
Query derive.mjs, groups()The most consistent group scored 5.9 points higher on the exam, after accounting for grade and activity
Each bar compares a group with the least consistent group, which is the baseline and sits at zero. Before accounting for grade and activity, the two end groups were 15.5 exam points apart. In the full model, each standard deviation of extra consistency goes with 2.17 more exam points, and the result is very unlikely to be chance (p <0.001). The code for these results was not kept.
Exam points above the least consistent group, after controls
Source: oulad_quartiles.csv, oulad_model.csv, derive.mjs, groups().
Show the numbers as a table
Consistency group Exam points above the least consistent group, after controls Students Final exam average Exam pass rate Q1 least 0.0 1,165 57.2 80.6% Q2 2.9 1,160 63.0 87.3% Q3 4.0 1,160 67.7 90.9% Q4 most 5.9 1,164 72.7 94.5% Finding 3
17% lower
Exam score against the control group, students given an open chatbot
In a Wharton field experiment, students given an open chatbot scored 48% higher than the control group on practice problems and 17% lower on the exam they took without it.
Dataset wharton_rct.csv
Query derive.mjs, rctArms()Students with an open chatbot scored 17% lower on the exam, and students with the hint-first tutor matched the control group
Exam score below the control group, in percent, for high school math students in Turkey (Bastani et al., PNAS 2025). The tutor gave hints and held back full answers. Its exam change is stored as 0 because the paper reports it as about the same as control.
Exam score below control (%)
Source: wharton_rct.csv, derive.mjs, rctArms().
Show the numbers as a table
Arm Exam score below control (%) Practice score against control Exam score against control GPT Base (open chatbot) 17% +48% -17% GPT Tutor (guardrails) 0% +127% 0%
My recommendation
The company should sell a hint-first tutor to schools. Schools should grade weekly quizzes and approve one tutor.
- A1
Guardrailed tutor as the default
Company track. Hints first, with full answers only after the student shows work.
- Evidence
- Finding 3. Students given the hint-first tutor gained 127% on practice and scored about the same as the control group on the exam.
- Main risk
- Khanmigo’s low regular use suggests students drift back to open chatbots. 15% of students with access use it regularly.
- A2
Sell to schools and colleges
Company track. Instructors assign review sets and see exam readiness for each student.
- Evidence
- Finding 2. Consistency gives the company a measurable outcome to sell. Each standard deviation of extra consistency goes with 2.17 more exam points.
- Main risk
- Schools are slow to buy, and they’re a new kind of customer for the company.
- B1
Grade weekly low-stakes quizzes
School track. Short weekly quizzes that mix in old material and count toward the grade.
- Evidence
- Finding 2. After accounting for grade and activity, the exam edge grows with each step up in consistency and reaches 5.9 points for the top group.
- Main risk
- More grading, which auto-graded quizzes cut down.
- B2
Approve one guardrailed AI tutor
School track. Approve one tutor with guardrails and set rules for open chatbots on coursework.
- Evidence
- Finding 3. Students with an open chatbot gained 48% on practice and lost 17% on the exam. Students with the hint-first tutor gained 127% on practice and held level with the control group.
- Main risk
- Rules are hard to enforce at home.
- The hint-first tutor is the one company option a trial has tested. In finding 3 it gained on practice and held level with control on the exam.
- Selling through schools answers the tutor’s main risk. Students drift away from a tutor they picked themselves, and an instructor who assigns review sets keeps them coming back.
- Weekly quizzes go straight at finding 2, where steady weekly study went with higher exam scores after controls.
- One approved tutor with clear rules for coursework answers finding 3, where the open chatbot cost students 17% on the exam.
Limits and next steps
What the data cannot support
- The four oulad tables are stored results. The raw Open University files and the code for the group means and the regression are not in this repo, so this page cannot rerun them. derive.mjs only checks that the stored tables agree with each other.
- The Open University data was collected before ChatGPT existed. It shows the gap between coursework and exam was there before AI, and it holds nothing on AI use.
- The regression shows association. Students who study steadily may differ in ways the data can’t see.
- A lower score on the exam than on coursework is no proof of forgetting. The two are different assessments, and the tables hold no retest of the same material.
- Total study clicks stand in for study time. In the model more clicks go with a lower exam score at the same coursework grade, 1.78 points per log unit, so I read clicks as activity and make no claim about hours.
- The AI evidence is one experiment in high school math in Turkey. Pew’s 54% answers a broader question than its 13% and 26%, so the Pew figures are separate readings.
What I would test with more access
- With the raw OULAD files I would rerun the group means and the regression and save the code here, so a reader can refit the model.
- With a school’s gradebook I would run a retention check on old material and compare the retest scores with GPA.
- With an edtech company’s logs I would test a weekly review mode against a holdout group and measure final exam scores.
- I would add a source URL and a retrieval date to every compiled row.
TL;DR
Studied
How well coursework grades predict final exams, and what an open AI chatbot does to that gap.
Found
71.7% of the 4,649 students scored lower on the final exam than on their coursework.
By the numbers
- Exam below coursework average
- 71.7%
- Exam edge for the most consistent studiers
- 5.9 points
- Exam score with an open chatbot
- 17% lower
The catch
The Open University figures are stored results, and the code that produced them was not kept.
Next
With the raw OULAD files, rerun the group means and the regression and save the code here.