Skip to content
Analyst work

Data study · October 2026

AI and student learning

Question

Do coursework grades show what students remember, and what does an open AI chatbot do to the gap?

Headline finding

71.7% of the 4,649 students scored lower on the final exam than on their coursework. Students who studied in the most weeks of the term scored 5.9 points higher on the exam than the least consistent quarter, even after accounting for coursework grade and total online activity. Both figures are stored Open University results, and the code behind them was not kept.

My recommendation

The company should sell a hint-first tutor to schools, with weekly review sets built in. Schools should grade short weekly quizzes on old material and approve one guardrailed tutor.

Tools
  • JavaScript on Node for the derived rates
  • D3.js for the first dashboard
  • React and SVG for the charts on this page

The question

The client

Two clients. One is a school or college deciding what to grade. The other is an edtech company that sells homework answers, with Chegg as the case. Chegg had 4.7 million subscribers in Q1 2024 and 3.2 million a year later.

The decision they face

The school has to choose what counts toward a grade and what to report beside GPA. The company has to choose what to sell now that 54% of US teens say they use an AI chatbot for schoolwork.

The data, 8 tables

  • oulad_summary.csv6 rows

    The size of the Open University sample and its totals. It covers 4,649 students in 2 STEM courses across 6 course runs, with the average gap between final exam and coursework and the share of students who scored below coursework on the exam.

    Where it came from
    Computed from the Open University Learning Analytics Dataset (OULAD), which covers UK distance learning.
    Why I chose it
    OULAD holds coursework scores and a cumulative final exam for the same students, along with their click logs. That lets a grade sit beside what the student kept.
    What it cannot tell us
    Be rerun. The raw OULAD files and the code that produced these rows are not in this repo, so the figures stand as stored results. The data was also collected before ChatGPT existed.
  • oulad_quartiles.csv8 rows

    Students split into 4 equal groups by weekly consistency and again by deadline cramming. Each row has the group’s coursework average, final exam average, exam pass rate and exam difference after controls.

    Where it came from
    Computed from OULAD. Weekly consistency is the share of course weeks before the exam with any study activity. Deadline cramming is the share of a student’s clicks in the few days before each coursework deadline, divided by the share of course days those windows cover.
    Why I chose it
    It puts a student’s study pattern beside both scores, the coursework average and the exam.
    What it cannot tell us
    Show cause. Students who study every week may differ in ways the data can’t see. The raw data and the code behind the group means and the adjusted differences are not in this repo.
  • oulad_gradeband.csv20 rows

    Final exam average for each consistency group inside 5 coursework grade bands, with the number of students in each cell.

    Where it came from
    Computed from OULAD.
    Why I chose it
    A reader can compare students with similar coursework grades without trusting the regression.
    What it cannot tell us
    Hold steady in the small cells. The smallest has 69 students. The raw data and the code that produced these rows are not in this repo.
  • oulad_model.csv4 rows

    One regression of final exam score on weekly consistency, deadline cramming, coursework average and the log of total study clicks. It has course-run fixed effects and standard errors that do not assume equal variance across students. Each row holds one term with its estimate and p-value.

    Where it came from
    Computed from OULAD.
    Why I chose it
    It tests consistency and cramming in the same model, with coursework grade and total clicks held equal.
    What it cannot tell us
    Be refit or audited. The repo holds 4 coefficients with no code and no raw data behind them. P-values were stored to four decimal places, so 2 of them read 0.0.
  • wharton_rct.csv2 rows

    Practice and exam score changes against a control group for two AI arms, an open chatbot and a tutor with guardrails. The students were in high school math classes in Turkey (Bastani et al., PNAS 2025).

    Where it came from
    Proceedings of the National Academy of Sciences.
    Why I chose it
    It is a field experiment with a control group, and it tested students on an exam with the AI taken away.
    What it cannot tell us
    Speak for other subjects or school systems. The tutor arm’s exam change is stored as 0 because the paper reports it as about the same as control, so that cell is a reading of the paper’s wording.
  • pew_ai_use.csv3 rows

    The share of US teens aged 13 to 17 who used ChatGPT for schoolwork in 2023 and 2024, and the share who use any AI chatbot for schoolwork in 2025.

    Where it came from
    Pew Research Center survey releases.
    Why I chose it
    It sizes how many students now do schoolwork with a chatbot at hand.
    What it cannot tell us
    Give a trend. Pew asked about ChatGPT in the first two surveys and about any AI chatbot in the third, so the last figure answers a broader question.
  • national_trends.csv5 rows

    Start and end values for US high school GPA and for ACT and NAEP grade 12 test scores.

    Where it came from
    Two NAEP publications, the High School Transcript Study and the Nation’s Report Card, plus ACT’s grade inflation report.
    Why I chose it
    It checks whether grades and test scores moved apart outside one university.
    What it cannot tell us
    Say anything about AI. The GPA series end in 2019 and 2021, before ChatGPT, and the 2024 NAEP seniors had a freshman year disrupted by the pandemic. Each series has two points, so the path between them is unknown.
  • edtech_signals.csv4 rows

    Chegg’s subscriber counts for Q1 2024 and Q1 2025 with the year-over-year change, and the share of students with Khanmigo access who use it regularly.

    Where it came from
    Chegg’s quarterly results and a Khan Academy statement, both taken from news coverage.
    Why I chose it
    It gives the company track two market readings, one for a paid answer service and one for a guardrailed tutor.
    What it cannot tell us
    Explain why subscribers left. The Khanmigo figure is Khan Academy’s own statement, and the table holds no definition of regular use.

The process

  1. Sample

    The Open University work used the 2 OULAD courses with a scored final exam, across 6 course runs. Students with almost no platform activity were removed, which left 4,649. The cutoff for that filter was not recorded.

  2. Measures

    Weekly consistency is the share of course weeks before the exam with any study activity. Deadline cramming compares a student’s clicks in the few days before each coursework deadline with the share of course days those windows cover. Each measure splits the students into 4 equal groups.

  3. Model

    One regression holds consistency, cramming, coursework average and the log of total study clicks, with course-run fixed effects and standard errors that do not assume equal variance. R² is 0.485. The raw files and the code for this step were not saved, so derive.mjs reads the stored coefficients and cannot refit them.

  4. Pivot

    oulad_gradeband.csv is long, with one row per grade band per consistency group. I pivoted it to one row per band and added the gap between the most and least consistent groups.

  5. Types

    Every value in oulad_summary.csv is stored as a decimal, so the student count reads 4649.0. P-values are stored to four places, and 2 of them read 0.0. The code turns the counts into whole numbers and prints those p-values as <0.001.

  6. Check

    The stored tables cannot be rerun, so I checked them against each other. Group sizes add up to 4,649 in both splits and in the grade band table. The student-weighted gap of the groups lands within 0.03 points of the stored average, and each group’s exam average rebuilt from its band rows lands within 0.05 points.

  7. Sources

    The other 4 tables were compiled from published reports. They carry no source URL and no retrieval date, and only national_trends.csv names a source on each row. Chegg reported a 31% fall in subscribers. The two rounded counts give 31.9%, and the rounding allows anything from 30.1% to 33.7%.

One grade band before and after the pivot.

Before, 4 rows of oulad_gradeband.csv
coursework_band,consistency_quartile,exam_avg,students,band_order
80 to 90,Q1 least,70.3,213,4
80 to 90,Q2,73.8,275,4
80 to 90,Q3,75.2,368,4
80 to 90,Q4 most,76.0,391,4
After, one row per band
{
  "band": "80 to 90",
  "students": 1247,
  "q1": 70.3,
  "q2": 73.8,
  "q3": 75.2,
  "q4": 76,
  "spread": 5.7
}

The 80 to 90 band holds 1,247 students. The spread is the most consistent group’s exam average minus the least consistent group’s.

The code for the key steps

The pivot from long rows to one row per grade band. derive.mjs
export function bandWide(oulad_gradeband) {
  const keys = { "Q1 least": "q1", Q2: "q2", Q3: "q3", "Q4 most": "q4" };
  const by = {};
  oulad_gradeband.forEach((r) => {
    by[r.coursework_band] = by[r.coursework_band] || { band: r.coursework_band, order: Number(r.band_order), students: 0 };
    by[r.coursework_band][keys[r.consistency_quartile]] = Number(r.exam_avg);
    by[r.coursework_band].students += Number(r.students);
  });
  return Object.values(by)
    .sort((a, b) => a.order - b.order)
    .map((d) => ({ ...d, spread: d.q4 - d.q1, in_order: d.q1 < d.q2 && d.q2 < d.q3 && d.q3 < d.q4 }));
}
The check of the stored average gap against the group rows. derive.mjs
export function gapCheck(oulad_summary, oulad_quartiles) {
  const stored = Number(oulad_summary.find((r) => r.metric === "avg_gap").value);
  const students = (r) => Number(r.students);
  const splits = [...new Set(oulad_quartiles.map((r) => r.grouping))].map((split) => {
    const rows = oulad_quartiles.filter((r) => r.grouping === split);
    const weighted = sum(rows, (r) => Number(r.gap) * students(r)) / sum(rows, students);
    return { split, weighted, stored, diff: Math.abs(weighted - stored) };
  });
  const rows = oulad_quartiles.map((r) => {
    const from_means = Number(r.exam_avg) - Number(r.coursework_avg);
    return { split: r.grouping, group: r.quartile, stored: Number(r.gap), from_means, diff: Math.abs(Number(r.gap) - from_means) };
  });
  return { splits, rows, worst_split_diff: Math.max(...splits.map((s) => s.diff)), worst_row_diff: Math.max(...rows.map((r) => r.diff)) };
}

Dead ends

  • Cramming

    Split on cramming alone, the heaviest crammers scored 2.6 points lower after controls. With consistency in the same model, one standard deviation more cramming goes with 0.51 fewer exam points, and the p-value is 0.076. That is above 0.05, so no option here rests on cramming.

  • The lowest grade band

    In the “Under 60” coursework band the pattern is gone. The most consistent group averaged 45.5 on the exam and the least consistent group averaged 45.9. The gap favors the consistent group in 4 of 5 bands.

  • AI in the national series

    The national table cannot carry a claim about AI. In the federal transcript study GPA rose from 2.68 in 1990 to 3.11 in 2019, and that series ends before ChatGPT. NAEP’s 2024 seniors also had a pandemic freshman year.

The 3 findings that change the decision

  1. Finding 1

    71.7%

    Students whose final exam came in below their coursework average

    71.7% of the 4,649 students scored lower on the final exam than on their coursework, and the average gap was 9.7 points.

    Dataset oulad_summary.csv
    Query derive.mjs, summary()

    Every consistency group scored at least 8.6 points lower on the exam than on coursework

    Coursework average minus final exam average for 4 equal groups of students. Q1 studied in the fewest weeks of the term and Q4 in the most. These are stored results from OULAD.

    Points below coursework on the final exam

    Source: oulad_quartiles.csv, oulad_summary.csv, derive.mjs, groups().

    Show the numbers as a table
    Consistency groupPoints below coursework on the final examStudentsCoursework averageFinal exam average
    Q1 least11.31,16568.657.2
    Q29.61,16072.663.0
    Q39.41,16077.267.7
    Q4 most8.61,16481.372.7
  2. Finding 2

    5.9 points

    Exam edge for the most consistent group over the least, after accounting for grade and activity

    After accounting for coursework grade and total online activity within each course, the most consistent quarter of students scored 5.9 points higher on the final exam than the least consistent quarter.

    Dataset oulad_quartiles.csv
    Query derive.mjs, groups()

    The most consistent group scored 5.9 points higher on the exam, after accounting for grade and activity

    Each bar compares a group with the least consistent group, which is the baseline and sits at zero. Before accounting for grade and activity, the two end groups were 15.5 exam points apart. In the full model, each standard deviation of extra consistency goes with 2.17 more exam points, and the result is very unlikely to be chance (p <0.001). The code for these results was not kept.

    Exam points above the least consistent group, after controls

    Source: oulad_quartiles.csv, oulad_model.csv, derive.mjs, groups().

    Show the numbers as a table
    Consistency groupExam points above the least consistent group, after controlsStudentsFinal exam averageExam pass rate
    Q1 least0.01,16557.280.6%
    Q22.91,16063.087.3%
    Q34.01,16067.790.9%
    Q4 most5.91,16472.794.5%
  3. Finding 3

    17% lower

    Exam score against the control group, students given an open chatbot

    In a Wharton field experiment, students given an open chatbot scored 48% higher than the control group on practice problems and 17% lower on the exam they took without it.

    Dataset wharton_rct.csv
    Query derive.mjs, rctArms()

    Students with an open chatbot scored 17% lower on the exam, and students with the hint-first tutor matched the control group

    Exam score below the control group, in percent, for high school math students in Turkey (Bastani et al., PNAS 2025). The tutor gave hints and held back full answers. Its exam change is stored as 0 because the paper reports it as about the same as control.

    Exam score below control (%)

    Source: wharton_rct.csv, derive.mjs, rctArms().

    Show the numbers as a table
    ArmExam score below control (%)Practice score against controlExam score against control
    GPT Base (open chatbot)17%+48%-17%
    GPT Tutor (guardrails)0%+127%0%

My recommendation

The company should sell a hint-first tutor to schools. Schools should grade weekly quizzes and approve one tutor.

  1. A1

    Guardrailed tutor as the default

    Company track. Hints first, with full answers only after the student shows work.

    Evidence
    Finding 3. Students given the hint-first tutor gained 127% on practice and scored about the same as the control group on the exam.
    Main risk
    Khanmigo’s low regular use suggests students drift back to open chatbots. 15% of students with access use it regularly.
  2. A2

    Sell to schools and colleges

    Company track. Instructors assign review sets and see exam readiness for each student.

    Evidence
    Finding 2. Consistency gives the company a measurable outcome to sell. Each standard deviation of extra consistency goes with 2.17 more exam points.
    Main risk
    Schools are slow to buy, and they’re a new kind of customer for the company.
  3. B1

    Grade weekly low-stakes quizzes

    School track. Short weekly quizzes that mix in old material and count toward the grade.

    Evidence
    Finding 2. After accounting for grade and activity, the exam edge grows with each step up in consistency and reaches 5.9 points for the top group.
    Main risk
    More grading, which auto-graded quizzes cut down.
  4. B2

    Approve one guardrailed AI tutor

    School track. Approve one tutor with guardrails and set rules for open chatbots on coursework.

    Evidence
    Finding 3. Students with an open chatbot gained 48% on practice and lost 17% on the exam. Students with the hint-first tutor gained 127% on practice and held level with the control group.
    Main risk
    Rules are hard to enforce at home.
  • The hint-first tutor is the one company option a trial has tested. In finding 3 it gained on practice and held level with control on the exam.
  • Selling through schools answers the tutor’s main risk. Students drift away from a tutor they picked themselves, and an instructor who assigns review sets keeps them coming back.
  • Weekly quizzes go straight at finding 2, where steady weekly study went with higher exam scores after controls.
  • One approved tutor with clear rules for coursework answers finding 3, where the open chatbot cost students 17% on the exam.

Limits and next steps

What the data cannot support

  • The four oulad tables are stored results. The raw Open University files and the code for the group means and the regression are not in this repo, so this page cannot rerun them. derive.mjs only checks that the stored tables agree with each other.
  • The Open University data was collected before ChatGPT existed. It shows the gap between coursework and exam was there before AI, and it holds nothing on AI use.
  • The regression shows association. Students who study steadily may differ in ways the data can’t see.
  • A lower score on the exam than on coursework is no proof of forgetting. The two are different assessments, and the tables hold no retest of the same material.
  • Total study clicks stand in for study time. In the model more clicks go with a lower exam score at the same coursework grade, 1.78 points per log unit, so I read clicks as activity and make no claim about hours.
  • The AI evidence is one experiment in high school math in Turkey. Pew’s 54% answers a broader question than its 13% and 26%, so the Pew figures are separate readings.

What I would test with more access

  • With the raw OULAD files I would rerun the group means and the regression and save the code here, so a reader can refit the model.
  • With a school’s gradebook I would run a retention check on old material and compare the retest scores with GPA.
  • With an edtech company’s logs I would test a weekly review mode against a holdout group and measure final exam scores.
  • I would add a source URL and a retrieval date to every compiled row.

TL;DR

  1. Studied

    How well coursework grades predict final exams, and what an open AI chatbot does to that gap.

  2. Found

    71.7% of the 4,649 students scored lower on the final exam than on their coursework.

  3. By the numbers

    Exam below coursework average
    71.7%
    Exam edge for the most consistent studiers
    5.9 points
    Exam score with an open chatbot
    17% lower
  4. The catch

    The Open University figures are stored results, and the code that produced them was not kept.

  5. Next

    With the raw OULAD files, rerun the group means and the regression and save the code here.