Calculus Adaptive Assessment: Technical Summary
How the project learns the difficulty of a derivative problem: generating problems, differentiating them step by step, collecting an instructor's comparisons, fitting and evaluating models, and explaining what they predict. This is a condensed web version; the lecture notes have every derivation.
At a Glance
| Piece | Details |
|---|---|
| Language | Python 3.11, with NumPy, SciPy, SymPy, scikit-learn, and PyTorch. |
| Problems | Generated by a probabilistic grammar over expression trees, so the supply of problems is effectively unlimited. |
| Labels | About 350 of my own “which is harder?” judgments, collected in a Streamlit labeling app and stored in SQLite. |
| Models | Bradley–Terry on 17 structural features, my own gradient-boosted Bradley–Terry, a mixed model for active learning, and a recursive neural network on expression trees. |
| Explanations | Exact Shapley values over 7 groups of features, turned into a one-sentence reason for each prediction. |
| Guessing game | A public Streamlit app on Streamlit Community Cloud, saving anonymous guesses to a Postgres database. |
| Write-up | Lecture notes with one section per step; every number and figure in them is regenerated from the code. |
| Testing | 79 pytest tests covering the generator, differentiator, features, every model, the explanations, and the game. |
The Pipeline
Sample derivative problems from a random grammar of expression trees.
Differentiate each one step by step and measure 17 features.
An instructor judges pairs: which problem is harder?
Fit a difficulty score that extends to problems nobody has labeled.
Why Comparisons, Not Ratings
Nobody observes difficulty directly. The obvious label, a rating from 1 to 5, drifts: the scale shifts during a session, and each rating is anchored on the previous one. A comparison only depends on which problem is harder, so that drift drops out. The labels are therefore pairwise comparisons, modeled by Bradley–Terry: each problem \(x\) has a difficulty \(\delta(x)\), and
\[\mathbb{P}(x \text{ judged harder than } x') = \sigma\bigl(\delta(x) - \delta(x')\bigr), \qquad \sigma(t) = \frac{1}{1 + e^{-t}}.\]A gap of 1 in difficulty means a 73% chance of being judged harder. The same equation is behind Elo ratings and the reward models used to train large language models. Because only a tiny fraction of problems will ever be labeled, the difficulty has to be a function of what a problem looks like, not one free number per problem.
Generating and Differentiating Problems
Problems come from a probabilistic grammar: a random tree of sums, products, quotients, powers, and compositions of trig, exponential, and logarithmic functions, with its branching chosen so the trees stay small. Problems that no instructor would write (stacked coefficients, for example) are rejected. Because every guarantee in learning is relative to a distribution, the generator is treated as a probability distribution in its own right, analyzed as a branching process.
A step-tracing differentiator then works out each derivative by structural recursion, recording which rules it used and how much the expression grows. Each result is checked against SymPy and with dual numbers: of 3,000 generated problems, 2,989 were verified exactly and the other 11 were inconclusive, never wrong.
Features
The models see 17 features of a problem and its derivative: its size and depth, the size of its derivative, how many times the chain, product, and quotient rules are used, how deeply chain rules nest, which function types appear, and tricky exponents such as negative or fractional powers. Several of these move together (chain-rule counts and nesting depth have correlation 0.92), so the features are standardized and the models are regularized.
Fitting and Evaluating
The basic model is linear Bradley–Terry: the difficulty is a weighted sum of the features, fit by ridge-regularized maximum likelihood with Newton's method, with the amount of regularization chosen by cross-validation. Evaluation always holds out whole problems, not just comparisons, because the point is to predict difficulty for problems the model has never seen, and confidence intervals come from a bootstrap over problems.
Every method was first rehearsed on a simulated instructor with a known true difficulty, so it could be checked against the right answer, before it was used on real labels.
Results on Real Labels
On my own judgments, the 17-feature model predicts held-out comparisons with 91.4% accuracy. That has to be compared with how consistent I am: on repeated pairs I agreed with my own earlier answer about 91% of the time, which puts the ceiling for any model around 94% (with a wide range, since there are few repeats). The model is close to it.
| Model | Accuracy | Log loss |
|---|---|---|
| Coin flip | 50.0% | 0.693 |
| Size of the problem only | 86.1% | 0.434 |
| Size of the derivative only | 92.5% | 0.378 |
| All 17 features | 91.4% | 0.310 |
“How much there is to write” does most of the work, and the quotient rule adds the clearest extra difficulty. The model also predicts things it was never trained on: my “about the same” answers concentrate on the pairs it considers closest, and I answered those more slowly. Reading its most confident mistakes turned up a bug in the problem generator, which a test had not caught.
Beyond the Linear Model
- Gradient boosting. My own gradient-boosted Bradley–Terry, where each tree's leaf values are found by a Newton step that turns out to be a graph-cut quantity. The best version uses single-split trees, an additive model with no interactions: there is no evidence that features combine in how I judge difficulty.
- Active learning. Choosing which pair to ask next. Uncertainty sampling clearly hurts; a V-optimal design on a mixed model with per-problem effects reaches the same accuracy with roughly a fifth fewer labels. The labeling app now uses it by default.
- A neural network on expression trees. A recursive network in PyTorch that reads the raw expression tree, pretrained on thousands of generated problems and then fine-tuned (linear probing, then fine-tuning). Trained from scratch it memorizes the few hundred labels; pretrained, it ties the feature model, with the best numbers so far (93.2% accuracy, log loss 0.297) but within the uncertainty.
Explaining a Prediction
For a single comparison, the predicted gap in difficulty is split among 7 groups of features (size, chain rule, quotient rule, product rule, function types, tricky exponents, powers and constants) using Shapley values, the only attribution rule satisfying four natural axioms. With 7 groups there are only 128 coalitions, so the values are computed exactly in one vectorized call, and they add up exactly to the model's predicted gap. The result becomes a sentence such as “Mostly the quotient rule and the size, despite the function types.”
Two caveats are spelled out in the notes. Models that predict equally well can divide the credit very differently, so the explanations describe the model, not the instructor. And they make blind spots visible: one problem the model rated hard simplifies to 1 before differentiating, something none of the 17 features can see.
The Guessing Game
The public app shows you two derivative problems and asks which is harder. After you answer, it shows the model's prediction, its Shapley explanation, and how often you have agreed with the model and with my own judgments. Each guess is stored with the role you choose (instructor, TA or tutor, student, or other) and an anonymous session ID. The app needs only NumPy and Streamlit: the model is exported as data, and a test checks that the deployed copy agrees exactly with the research code.
Next Steps
The guesses collected by the game will feed a multi-rater Bradley–Terry model, to study how instructors, TAs, and students see difficulty differently. After that, an optional final step is the adaptive quiz itself, using item response theory and validated on real student answers.