From formula application to model judgment
Modified
Input
Formula application asks whether a student can execute a known procedure; model judgment asks whether the procedure belongs to the problem. Small changes in assumptions can require a different estimator, validation design, or decision rule. AI education should assess transfer across altered problems rather than memory of a fixed solution template.
Knowing a Formula Is Not Knowing a Model
A student can correctly apply a formula without understanding the model that gives the formula meaning.
Consider ordinary least squares:
Executing the expression demonstrates matrix manipulation. It does not demonstrate that $X^{\top}X$ is invertible, that the variables were measured at the correct time, that the conditional mean is appropriately represented, or that the coefficient answers the institutional question.
Formula application begins after the difficult choices have been made. Model judgment begins before them.
It asks:
- What is the estimand?
- Why do the observations exist?
- Which variables belong in the system?
- Which assumptions connect the estimator to the target?
- What changes when the environment changes?
- Which conclusion remains defensible after those changes?
This distinction is central to AI education because modern libraries can execute formulas more quickly and reliably than most humans. The durable human contribution lies in choosing, adapting, and criticizing the formal system.
The Layers of a Model-Based Answer
Table 1. Layers of a model-based answer
| Layer | Formula-oriented question | Judgment-oriented question |
|---|---|---|
| Target | Which variable is $Y$? | What quantity or decision is actually needed? |
| Sample | How many observations are available? | Why were these observations selected? |
| Representation | Which model has been assigned? | Which structure is plausible and useful? |
| Estimation | Which formula calculates $\widehat\theta$? | Under which assumptions does it estimate the intended target? |
| Validation | Is the numerical answer correct? | Would the conclusion survive new data or altered conditions? |
| Decision | What prediction is produced? | What action follows, at what cost, and for whom? |
A conventional exercise often supplies the first column and asks the student to compute. A serious AI problem withholds part of the structure and asks the student to construct it.
An Example: Omitted Structure
Suppose the true process is
but the fitted model omits $Z$:
The resulting estimator is
The formula is calculated correctly. Yet if $X$ and $Z$ are related, the coefficient combines the effect attributed to $X$ with structure carried by $Z$.
No amount of faster matrix multiplication repairs the missing variable. The remedy may require better measurement, a design that creates exogenous variation, a different estimand, or an explicit acknowledgement that the desired effect is not identified.
Model judgment is the ability to locate this problem before celebrating numerical precision.
The Importance of the Data-Generating Process
The same observed variables can support different models depending on how the data were generated.
Suppose $X$ is participation in an optional training program and $Y$ is later productivity. A regression of $Y$ on $X$ may estimate an association. It does not automatically estimate the effect of training because participation may depend on motivation $M$:
If $M$ is unobserved, the regression coefficient generally combines $\tau$ and selection through $M$.
Now change one sentence in the problem:
Participation was randomly assigned among eligible employees.
The mathematical task changes. Randomization can make $X$ independent of potential outcomes within the eligible group, permitting a causal interpretation under the design.
A student who memorized a regression template may produce the same calculation for both problems. A student with model judgment recognizes that a small verbal change has altered the inferential structure.
Assumptions Are Active Components
Students often treat assumptions as a list to reproduce after the calculation. In model design, each assumption performs work.
For the linear conditional-mean model,
the zero-conditional-mean assumption connects observable variation in $x_i$ to the coefficient $\beta$. If it fails, the same estimator may target something else.
Homoskedasticity,
is different. Its failure does not necessarily bias the OLS coefficient under zero conditional mean, but it changes conventional variance calculations and efficiency claims.
The distinction matters because remedies should match failures. A robust standard error may address heteroskedastic inference. It does not correct endogeneity.
Read as a diagnostic rather than a second scorecard, the framework becomes clear: Heteroskedasticity — What changes: Standard errors and efficiency; What does not automatically change: Conditional-mean identification. Omitted confounder — What changes: Interpretation and generally the estimand; What does not automatically change: Numerical ability to run regression. Nonlinearity — What changes: Functional representation and marginal effects; What does not automatically change: The need to define a target. Distribution shift — What changes: Deployment risk; What does not automatically change: Historical in-sample fit. Label error — What changes: Measurement and attainable performance; What does not automatically change: Code correctness.
Model judgment prevents the ritual use of a familiar correction for an unrelated problem.
Prediction and Explanation Require Different Judgments
A predictive model selects a function to perform well on future observations:
An explanatory or causal model may target a parameter such as
The first problem asks which mapping predicts $Y$. The second asks how $Y$ would change under an intervention. A variable can improve prediction while damaging causal interpretation. A valid instrument can identify a causal effect while offering little predictive improvement.
The appropriate formula depends on the question. Model judgment begins by refusing to use “accuracy” as a universal objective.
A Rubric for Model Judgment
Expressed as a practical comparison rather than another table, the distinctions are clear: Target — Weak answer: Repeats the supplied outcome; Competent answer: Defines the estimand or decision; Strong answer: Distinguishes alternative targets and consequences. Assumptions — Weak answer: Lists memorized conditions; Competent answer: Connects conditions to validity; Strong answer: Ranks assumptions by vulnerability. Method — Weak answer: Names a familiar algorithm; Competent answer: Matches method to structure; Strong answer: Compares methods and explains trade-offs. Diagnostics — Weak answer: Reports a performance number; Competent answer: Tests relevant failure modes; Strong answer: Designs falsification and sensitivity analysis. Interpretation — Weak answer: Describes coefficient or prediction; Competent answer: States scope and uncertainty; Strong answer: Explains what cannot be concluded. Adaptation — Weak answer: Reuses the original solution; Competent answer: Revises after a changed assumption; Strong answer: Anticipates the change before calculating.
The rubric rewards reasoning that remains useful when tools and datasets change.
Applied Example: When a Count Model Changes
Suppose an insurer models the number of claims $Y_i$ for policyholder $i$ as
The Poisson model implies
A student may correctly estimate the model with standard software. Model judgment begins when observed variance substantially exceeds the conditional mean.
One explanation is unobserved heterogeneity. Let
where $U_i$ varies across policyholders. Integrating over $U_i$ can produce overdispersion and motivate a negative-binomial model.
Another explanation is excess zeros, perhaps because some policies were inactive. A zero-inflated or hurdle structure may be relevant. A third is dependence from repeated observations on the same policyholder.
The appropriate revision depends on the DGP. “Use a negative-binomial model when variance exceeds the mean” is itself only a new template. Judgment asks why the variance is larger and whether the proposed model represents that reason.
Working with AI-Generated Solutions
Generative tools can now produce a formula, estimator, and code from a short description. This increases the value of model judgment.
Students should be asked to audit a generated solution:
- Identify the implied target.
- List the assumptions that were not stated.
- Check timing and information availability.
- Compare the loss function with the decision cost.
- Construct a counterexample.
- Revise the solution under one changed assumption.
The ability to recognize a polished but inappropriate answer is becoming more important than the ability to produce a familiar formula from memory.
Knowing When Not to Calculate
Model judgment sometimes leads to a number. At other times, it leads to a refusal.
Suppose a firm asks for the causal effect of training on employee productivity using a table containing training attendance and annual performance. A regression can always be computed:
But the coefficient $\widehat\tau$ is not automatically the training effect. If ambitious employees select into training and ambition is unobserved, then
Adding more controls or a more flexible learner does not guarantee identification. The responsible response may be:
- the available data support an association, not the requested causal claim;
- a valid estimate requires a stronger design or assumption;
- the decision can be reframed as a prediction problem only if that new target is useful; and
- data collection should be redesigned through randomization, a credible natural experiment, or measured selection variables.
Refusal is not anti-quantitative. It is a precise statement about the boundary between what can be computed and what can be learned.
The same principle applies to prediction. If a model must serve a population absent from training data, a point estimate of future performance may be false precision. A modeller can instead report conditional scenarios:
where $\mathcal Q$ is a set of plausible deployment distributions. The question becomes whether performance remains acceptable over that set, not whether one validation score looks convincing.
A mature assessment should therefore reward a well-defended “not identified,” “not transportable,” or “not decision-relevant.” Otherwise students learn that every prompt must terminate in a formula, even when the assumptions needed to interpret it are absent.
This distinction now extends well beyond the classroom. Value-Maxxing argues that AI performance must be evaluated against the decision value it creates, not the volume of output it produces. In education, Redesigning Education Beyond Procedure in the Age of AI reaches the corresponding conclusion: when machines can reproduce known procedures, students must be assessed on framing and defense. SIAI’s analysis of failed AI-detection policies reinforces the point by showing that policing finished text reveals less than examining the reasoning process that produced it.
Conclusion
Formula application is necessary. AI systems depend on exact computation, and a modeller who cannot manipulate the formal system is limited. But formula application is no longer a sufficient definition of quantitative competence.
Libraries provide formulas. Automated systems increasingly provide code. The educational task is to develop people who can determine what should be calculated, why the calculation answers the question, and when a changed assumption requires a changed solution.
Model judgment is visible when a student can:
- reconstruct the problem rather than recognize a template;
- explain which assumption supports which conclusion;
- distinguish prediction from explanation and decision;
- diagnose the source of failure; and
- redesign the model under a new environment.
That is the transition from using mathematics to thinking with models.
References
George E. P. Box, “Science and Statistics”, Journal of the American Statistical Association 71, no. 356 (1976): 791-799.
Leo Breiman, “Statistical Modeling: The Two Cultures”, Statistical Science 16, no. 3 (2001): 199-231.
Galit Shmueli, “To Explain or to Predict?”, Statistical Science 25, no. 3 (2010): 289-310.
The Economy (2025) ‘Redesigning Education Beyond Procedure in the Age of AI’, The Economy Review, 17 September.
The Economy Editorial Board (2026) ‘Value-Maxxing: The AI Metric That Puts Judgment Back in Charge’, The Economy Review, 14 May.
Swiss Institute of Artificial Intelligence (2026) ‘Why AI Detection Is Failing Higher Education’, SIAI AI Memo, 30 August.
Similar Post
