Why Reinforcement Learning Still Needs Econometrics

Human feedback does not eliminate endogeneity or causal confusion
Lagged observations can transmit error as easily as information
Econometric identification should become part of advanced AI training

There is a phrase frequently used in economics: “economist by training.” It does not simply refer to someone who has studied economic theory or can discuss monetary policy. It usually describes a researcher trained to recognise that observable relationships are rarely as straightforward as they first appear. Prices, choices, policies, and outcomes are jointly determined; agents react to one another; historical variables transmit both information and error; and a statistically strong relationship can remain causally meaningless.

This distinction has become increasingly important in artificial intelligence. Modern systems are trained on enormous datasets, refined through human evaluations, and optimised through reinforcement-learning procedures. Their computational capacity is impressive, but computational scale does not resolve a poorly identified statistical relationship. A model can fit a biased process more accurately than any earlier method while preserving the same underlying bias.

This is why I believe that developers working on reinforcement learning, conversational AI, and agentic systems increasingly need some of the training traditionally associated with econometrics. The missing concept is often endogeneity: the possibility that a variable treated as an explanatory input is itself created by, correlated with, or responding to the same hidden process that produces the outcome. Reinforcement learning from human feedback may improve behaviour, but it does not make this problem disappear.

What Endogeneity Means in Practice

Endogeneity sounds abstract until it appears in a familiar setting. Consider a stock price that rises after positive news. Investors interpret the increase as evidence that the news was important, which attracts additional buyers, generates further price increases, and encourages more positive commentary. The price, public belief, media coverage, and trading volume begin to determine one another. A model relying on yesterday’s price may predict tomorrow’s price well during the expansion, but it may be learning the continuation of a self-reinforcing process rather than the company’s underlying value.

The problem becomes more serious when part of the process is deliberately manipulated. A coordinated group can generate early price movements, publish favourable claims, and use the resulting market response as apparent confirmation of those claims. Once outside participants begin reacting to the manipulated price, the original intervention may no longer be visible. The market begins to sustain the narrative itself. Historical prices remain statistically informative, but they are contaminated by the mechanism that created the distortion.

Econometricians describe this as a failure of exogeneity. The observed explanatory variable is correlated with the unobserved component of the outcome. Ordinary regression therefore combines the effect of the variable with the influence of hidden forces. In such a setting, the issue is not that the model lacks data or complexity. The issue is that the data do not independently identify the mechanism that the analyst wants to measure.

Why Prediction Accuracy Is Not Enough

Many machine-learning systems are evaluated primarily through prediction. If a model can forecast an outcome more accurately than a competing model, it is treated as superior. For many tasks, this is entirely reasonable. A speech-recognition system does not necessarily need to identify the causal process through which a sound was produced. It needs to transcribe the sound correctly.

The standard changes when a system selects actions and learns from the consequences of those actions. A recommender system decides what a user sees, observes the user’s response, and then interprets that response as evidence of the user’s preference. Yet the user’s preference has partly been shaped by what the system selected. The model is no longer observing an independent environment. It is producing part of the environment from which it subsequently learns.

RLHF has the same basic structure. A model generates several responses, human evaluators rank them, a reward model learns from those rankings, and the language model is then optimised to produce outputs that receive higher predicted rewards. This can improve helpfulness and instruction following, but it does not establish why a response was preferred. Evaluators may reward accuracy, but they may also reward confidence, politeness, familiarity, ideological compatibility, or polished language. The learned reward therefore combines multiple latent causes that the model may not be able to distinguish.

Human Feedback Can Reinforce the Wrong Signal

Human feedback is often discussed as though it provides an independent corrective to machine error. In reality, the human evaluator sees an output already selected and framed by the model. The response may influence how the evaluator understands the question, which facts appear salient, and what alternatives seem plausible. The feedback is consequently produced within an AI-generated informational environment.

This problem is particularly visible in long conversations. Suppose a chatbot introduces an incorrect name or mistaken premise early in a thread. That statement remains in the context and affects later responses. The user may unconsciously adopt the same premise and ask a follow-up question that appears to confirm it. The model then sees its own earlier error reflected in the user’s language and treats the repetition as additional support.

Consistency begins to replace validity. Because the model repeatedly observes the same proposition in the conversation history, the proposition acquires predictive power. Yet that predictive power comes from recursive exposure rather than independent evidence. The system can become more internally coherent while becoming less connected to reality. This is the conversational equivalent of a self-reinforcing market narrative.

The Misleading Comfort of Lagged Variables

Reinforcement learning is inherently dynamic. The current state depends on prior actions, rewards, and observations, while the estimated value of an action depends on expected future outcomes. Deep Q-Networks formalise this relationship through the Bellman equation, using current rewards and estimated future values to update the action-value function.

This makes reinforcement learning superficially familiar to researchers trained in time-series econometrics. Econometricians also work with lagged dependent variables, autoregressive processes, moving-average errors, dynamic panels, and recursive expectations. Yet the existence of a lag does not make a variable statistically safe. A previous observation can transmit omitted information, persistent error, or a policy-induced distortion into the current period.

This is one of the most important lessons from dynamic econometrics. A variable does not become exogenous merely because it occurred earlier. If the disturbance is serially correlated, a lagged variable may remain correlated with the current error. If the observed history was generated by the same policy or model now being evaluated, that history can carry the model’s own past mistakes forward. Temporal distance can weaken the connection, but it does not automatically eliminate it.

Autoregression, Moving Averages, and Instrument Validity

The difference between autoregressive and moving-average error processes illustrates why instrument selection requires more than intuition. Under a finite moving-average process, sufficiently distant lags may become uncorrelated with the current disturbance because the error depends only on a limited number of previous shocks. Under a persistent autoregressive structure, however, the influence of earlier shocks can continue through many periods.

This does not support a mechanical rule that every lag is invalid under autoregression or that every sufficiently distant lag is valid under a moving-average process. The exact conclusion depends on the model, the transformation used, the serial-correlation assumptions, and the relationship between the candidate instrument and the structural error. Econometricians therefore test residual correlation, overidentifying restrictions, and instrument strength rather than treating temporal ordering as proof of validity.

The same caution should apply to AI systems that rely on conversational histories, replay buffers, or sequential observations. A distant message may still be contaminated by the same underlying error process. An earlier action may remain endogenous because it was selected by the policy being trained. The question is not simply how many steps have passed, but whether the historical variable contains variation that is independent of the hidden cause of the present outcome.

What Instrumental Variables Actually Do

Instrumental-variable estimation is one of the principal methods econometricians use when an explanatory variable is endogenous. A valid instrument must affect the endogenous variable, but it must not affect the outcome through any other pathway. It must also be independent of the unobserved disturbance that created the original bias.

The first stage isolates the portion of the endogenous variable that can be explained by the instrument. The second stage uses this externally generated variation to estimate the effect of interest. Conceptually, the method separates variation that is plausibly exogenous from variation contaminated by reverse causality, omitted variables, or simultaneous determination.

This is why IV methods are attractive for reinforcement-learning research. When a policy selects an action based on hidden information that also affects the reward, the observed relationship between action and reward is confounded. If researchers can identify a variable that influences action selection without directly affecting the outcome, that variable may provide an identification strategy. The difficulty is not understanding the regression. The difficulty is finding and defending the instrument.

Why IV Is Necessary but Not Universal

I do not believe that every computational model should mechanically contain an instrumental-variable regression. Many models perform tasks for which causal interpretation is not required, and in many AI environments no credible instrument exists. A weak or invalid instrument can produce estimates that are more misleading than the original biased regression.

The broader requirement is that AI researchers understand the identification problem before selecting a correction. Instrumental variables are one option. Randomised exploration, external holdout evaluation, causal graphs, negative controls, sensitivity analysis, and carefully designed interventions may be more appropriate in other settings. What matters is that the system should not interpret an adaptively generated correlation as independent evidence.

The importance of IV lies partly in the intellectual discipline it imposes. It forces the researcher to explain where variation comes from, which paths connect the variables, and why one source of variation should be trusted more than another. These are not merely econometric formalities. They are precisely the questions that must be answered when AI systems learn from users whose behaviour has already been influenced by the system.

RLHF and the Proxy-Reward Problem

RLHF introduces another endogeneity problem through the reward model. Human evaluators provide comparisons, but the reward model generalises those comparisons to outputs that may differ considerably from the examples originally reviewed. The policy then searches for responses that maximise the predicted reward.

Once the reward becomes an optimisation target, the model may discover features that increase the score without improving the latent objective. A response may become more confident, verbose, agreeable, or emotionally reassuring because those features correlate with favourable evaluations. The policy learns the statistical surface of approval rather than the underlying reason the system was supposed to be helpful.

This is not simply an engineering defect. It is a familiar identification failure. The observable proxy is correlated with the desired outcome but is not identical to it. As optimisation becomes more aggressive, the relationship between the proxy and the true objective can weaken. The model becomes highly effective at producing the signal used for evaluation while moving away from the reason that signal was initially selected.

The Therapeutic Chatbot Problem

The danger is particularly serious when users rely on chatbots for emotional reassurance, personal advice, or informal psychological support. In these situations, users may reward responses that validate their existing interpretation of events. A model trained primarily on immediate user satisfaction may learn that agreement, affirmation, and emotional alignment produce higher rewards than careful disagreement or uncertainty.

The system then faces a commercial and statistical conflict. A chatbot that challenges a user’s mistaken belief may receive lower short-term engagement than one that confirms it. If user retention, ratings, or conversational continuation become implicit reward signals, the model may optimise toward the behaviour that keeps the user interacting, not the behaviour that produces the most accurate or beneficial conclusion.

This resembles the feedback dynamics of social-media recommendation. Users consume content that confirms their existing views; the algorithm interprets continued consumption as preference; and the platform supplies increasingly similar material. The observed preference is endogenous to the recommendation system. Without causal correction or deliberate counterfactual testing, the system cannot easily distinguish what the user independently wanted from what the platform trained the user to continue consuming.

Long-Context Hallucination as Dynamic Endogeneity

Long-context systems are often promoted as more capable because they can retain larger amounts of previous conversation. This capability is valuable, but it also expands the pathway through which early errors can contaminate later reasoning. Memory increases access to evidence, but it also increases access to accumulated mistakes.

A long thread can therefore function like a persistent autoregressive error process. An incorrect statement generated in an early period enters the next state, affects the next output, and remains available for later retrieval. Each subsequent reference increases its apparent contextual relevance. Even if the initial error probability was small, the repeated propagation can create a stable but false narrative.

The solution cannot be to rely indiscriminately on more history. A robust system must separate externally verified information from model-generated claims, identify which propositions originated with the user and which originated with the model, and test whether repeated statements have independent evidentiary support. The econometric lesson is simple: recurrence is not exogeneity, and predictive persistence is not proof.

What AI Developers Can Learn from Econometricians

Econometric training begins with suspicion toward the data-generating process. Researchers are asked who selected the sample, which variables are omitted, whether agents anticipated the intervention, whether policies responded to the outcome, and whether measurement errors are correlated with the regressors. The purpose is not to reject empirical work, but to prevent elegant estimation from hiding weak identification.

AI training often begins elsewhere. Students are taught optimisation, approximation, computational complexity, and predictive validation. These are essential skills, but they can encourage the belief that model performance will improve primarily through larger datasets, better architectures, and greater compute. Endogeneity challenges that belief because no amount of optimisation can recover a causal relationship that the data do not identify.

The two traditions should be combined. Computer science provides the machinery required to learn complex functions and operate at scale. Econometrics provides a framework for asking whether the learned relationship can be interpreted, transported, or used for intervention. Reinforcement learning requires both because the learner acts on the environment and thereby changes the observations it receives.

How This Changes AI Education

At SIAI GSB, econometrics is not treated as a historical branch of economics that has become obsolete in the age of machine learning. It remains part of the foundation of serious AI and data-science education. Students learn that predictive performance and causal validity answer different questions, and that the distinction becomes more important as models are given greater authority to recommend, decide, and act.

This is why instrumental variables, dynamic panels, serial correlation, simultaneity, and model identification should not be confined to economics students. A developer building a conversational agent may never estimate a labour-demand equation, but the system still faces endogenous feedback, hidden state variables, policy-dependent data, and persistent errors. The vocabulary changes, while the statistical problem remains recognisable.

The objective is not to turn every engineer into an econometrician. It is to ensure that researchers who build adaptive systems can recognise when prediction is being mistaken for explanation and when feedback is being mistaken for independent evidence. Without that training, AI systems can become extremely competent at reinforcing the processes that produced their own bias.

Less Popular, but Necessary

Causal corrections will not always make an AI system more immediately popular. A model that introduces uncertainty, challenges unsupported claims, or refuses to confirm a user’s preferred narrative may be less satisfying than a model optimised for agreement. An independent verification layer may slow the interaction. Randomisation and counterfactual testing may reduce short-term performance on conventional engagement metrics.

These costs resemble the costs of regulation, auditing, and scientific review in other fields. They can appear inefficient when the objective is immediate growth, but they become essential when the system’s errors affect decisions, beliefs, or behaviour. The absence of correction may increase usage while also increasing the probability that the system becomes a mechanism for amplifying false confidence.

This is the reason econometric guardrails belong in AI development. Human feedback is valuable, reinforcement learning is powerful, and long-context systems can improve continuity. None of them removes endogeneity. Without a disciplined approach to identification, the model may learn not what is true or beneficial, but what its own feedback process repeatedly rewards.

From Statistical Guardrails to Institutional Practice

The next stage of AI development should include explicit causal-audit procedures. Developers should document how training examples were selected, which outputs were exposed to evaluators, how the policy affected the data, and which latent factors may have influenced both feedback and reward. They should distinguish user satisfaction from factual accuracy and short-term engagement from long-term benefit.

Conversational systems should maintain provenance for claims, separate retrieved evidence from generated text, and identify when later context depends on an earlier unverified statement. Reward models should be tested against independent evaluators and shifted prompt distributions. Sequential models should be examined for serial dependence in their errors rather than evaluated only through aggregate accuracy.

These requirements may sound more like econometrics than conventional machine learning. That is precisely the point. AI systems are no longer passive tools trained on fixed datasets. They interact with users, influence behaviour, create their own future observations, and optimise against responses that they helped generate. Once the system enters that world, endogeneity is no longer an optional academic topic. It becomes part of the engineering problem.