AI is not software development

Software is necessary to implement AI, but correct code does not establish that an AI system is valid.
AI systems inherit uncertainty from data, model choice, deployment conditions, and feedback from their own decisions.
AI education must teach students to evaluate the entire model-based decision system, not merely its code.

A Necessary Distinction

Artificial intelligence is implemented in software. It does not follow that AI is a branch of software development.

Traditional software transforms inputs according to rules written by a developer. If a tax calculator is given the same inputs and the same rules, it should return the same result. Its central quality question is whether the implementation conforms to its specification.

An AI system is different. Part of its behavior is inferred from data. Its effective specification is distributed across a dataset, a model class, an objective function, an optimization procedure, a deployment environment, and a decision rule. Even flawless code can implement a poor statistical model, learn from an unrepresentative sample, optimize the wrong target, or produce decisions that alter the future data it receives.

Let a conventional program be written as

where $r$ is an explicit set of rules. An AI system is better represented as

where $\mathcal{D}$ is training data, $\mathcal{M}$ a model class, $\ell$ a loss function, $\lambda$ a collection of regularization or tuning choices, and $s$ the stochastic state of training.

The program executes $\mathcal A$, but the behavior of $\widehat f$ is not written line by line. It is learned under assumptions about how $\mathcal D$ relates to the environment in which the model will operate.

Correct Code and Correct Inference

The difference can be summarized through two forms of correctness.

Table 1. The layers of a reliable AI system

Dimension	Conventional software	AI system
Primary specification	Explicit rules and interfaces	Data, model, objective, rules, and deployment policy
Main correctness question	Does the code implement the rule?	Does the learned system support the intended inference or decision?
Expected output	Determined by programmed logic	Estimated from data and often stochastic
Main testing object	Functions, states, and interfaces	Code, data, model behavior, uncertainty, and distribution shift
Typical failure	Bug or violated software requirement	Bug, biased sample, misspecification, poor calibration, drift, or feedback
Maintenance trigger	Code or requirement changes	Code, data, population, incentives, policy, or user behavior changes

Software testing remains essential. An incorrectly indexed tensor or a leakage-prone preprocessing step can invalidate a model. But unit tests answer only whether the implementation behaves as designed. They cannot determine whether the design itself represents the relevant world.

Suppose a medical classifier is internally correct and achieves high validation accuracy. It can still fail if its training sample contains patients from one hospital, its target label reflects a local diagnostic practice, or its deployment changes which patients receive further testing. None of these is necessarily a coding error.

The central AI question is therefore not:

Does the program run?

It is:

Under which data-generating conditions does this program produce a decision we should trust?

The Five Layers of AI Failure

A useful educational framework separates system loss into five layers:

This is a diagnostic decomposition rather than a universal additive identity.

Read as a diagnostic rather than a second scorecard, the framework becomes clear: Code — Central question: Was the intended computation implemented correctly?; Example failure: Training and inference apply different transformations. Data — Central question: Do observations measure and represent the intended population?; Example failure: A label is a proxy generated by unequal access to service. Model — Central question: Does the model capture the relationship relevant to deployment?; Example failure: A stable correlation is mistaken for an invariant mechanism. Decision — Central question: Does the prediction support the correct action?; Example failure: Accuracy is optimized despite asymmetric costs. Feedback — Central question: Does using the model change its future data?; Example failure: A recommendation system makes displayed products appear preferable.

The distinction explains why adding developers cannot by itself solve an AI problem. More programmers may reduce $\mathcal L_{\mathrm{code}}$. They do not automatically reduce selection bias, identify a causal effect, select an appropriate loss function, or anticipate equilibrium responses to deployment.

Conversely, a mathematically sophisticated model is not sufficient. A model that cannot be implemented, monitored, secured, and maintained is not a useful AI system. The lesson is not that software engineering is unimportant. It is that AI requires software engineering plus several additional disciplines.

Data Are Executable Assumptions

In ordinary software, code determines behavior. In machine learning, data partly determine behavior. A change in data can function like a change in source code.

Let the training and deployment risks be

Empirical training makes $R_{\mathrm{train}}$ observable. Institutional value depends on $R_{\mathrm{deploy}}$. The two coincide only under assumptions connecting $P_{\mathrm{train}}$ and $P_{\mathrm{deploy}}$.

This is why a data pipeline needs more than engineering reliability. It also needs semantic control:

• What event causes a row to exist?
• Which people or events are absent?
• Did the meaning of a variable change?
• Is a feature available at the time of decision?
• Was the label produced independently of the policy the model will replace?
• Will deployment change the future sample?

These are questions about the data-generating process. They cannot be answered by inspecting syntax alone.

Prediction Is Not Yet a Decision

AI systems are usually built to change an action. The relevant objective therefore includes the cost of decisions, not merely predictive error.

For a binary decision with threshold $\tau$, define

If false positives cost $c_{\mathrm{FP}}$ and false negatives cost $c_{\mathrm{FN}}$, the threshold should solve

No software library can choose $\tau^{*}$ without institutional judgment about costs. A fraud model, a medical triage model, and a content-moderation model may use the same algorithm while requiring very different decision thresholds.

The model’s probability must also be calibrated. If events assigned probability $0.8$ occur only half of the time, a ranking may remain useful while a resource-allocation policy becomes dangerous. Evaluation must match the action.

AI Systems Interact with the World

Deployment can invalidate the data on which a model was trained.

Suppose a hiring model predicts performance and determines who receives an interview. Future performance is then observed only for selected applicants. If $S=1$ denotes selection, the next training sample comes from

not from the original applicant population $P(X,Y)$. The model influences $S$, and $S$ influences which labels become visible. Retraining on this apparently fresh data can reinforce the original selection rule.

This feedback loop has no exact analogue in a static calculator. It is a model-governance problem, an experimental-design problem, and sometimes a causal-inference problem.

The engineering literature reaches a related conclusion from the system side. Sculley and colleagues’ paper on hidden technical debt in machine-learning systems emphasizes that live ML systems accumulate dependencies through data, configuration, feedback, and changing external conditions. The learned model is only a small component of the deployed system.

What AI Education Must Add

An AI curriculum organized around software tools can produce students who know how to call an interface but cannot determine whether its output should be used.

Expressed as a practical comparison rather than another table, the distinctions are clear: Software and systems — Capability: Implement and maintain reliable computation; Evidence of mastery: Tests, versioning, interfaces, monitoring. Mathematics and optimization — Capability: Understand representation and calculation; Evidence of mastery: Derivation, conditioning, convergence analysis. Statistics and econometrics — Capability: Connect samples to populations and effects; Evidence of mastery: Uncertainty, identification, sampling, validation. Domain knowledge — Capability: Judge whether variables and mechanisms are meaningful; Evidence of mastery: Defensible assumptions and relevant constraints. Decision analysis — Capability: Connect prediction to action and welfare; Evidence of mastery: Costs, thresholds, utility, scenario analysis. Governance — Capability: Define accountability and limits; Evidence of mastery: Documentation, escalation, monitoring, withdrawal rules.

Students should learn to move in both directions. They must translate a model into reliable implementation and translate an institutional problem into a model whose assumptions can be examined.

That is why an AI course cannot be evaluated solely by whether students produce working code. A notebook that runs may demonstrate implementation. It does not demonstrate model judgment.

The AI System as a Decision Contract

A useful way to govern an AI system is to treat deployment as a contract between four objects:

where $\mathcal P$ is the population for which the model is validated, $\mathcal I$ the information available at decision time, $\mathcal A$ the permitted actions, and $\mathcal G$ the governance rules for monitoring and intervention.

The contract prevents a model from silently expanding beyond its evidence. A credit model validated for salaried applicants should not automatically be applied to new businesses. A triage model trained on adult patients should not silently become a pediatric model. A prediction produced with information recorded after an event should not become a real-time decision rule.

Software interfaces can enforce parts of $\mathfrak C$. Access controls can restrict consumers, schemas can require variables, and monitoring can detect range violations. But the content of the contract comes from statistical and institutional reasoning.

The model should be reconsidered when any component changes:

This is more demanding than conventional version control. The code may remain unchanged while the decision contract has materially changed.

Applied Example: Demand Forecasting

Suppose a retailer deploys a model that forecasts daily demand. The software team verifies the data interface, model service, and inventory-system connection. Historical tests show low mean absolute error.

Three non-software failures remain possible.

First, promotional prices may be entered after the forecasting cutoff in the historical database. Validation then uses information unavailable at deployment. This is a DGP and timing failure.

Second, the model may minimize average error while the institution incurs most of its cost from stockouts of a small set of critical products. This is an objective and decision failure.

Third, managers may begin ordering according to the forecast. Observed sales then become censored by inventory decisions: a product cannot record demand above the quantity made available. Future labels are partly created by the model’s own policy.

The observed sales process becomes

where inventory $I_t$ depends on the forecast. Retraining on $Y_t^{\mathrm{obs}}$ without modelling censoring can teach the system that constrained sales equal unconstrained demand.

Every component may be implemented exactly as specified. The AI system is still wrong because the specification no longer matches the process it influences.

Current SIAI research gives this distinction an organizational consequence. From AI Access to Organizational Capability shows that licenses and model access create little value without redesigned workflows, while The AI Premium Is About Implementation, Not Access finds that serious, integrated use carries a stronger economic signal than superficial experimentation. The same principle appears at the research frontier: agentic systems perform most reliably when verification is built into the task. Software quality is necessary, but the validity of the model, objective, evidence, and decision remains a separate responsibility.

Conclusion

AI is not software development, but it cannot succeed without software development. The relationship is one of inclusion rather than identity.

Software engineering makes a computational system reliable relative to its specification. AI/data science must also justify the specification, the sample, the model, the decision criterion, and the conditions under which deployment remains valid.

The distinction matters for organizations. Calling every AI problem a software problem encourages investment in infrastructure while neglecting model validity. It also matters for education. Students trained only to implement existing algorithms become dependent on libraries and benchmarks. Students trained to examine the entire system can decide when an algorithm is appropriate, when its output is misleading, and when the problem must be reformulated.

The objective is not to produce fewer capable engineers. It is to produce AI professionals who understand that code is one layer of a much larger claim.

References

D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, Advances in Neural Information Processing Systems 28 (2015).
Margaret Mitchell et al., “Model Cards for Model Reporting”, Proceedings of FAT* (2019).
Timnit Gebru et al., “Datasheets for Datasets”, Communications of the ACM 64, no. 12 (2021): 86-92.
Swiss Institute of Artificial Intelligence (2026) ‘From AI Access to Organizational Capability: Pricing the Corporate AI Transition’, SIAI Working Papers, 9 August.
Swiss Institute of Artificial Intelligence (2026) ‘The AI Premium Is About Implementation, Not Access’, SIAI AI Memo, 11 August.
Swiss Institute of Artificial Intelligence (2026) ‘What Agentic AI Can Prove and What It Still Fabricates: The Real Frontier of Machine Science’, SIAI Science Review, 15 August.