Your favourite AI can compose a flawless sonnet, generate syntactically perfect ISO 8583 messages, and produce compilable C++ on the first attempt. Ask it whether that ISO message actually makes business sense, and you may get a confident, well-structured, beautifully formatted hallucination.

That asymmetry — syntactic excellence, semantic fragility — is not a bug that will be patched in the next release. It is a structural property of how these models work. Understanding it is the difference between using AI effectively and trusting output that looks right but isn’t.

The Syntax-Semantics Divide

Two terms that even experienced engineers conflate. Syntax asks: is this artefact well-formed according to the rules of its language? Semantics asks the harder question: given that it is well-formed, does it mean something valid in this context?

The distinction is universal. In natural language, “Colourless green ideas sleep furiously” is syntactically flawless English — Chomsky’s famous example — but semantically nonsensical. In code, int x = "hello"; may parse in some grammars but violates type rules. In payments, an ISO 8583 authorisation request can have every field correctly encoded in BCD and length-prefixed, yet carry an impossible combination of processing code and merchant category — syntactically perfect, semantically absurd.

Keep that payments example in mind. We’ll return to it.

Where Models Excel: The Syntax Engine

Large Language Models are statistical pattern machines trained on trillions of tokens. That architecture makes them extraordinary syntax engines. They internalise the distributional regularities of language, code, and structured data at a scale no human could match.

Grammar and natural language. Modern LLMs almost never produce ungrammatical English, Spanish, or Mandarin. Subject-verb agreement, tense consistency, pronoun resolution — these are solved problems for frontier models. The syntactic error rate in generated prose is vanishingly small, often lower than that of hurried human writers. This is not understanding; it is extremely refined pattern matching. But the results are indistinguishable in practice.

Code generation. Ask a model to scaffold a REST API in Python, Java, or Rust and the output will almost certainly compile or pass a linter on the first attempt. Bracket matching, indentation, import ordering, type annotations — the surface-level structure is handled with remarkable precision. Models have effectively memorised the formal grammars of dozens of programming languages.

Structured data. JSON, XML, YAML, Protocol Buffers, even ISO 8583 field layouts — models reproduce these structures faithfully. They know that a JSON object needs matching braces, that XML demands closing tags, that a bitmap in an ISO 8583 message is 64 bits representing which data elements follow. The form is rarely wrong.

Where Models Quietly Fail: The Semantics Gap

Syntax is necessary but never sufficient. A programme that compiles is not necessarily correct. A sentence that parses is not necessarily true. And this is precisely where the cracks appear.

Hallucination: Fluent Nonsense

The flagship semantic failure of LLMs is hallucination — generating statements that are syntactically perfect but factually wrong or logically incoherent. A model can write “The Treaty of Westphalia was signed in 1748 by Napoleon III” with the same confident cadence it uses for accurate history. The sentence is well-formed. It is also complete fiction. The model has no mechanism to verify truth against a grounded world model; it predicts the next plausible token, not the next true one. Ji et al.’s comprehensive survey of hallucination in NLG systems documents this across summarisation, dialogue, question answering, and translation — the problem is pervasive, not anecdotal.

Logical Consistency Under Pressure

Give a model a chain of logical constraints and ask it to maintain them across a long output and you will find the seams. In software architecture, this manifests as a model that produces a beautiful class diagram but introduces circular dependencies. In legal drafting, it writes clauses that individually look correct but collectively contradict each other. The local syntax is perfect; the global semantics are broken.

Domain Constraint Violations

This is the failure mode that matters most to engineers. Domain semantics are the rules that cannot be inferred from syntax alone — they require knowledge of the business, the physics, or the regulation. No amount of syntactic fluency can compensate for the absence of domain grounding.

The ISO 8583 Thought Experiment

Let me make this concrete with a domain I know well. Consider a model asked to generate a sample ISO 8583 authorisation request:

Every field is correctly formatted. The MTI is a valid four-digit code. The PAN passes Luhn. The processing code is a legitimate purchase indicator. The BCD encoding would be flawless. Syntactically, this message is impeccable.

Semantically, it is impossible.

Look at DE 22, DE 25, and DE 55 together. DE 22 says chip read — the card was physically inserted into a terminal. DE 55 contains EMV ICC data, confirming a chip interaction. But DE 25 says mail/phone order — a card-not-present transaction where no physical terminal is involved. You cannot simultaneously read a chip and conduct a mail-order transaction. Any payment processor’s validation engine would reject this message instantly. Any experienced payments engineer would catch it in seconds.

The model didn’t catch it because it doesn’t know what these fields mean in relation to each other. It knows what values are syntactically valid for each field independently. It does not understand the semantic contract between them — the domain invariant that says: if DE 22 indicates chip-read, then DE 25 cannot indicate card-not-present.

This is the syntax-semantics gap in action. Domain semantics are relational constraints that span multiple fields, layers, or concepts. Models excel at local correctness — each field in isolation — but struggle with global coherence — the fields together.

The Pattern Repeats Across Every Domain

The ISO 8583 example is specific to payments, but the structural problem is universal. Every domain with rich semantic constraints faces the same risk.

Medicine. A model generates a syntactically perfect prescription: drug name, dosage, frequency, route of administration — all individually valid. But the combination is a known lethal drug interaction. The form is right; the meaning could kill.

Legal contracts. Generated clauses that individually look correct but collectively create contradictory obligations. The indemnification clause in Section 4 conflicts with the liability cap in Section 7. Each section reads perfectly. Together they are unenforceable.

Infrastructure as code. A Terraform configuration that is syntactically valid HCL, passes terraform validate, and even terraform plan — but opens port 22 to 0.0.0.0/0 on a production database. The deployment tool sees correct syntax. The security team sees a catastrophe.

The common thread: syntactic validity provides no guarantee of semantic correctness. The gap exists in every domain where rules are relational, contextual, or normative rather than purely structural.

The Searle Parallel: Syntax Was Never Enough

Philosopher John Searle argued this point decades ago through his Chinese Room thought experiment. A person in a room follows rules to manipulate Chinese characters. They produce perfectly formed Chinese responses without understanding a single word. The room passes any syntactic test; it fails every semantic one.

LLMs are, in a very real sense, the most sophisticated Chinese Rooms ever built. They manipulate tokens according to learned statistical patterns with extraordinary fidelity. The patterns are so good that the output appears to understand. And often, for practical purposes, that appearance is sufficient. But when domain semantics demand genuine constraint satisfaction — when the relationship between fields, clauses, or concepts must be logically valid and not merely statistically plausible — the room’s walls become visible.

Bender and Koller formalised this intuition for the modern ML context in their ACL 2020 best paper: a system trained only on linguistic form cannot in principle learn meaning. The distributional signal in text encodes co-occurrence patterns, not the grounded relationships those patterns refer to. This doesn’t make the models useless — far from it — but it does explain why their failure mode is consistently semantic rather than syntactic.

Bridging the Gap: What We Can Do Today

This is not a counsel of despair. The syntax-semantics gap is real but manageable. Here is how experienced engineers are already working around it.

Structured validation layers. Use the model for generation, then pass output through domain-specific validators. In payments, that means running generated messages through your scheme’s validation engine. In code, that means static analysis, type checking, and property-based testing. The model drafts; the validator verifies.

Semantic guardrails in the prompt. Explicitly state domain invariants in the system prompt. “DE 25 must be consistent with DE 22” is a constraint the model can often respect when told — but will happily violate when not. This is the prompt engineering principle I’ve written about before: treat prompts as configuration contracts, not chat messages.

Human-in-the-loop for critical domains. Treat model output as a first draft, never a final artefact, in domains where semantic errors carry real consequences. The model drafts; the domain expert validates. This is the amplifier model — AI scales what you bring to the table, but you need to bring something to the table.

Retrieval-Augmented Generation (RAG). Ground the model in authoritative domain documentation. If the ISO 8583 specification is in the retrieval index, the model is far less likely to produce impossible field combinations. RAG doesn’t eliminate semantic errors, but it narrows the gap substantially by giving the model access to the constraints it would otherwise lack.

Fine-tuning on domain corpora. Expose the model to thousands of validated, semantically correct transactions (or contracts, or prescriptions) so it absorbs domain constraints statistically, even if it never truly “understands” them. This shifts the probability distribution toward correctness without guaranteeing it.

Will the Gap Close?

Frontier models are improving at semantic tasks. Chain-of-thought reasoning, tool use, and long-context architectures are pushing the boundary. But there are structural reasons to believe a gap will persist.

Statistical plausibility is not logical necessity. Training data encodes what was, not what must be. Domain constraints are normative, not descriptive — the ISO 8583 spec defines what shall be valid, not just what has historically appeared in message logs. A model trained on text cannot reliably distinguish between the two.

Grounding remains absent. What a chip reader actually does, what a drug interaction actually causes, what an open port actually exposes — these are facts about the physical world that text-only training cannot capture. Multimodal and tool-augmented systems are beginning to address this, but the gap between reading about something and knowing what it does is not trivial to close.

Edge cases are adversarial. The long tail of domain semantics — the unusual field combinations, the regulatory exceptions, the rarely-triggered invariants — is precisely where models are weakest. These cases are underrepresented in training data and overrepresented in production failures.

Models will get better at semantics. But syntactic fluency will likely remain ahead of semantic reliability for the foreseeable future. That asymmetry has profound implications for how we architect systems that incorporate AI.

The Engineer’s Takeaway

Trust the syntax. Verify the semantics. Always.

AI models are the most powerful syntactic engines humanity has ever built. They produce well-formed text, code, and structured data with a fluency that is genuinely impressive. But fluency is not understanding. Form is not meaning. A perfectly formatted ISO 8583 message that violates domain invariants is not a valid transaction — it is a beautifully dressed lie.

As engineers, our job has always been to ensure that systems are not just well-formed but correct. In the age of AI, that responsibility doesn’t diminish. It sharpens. The model handles the syntax so you can focus on what it cannot yet do reliably: ensuring that the output means what it should.

That is not a limitation to lament. It is a division of labour to embrace.

References

• Searle, J. R. (1980). “Minds, brains, and programs.” Behavioral and Brain Sciences, 3(3), 417-424. cambridge.org
• Bender, E. M., & Koller, A. (2020). “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.” Proceedings of ACL 2020, 5185-5198. aclanthology.org/2020.acl-main.463
• Ji, Z., Lee, N., Frieske, R., et al. (2023). “Survey of Hallucination in Natural Language Generation.” ACM Computing Surveys, 55, Article 248. arxiv.org/abs/2202.03629
• Chomsky, N. (1957). Syntactic Structures. Mouton & Co. — the “Colourless green ideas sleep furiously” example
• ISO 8583 — Financial transaction card originated messages — Interchange message specifications
• Point-of-Sale Systems Architecture — Volume 1: A Practical Guide to Secure, Certifiable POS Systems — broader context for ISO 8583 and EMV in production systems
• The Obsolescence Paradox: Why the Best Engineers Will Thrive in the AI Era — engineering perspective on AI adoption
• AI as an Amplifier, Not a Replacement — related post on why domain expertise is the multiplier
• Prompt Engineering for POS — companion post on treating LLM inputs as architecture