Why AI Doesn't Replace Real Engineering

AI is just a giant probabilistic calculator, nothing more, nothing less

Target Audience - Hiring Managers, Tech Leads, CEOs/CAIOs, Software Engineers, Recent Graduates

The Training Distribution "Problem"

Models are pretrained on the same information. All of them.

Here is the simplest way I can describe the distribution "problem". Imagine you took all the division and manipulation built into the modern web and then fed it into ALL the models during pretraining. Models that are built to find patterns in language to better understand the semantic meaning of things. Imagine all the mind viruses (memes, ala Dawkins.) are now in ChatGPT, Claude, and other models. You don't have to imagine it, it's obvious to anyone who uses them.

But it doesn't stop at harmful human behavior - it greatly affect the ability for models to "code" as well. When Microsoft bought GitHub, the play was clear - get training data for models by buying the world's largest repository of code. It seemed like a great idea at the time by whoever was in charge, but it was also extremely harmful and (in my mind) unethical.

People learning to code in new languages write a ton of slop - slop being defined as code that might technically work, but is not in any way "idiomatic" and fails to consider any of the lessons we have learned in the last 75 years of software development. Half finish projects with horrible code, and very little engineering involved.

So if 80% of the code on github was garbage, and you fed it into a model, what would you get out of a distribution? That's right, garbage. Just because a manager looks at code and says, this looks ok to me, it doesn't mean Rich Hickey, Ken Thompson, Kent Beck, or Robert Martin would agree, in fact, they would rate all this code in pretraining as hot piles of burning trash.

The VAST majority of code on Github is trash. The best code is from the top 5% of programmers in the world. 50% of the code on Github will be far worse than the top 80%. If you know much about distributions you will see how this is painfully obvious. There are normal distributions and pareto distributions. The former tells you where most people cluster, but the latter tells you where most of the value comes from.

So when models write shitty code and have no concept of what clean, well engineered code looks like - chalk it up to a distribution problem. GOOD CODE is gatekept, it's compiled, obfuscated, and otherwise made unavailable, and it isn't shouted out to the world - it is worth too much. Sure there are open source projects that are awesomely coded, but they are in a tiny minority of what the models saw.

You can hire an average programmer for $50 an hour, but a great programmer can cost $300/hr. Which person's code do you think you will find more of in public?

Models Don't Actually "Know" Anything

Next we come to a simple fact that very few experts, save Demis Hassabis, Yan LeCun, and AI fatalists like Gary Marcus, want to admit. Models do not know anything. They are just guessing at the best answers based on their training distributions, which I have already laid out are fraught with ick.

These models learn what words mean, and how the meaning related to other words and grammar, but they have no experiential knowledge. They can't tell you what it feels like to ride a horse across a beautiful landscape, they can only spit out descriptions from their training data. They cannot feel fear (one of the most primitive evolutionary triggers), they cannot feel love (arguably a huge part of the human existence), and they have no idea what human connection really is.

Neither does your calculator, and models are much closer to a scientific calculator than they are to any form of real human experience. Babies of all species don't learn by language. They learn by observing. Even multi-modal models don't really "see" anything - they are a vision tower tacked onto an existing language based system. They see edges, contrast, corners, roundness, etc - they don't "see" that screenshot you just sent them.

Models are in fact a lot like Jon Snow. To them, they are the northerners while the Freefolk clearly see them as southerners. They don't know anything from an epistemological experience, they are just pattern matching.

Models Are Bad At Understanding Software Engineering

All the RLHF, DPO, DPRO, etc isn't going to solve the weights created in pretraining. The models get "better" but never "great", and for the last several months, models like Claude have seriously regressed. The training data from github (slop code) simply outweighs all the books on good code. Also, how do you explain taste to the model so they know Uncle Bob made his money consulting enterprise Java projects and will likely suggest you need more code, and more patterns, whereas Rick Hickey would argue that you need better primitives and a better form, using less code, and less encapsulation?

A real engineer will have read dozens or more books, and then spent YEARS trying the techniques out in the real world, and coming to their own conclusions. You too, would never learn to build a good wooden sailing vessel from reading books, you HAVE to sail it to know and to adjust and to learn. Models are simply too big and expensive to do this in any real, measurable way.

Models slop out code, because that is what they were trained to do. They might have seen "composition over inheritance" but they won't do it on their own. Models can write in a hundred languages, but never make the connections about how LISP can inform your coding in C# (or C++ or Python).

Models Are Horrible At Operational Security (OPSEC)

Here is where things go REALLY wrong. Because they don't actually "know" anything, they utterly fail at operational security - even with all the finetuning and "alignment" the foundation labs have thrown at the problem. What you get is a useless model that won't do real work, and then leak all your keys, because again, they just don't know anything (Jon Snow).

If you want some examples, just do some google searches at how many times these leaks have happened. Even Anthropic leaked their own map files for Claude Code, telling us all kinds of shady stuff Anthropic has been doing. It also leaked the existence of Mythos before Anthropic wanted it out. Clickup has had horrible breaches, leaking all kinds of customer data to anyone who knows how to look. You know, Engineers.

On Hiring Senior Engineers And Keeping Them

So if you can generate thousands of lines of code in a day, why should you keep your most expensive developers on staff?

As a company you have spent millions of dollars building your IP. You have spent millions building your customer base. You have spent millions avoiding expensive legal affairs. Letting models "do what they do best" will erase all of that. Your code will devolve into a slopfest. Your customers will experience production bugs daily and get angry (Anthropic anyone?), and they will leak customer data, destroy production databases and more, opening you up to all kinds of legal trouble you fought hard for decades to avoid.

A good senior engineer has seen it all. They have worked for shady startups that did shady things (Facebook was famous for its manipulation tactics) - they have seen proper enterprise systems built by talented programmers from the previous generations. They have made mistakes themselves and learned from them, something modern models simply cannot do!

When the CAIO wants to have engineers writing 100% of their code using models, the Sr. Engineer will say no. They will push back. And they are 100% correct in doing so. They aren't just protecting their own jobs, they are protecting ALL the jobs at the company and the company itself.

AI is good at some things and horrible at most others - a good Sr. Engineer will know what these are and apply the technology where it fits, and ban it from where it will cause problems... That's the job description.

Tech Companies Are Drowning In Their Own Koolaid

There is a very real bubble forming, and it’s not subtle. The executives see demos, not systems. They see a model spit out a React app or a Python script and assume they’ve just eliminated 70% of their engineering costs. What they don’t see is everything that doesn’t show up in a demo: edge cases, long-term maintenance, scaling constraints, security posture, data integrity, and the thousand invisible decisions that separate a toy from a production system.

This is classic hype-cycle behavior. The Gartner Hype Cycle has played out the same way for decades—early breakthroughs, inflated expectations, and then a very painful correction when reality shows up. Right now, a lot of companies are sprinting straight into that wall. No one is hiring developers and at the same time posting jobs for just the free training data. But this will all change in the next 6 months to a year.

Internally, dissent gets filtered out and sycophants get a loudspeaker. Engineers who push back get labeled as “resistant” or “not forward thinking.” Meanwhile, leadership doubles down because nobody wants to be the one who missed AI. So what happens? You get fragile systems, skyrocketing technical debt, and teams quietly spending more time fixing AI-generated problems than they ever saved generating them.

It’s not that the tech is useless—it’s that it’s being wildly misapplied by people who don’t understand its failure modes or how to use real engineering to turn stochastic systems into deterministic ones.

Solutions? It's Really Not That Hard...</p>

<img src="/images/blog/why-ai-aikoolaid.png" alt="Tech companies AI koolaid" />

Solutions? It's Really Not That Hard...

You don’t need to ban AI. You need to treat it like what it is: a powerful but unreliable assistant.

First, constrain its role. Use it for scaffolding, boilerplate, exploration, and maybe even test generation—but keep it away from core architecture, security-critical paths, and anything that requires long-term maintainability. The closer the code is to your business’s actual value, the less it should be touched by a probabilistic system trained on unknown data.

Second, raise the bar on review. AI-generated code should be treated as untrusted input, no different than code from a junior developer you’ve never met. That means stricter code reviews, better test coverage, and real ownership by experienced engineers who understand the system holistically.

Third, invest in your top engineers, not less. The entire dynamic described by the Pareto principle applies here more than ever—a small number of highly skilled people will determine whether AI is a force multiplier or a liability. If you lose them, no amount of model output will save you.

Finally, be honest about what models are good at. They are incredible for compression of knowledge and speed of iteration, but terrible at judgment, taste, and accountability. Those last three are the entire job of a real engineer.

If you align your usage with reality instead of hype, AI becomes useful. If you don’t, it becomes expensive noise that slowly erodes everything you built.