Most Integration Failures Are Stupid

There's this claim going around that AI programming is going to kill "traditional CI" and transform it into something novel and complicated and based on... gamedev programming? I guess?

The logic is something like this: Thanks to AI there will be far more commits. At a certain point, the commits will arrive so frequently that CI isn't able to finish running before the next one arrives, and then you will be in never-ending queue hell. And then you talk to the people who are having this problem and it turns out that their build (including tests) takes like 10 or 20 or 30 minutes to run, and I have to go lie down on the floor somewhere for a minute because it's 2026 and we don't have to live like this.

Let's set aside for a moment the question of, "Why is the build so fucking slow?" In the vast majority of cases it's slower than it needs to be, but sometimes the build is just slow. I have worked on software where the full end-to-end build and test pipeline for final artifacts took several hours to run, because to be absolutely sure that the final artifact was ready to release we had to (among other things)

Set up a bunch of VMs
Build a ~10GB artifact and stream it onto those VMs
Start a bunch of processes on each of those VMs
Build a different ~10GB artifact and stream it onto those VMs
Shut down and restart all of those VMs, a process that took a couple of minutes on AWS
Run a series of end-to-end tests that simulated multi-step workflows by exercising the real CLI and a lot of those tests had to take 30+ seconds because we were testing a container application platform and that's how long it takes to stream and start up an application, and the tests could be run somewhat in parallel but there were limits to how many requests a test system could process simultaneously okay?

And we did this on AWS, GCP, Azure, and vSphere, and we did it for a "clean install" scenario as well as the upgrade, and we also ran a version that wasn't connected to the internet so we could verify that the whole thing could work in an air-gapped datacenter.

On that final build process we did something a lot like what Steve Yegge is describing, where we batched a ton of commits into each test run, and then hand-diagnosed anything that failed. (Though we did not "roll forward on red" because we were fucking civilized and we understood how git revert works, and because we had a reasonably well-architected system that was split into services and if something broke you could usually tell roughly where the problem was just by eyeballing the problem.) But this was not the first fucking time we tested those commits.

Most Integration Failures Are Stupid

The dark secret of integration testing is that the vast, vast majority of problems that an integration environment will catch don't have anything to do with integration. If I bundle commits A and B and C together into a test run and that test run fails, the first thing I'm going to check is not "is there some exotic failure mode that only happens with these three commits together?" it's "so were A and B and C tested individually at all before they got pushed to the shared pipeline?"

Friends: The commits were not tested.

Even when you have developers who are very diligent and write tests and carefully run those tests before committing there will be blind spots. For a while, the artifact I'm describing above was then taken and deployed with a bunch of other artifacts in a complex pipeline designed to check compatibility between 3-4 versions each of a large suite of interacting products. Very big, very expensive pipeline. Do you know what it mostly caught?

That's right: It mostly caught problems where the other products (not our beautiful, beautiful artifact) would fail entirely to deploy on a clean install, and could only be deployed as an upgrade. The teams building these products were unusually diligent about testing. They ran a lot of tests, both before and after committing. But their own internal CI pipelines often just rolled forward continuously and they didn't consistently test that their artifact could build and deploy on a fresh system. So the big, expensive integration pipeline mostly caught something that could have been caught cheaply and easily upstream.

The Good News: You Can Buy More Than One Computer

Perhaps I can illustrate with this incredibly terrible diagram.

By the time we get to the big pipeline that integrates everything and takes multiple hours to run, our heroic commit has been run through

local unit tests
the same unit tests in CI again (just in case)
component-specific acceptance tests
shared across-the-whole system acceptance tests
deploy and upgrade in one set of configurations
shared across-the-whole system acceptance tests again (just in case)

By the time it hits that big, slow, expensive pipeline we can be pretty confident that it works at least some of the time.

Always Be Pushing Upstream

The system at this stage got built up over a period of several years, mostly by pushing more and more of the testing "upstream" towards individual component teams. When I first joined this organization the process looked more like this

That shared environment was broken all the time. When it broke it would send a message to an app named "Can I Bump?" and the video screen in the office running "Can I Bump?" would turn red, and the app would send a Slack message to every team that was committing to this thing to notify them that the build was broken and that they were not to push until it was fixed.

I was on the team responsible for that shared server at the time, so at this point my pair and I would go over to the team that had pushed the commit that broke it and ask them, "Did you run the shared acceptance tests yourself before you pushed?" and they would say "No we did not run the shared acceptance tests ourselves before we pushed" and we would say "Okay well it broke the build" and they would say "Oh" and then we would tell them that we were reverting their change and that they were welcome to push it again once they had figured out why it broke the acceptance tests and fixed whatever it was.

So, we built a bunch of tooling to make it easy for every team to build their own CI environment, so those tests would always be run automatically before it even made it to the commits-from-every-team-crashing-into-each-other environment, and we could stop having that conversation.

The Conditions Where This Is Necessary are Weird

A lot of this goes away if you can make the build fast. There many techniques.

Isolate tests and test setup, so they can be run asynchronously
Reduce test dependencies (database, other systems especially) to make them easier to isolate
Build incrementally
Architect your system so it's easier to reduce test dependencies and build incrementally
Cache
Buy Moar Computer

The vast majority of projects that I have ever worked on had builds that were slower than they needed to be and tests that were slower than they needed to be. This was obnoxious but also kind of understandable, because doing this stuff takes a certain amount of experience and a certain amount of discipline and was annoying to retrofit if the people before you made certain bad decisions.

Now however we live in a blessed age where there is a Magic Box and you can type into the Box something like "I would like to run all my tests in parallel please do it" and the idiot robot that lives inside the box will go through and wrap all your tests in database transactions or fix the way you're instantiating your Spring Beans or rewrite every test in your Slow AF Playwright Suite that can possibly be written as a test that just hits the API directly instead or otherwise fix whatever other fucked up thing is forcing your tests to be slow.

Every time I have personally seen software that really need a 30-minute plus pipeline it has been software that was deployed and installed on someone else's computer. Databases, container orchestration systems, that kind of thing. When you're deploying to someone else's computer you have to test your packaging and configuration and deploy system and there just isn't a good way to do that other than actually package and deploy for real, and this takes time. I also believe it when game programmers say that their builds take a long time because they've got all those pixels.

But most software isn't that. Most vibe coded software isn't that. Most software is like, a web server and a database and a bunch of code that folds up the data before it puts it into the database. This is a well-understood problem. We have gotten good at making the builds for this kind of program very fast.

Most Companies are Pretty Bad at Continuous Integration

While writing this I discovered that I actually agree with the core of Steve Yegge's claim, which is that the way that most companies do continuous integration will not work if they start dealing with vastly more commits. I just don't see this as a radical departure from good CI practice. Yeah, if your idea of CI is "we have exactly one build server and it's the first time any of the tests are run" or "most work lives on a branch for several months before we even think about merging it back into main."

Most companies that think they're doing continuous integration, uh, aren't.

But the trunk-based development world has been dealing with the problem of "wow so many commit" for a long time. There's prior art!