Concourse Lives!

Did you know that Concourse has full time developers working on it? And that development for it is supported by the Cloud Foundry Foundation, SAP, RamNode, and Pix4D? And that if you need commercial support you can reach out to the developers through their Concourse consulting company, Pixel Air IO?

If you worked on the Cloud Foundry project you likely have fond memories of Concourse. It produced beautiful visualizations of its workflows, and clean, versatile abstractions for things that most CI systems kind of kludge together.

You probably also think of it as a dead project. And for a while, it was pretty dead! VMware removed all of the staff maintaining it without, uh, telling anyone that they had done that, which gave the whole project a sort of cursed-ship vibe.

Anyway, the important thing is: You can use Concourse now!

I'm personally using Github Actions still but that's pure inertia. If I was starting StoryTime again today I'd set up a single instance Concourse using Docker Compose to run my tests and my Gigalixir push. And if and when I ever get off of Github, Concourse will be my first choice.

Should you, too, run Concourse?

Maybe!

If you already know and like Concourse, and you were only not using it because it was kind of a dead project, then you already know what it's relatively good at (complex pipelines testing large artifacts), what it's relatively bad at (PR-based workflows) and what it's basically overkill for (application CI where mostly what you're doing is testing an artifact in the MB range and then deploying it.)

If Concourse would be a good fit for your workload, you should use it. It's getting security patches, it has commercial support, if you need help upgrading it you can get it.

If you don't already know and love Concourse, I still recommend picking it up and playing around with it because it will change the way that you see CI systems forever.

Where Concourse Shines

There are a few kinds of workloads where there is absolutely nothing else that can do what Concourse does.

• Big blobs. Concourse was originally designed to deploy and test a distributed system with a total size of about 10GB for all of its various packages. If you need to download a gigantic artifact and do a bunch of different things to it, and you need to minimize the number of times you stream that artifact different places, you are going to love Concourse.
• Fan out, fan in. Do you, perhaps, need to deploy your gigantic blob to 10 different environments and run 10 different sets of tests? If you need to run a bunch of different tests on an artifact and only promote the artifact when all of those tests pass, Concourse just might be for you. Especially if each of those tests has multiple steps, and any of those steps might fail and need to be repaired and then restarted.
• Data workflows. Do you use Airflow? Do you hate Airflow? Do you in particular hate operating Airflow? And dealing with Python bullshit? Good news! Concourse will let you build ETLs and other data workflows (and give you fancy visualizations!) without any of Python's bullshit, and it does all of its internal state tracking with Postgres, as is right and just. It is
• When you need to look really really cool. Concourse's other big advantage is workflow independent: The visualizations it makes look really cool. Are you a consultancy that sets up these kinds of workloads for clients? Do you have clients come tour your office? In both cases Concourse will make you look extremely cool and extremely smart.

Still not sure if Concourse is right for you?

If you have a workload that you think might be a good fit for Concourse but you're not sure how to adopt it, send me an e-mail and I'll tell you what I think.