Why CI/CD Pipelines Are So Messy?

Welcome to the Issue #006 of the /dev/stdout newsletter. In this edition, I wonder why CI/CD pipelines are often so messy, even though there are ways to make them cleaner.

I have encountered—and admittedly created—my share of messy CI/CD pipelines throughout my career. Whether I have worked with a mature enterprise or an early-stage startup, the problem of messy pipelines often lurks in the corners.

I have worked with teams that have high standards in software development and who take pride in their quality of code. But even with those companies, as soon as you open the CI/CD pipeline YAML file, you’re often welcomed with a messy combination of YAML with varying levels of indentation and some wild bash or similar scripts there in between. An immediate bye-bye to readability and clean coding principles the very same team vouched for in the application side of their stack.

What’s worse, the logic there often tends to be deeply coupled with their choice of pipeline provider, be it any of the likes of GitHub Actions, Jenkins, or Azure DevOps Pipelines. Good luck trying to migrate to another pipeline service provider. 🫠 

Why is it like this?

Looking back to how I approached pipeline implementation work earlier in my career, was along the lines of—I need to finish this task quickly to get back to the development work, or I need to finish this task quickly to make deployments work. And then I went and googled for examples.

While it’s easy to blame the general busyness, and it’s often true, I think we as an industry have failed elsewhere when it comes to the quality of CI/CD pipeline code. We have failed to learn or at least teach others how to approach examples in library documentation or code snippets in any blog post. Those examples are often focused on making clear how to use the library, pipeline YAML config, or whatever the article is about. It’s usually not intended to take that example and use it in one’s production pipeline without thinking about the context first. Especially without thinking about the readability and, hence, maintainability of the implementation as a whole.

I’m not saying this does not happen elsewhere in software engineering. But what makes it more problematic with CI/CD, or generally in scripting, is the broader lack of solid coding practices. When a team develops applications there are more principles agreed upon regarding the architecture of the code base, that’s often not the case with scripts.

How it should be, in my opinion at least?

I think CI/CD pipelines are essential in the software development organization or team. As the software evolves, the pipeline must evolve, too. Therefore, it should be easy to change it and develop it further, and generally, it should be easy to understand what’s going on there, at least on a higher level, if not in detail.

That is not the case with the loooooooong YAML file with varying levels of indentation sprinkled with all kinds of docker commands and what have you.

I have written my share of bash to say that it’s not the programming language to blame here, it’s rather the conditions (time pressure, lack of code principles, etc.) in which the code was in the first place. While bash lacks a lot of constructs and features in, as some like to say, “real” programming languages, it’s still totally possible to write a clean and understandable script.

Making great pipelines is not limited to writing clear and idiomatic Bash , in fact, that only gets you so far. One might say, that changing a programming language helps, but while it can be the case it still does not magically improve the quality.

Reusability, task organization, and understandability of a great pipeline need a little different mindset.

• Think about whether it makes sense to write the same Docker commands in all the organization’s pipelines separately.

• Instead, consider if it makes sense to package some of the most common sets of commands as reusable tasks, rather than repeating all the related Docker commands over and over again.

Sounds reasonable, right? Heck, you could even have the different pipeline tasks tested too, in a proper TDD style, if you wish. This can be accomplished regardless of your choice of programming language.

Imagine having your pipeline tasks tested, packaged nicely, and distributed organization-wide, with no tight coupling to any specific pipeline service provider.

Thought so! Sounds valuable to me too. What’s best, it’s also totally doable.

If you want to delve more into the topic, take a read in one of my previous newsletters where I gave a few examples on how to achieve this.

Would you like to know more about how to refine CI/CD pipelines in your team or organization? You know what to do!

That’s it for now, I hope you enjoyed reading today.

Best,
Pyry

P.S. Writing this post was significantly boosted by this song. I saw them live a while ago and got sort of stuck with listening to them.

*You might wonder what the song recommendation has to do with anything. Well, as I often listen to music while I write, be it this newsletter, documentation, or code, I wanted to share some of that with you too. I hope you like them!