Jenkins is the way to bring our community together
NEBULA: New Build Infrastructure for All
Deploying timetables, station, and local stop info, and track services in realtime using CI/CD & Jenkins.
Background: As a recently merged group — one spanning multiple technology stacks, countries, and cultures — we were nonetheless simultaneously challenged to undergo a DevSecOps transformation. However, coming from a place where we built code in different ways and had different DevSecOps maturity levels to being with, working together efficiently was always going to be the first barrier to overcome.
Goals: To create a collaborative, shared, modern build infrastructure to bring together a group newly formed from disparate parts of the company.
"The flexibility of Jenkins as an automation tool has made it a one-stop-shop for building, deploying, monitoring, testing and even self-managing itself through running its own helm charts to upgrade itself :)."
Solution & Results: To overcome our various barriers to making progress together — rather than separately — we identified the need to come up with a new approach to our build infrastructure that would not only be an example of as-code provisioning but also would be built by a community for a community, to spread the cultural changes we wanted to see our group move towards.
As our group had some common experience using Jenkins before, we identified that a Kubernetes-based Jenkins-as-code approach, using public Jenkins Helm Charts — as well as other Kube friendly wrappers for tooling such as Selenium, Sonarqube, etc. — would allow us to build a stable, repeatable infrastructure which we could maintain and evolve centrally. This would also allow individual teams to spin up their own instances (using Helm inheritance patterns) with minimal customization, but complete freedom to build on top of the base definitions.
Running all of this across multiple PKS-provided Kube clusters (to provide Teams their own isolated workspaces), we were able to make further use of Kube native resources such as Secrets, — in combination with the fantastic Kubernetes plugin for Jenkins — to provide a far more robust way for handling credentials. This was an area that some of our teams had previously struggled with, leading to fragility in their own environments.
The proof of this overall approach came up recently when a lab outage forced us to completely relocate to another data center where we had many teams back up and running in days, a restoration that was driven by the community itself.
Our base plugin setup is as follows: kubernetes, workflow-aggregator, git, configuration-as-code, timestamper, matrix-auth, CloudBees-bitbucket-branch-source, ssh-credentials, basic-branch-build-strategies, ws-cleanup, logstash, embeddable-build-status, greenballs, blueocean, ansicolor, gitlab-branch-source, ldap, and prometheus. Also, we couldn’t have done all of this without the official Jenkins Community Kubernetes Helm Charts.
The results were phenomenal:
Shared build pipelines are consistent and evolve faster
Credential management has gotten much simpler & stricter
Jenkins-as-code has freed teams up to experiment more freely
Teams are more self-empowered to provision and support their own builds