Centralized Infrastructure

Centralizing Jenkins allows domain teams to focus on delivering more value to healthcare customers.

Background: Working for a major provider of healthcare software, information technology, and related services in the UK, we wanted to have a centralized deployment, and therefore view, of all builds across our organizational units. We were using AWS CodePipeline previously, and each team was having to learn how to spin up, deploy, and manage the infrastructure. We were lacking a holistic view of builds and deployments across an organizational reporting unit. Access to the information was linked to AWS accounts, and therefore control was outside of engineering to provide access. Each team was using different languages and tools, therefore the task was to provide a flexible solution that teams could use. We needed to provide Jenkins agents that teams could use, that had their language and frameworks pre-installed.

Goals: Centralize the Jenkins solution so that domain teams could focus on delivering feature value to our customers.

"Jenkins provides a centralized, well documented, well supported, well known (within the industry) solution for our engineers. Suddenly support can be gained from across the community, with great resources regularly being shared far and wide. Jenkins is cloud-agnostic, so the solution can morph as and when required into Hybrid Cloud configurations."

Ben Selby, Principal Engineer

Solution & Results: We built a dockerized Jenkins solution in AWS Elastic Container Service. We had an ECS cluster per reporting unit, so management and access could be granted at an Engineering Manager level. A nice side effect was that it also provided some cost reporting capabilities.

The solution was built in Terraform and made use of the Official Jenkins Docker images that are provided to the community. We then overlaid language/framework specifics to produce standard flavors of Agent, e.g. Go, Python, Node, Dotnet, etc.

We used SSO capabilities from AWS, but access could still be granted per Jenkins instance. We made use of the Jenkins Configuration-as-Code project to provide consistent Jenkins images to teams. At its peak, we were running thirteen Jenkins instances within ECS (backed by EC2 compute), all managed with Terraform and a little Go CLI tool we built in-house. The CLI tool made use of AWS and Jenkins APIs to provide us a neat little DevOps tool that helped us manage and monitor. Wholistic solution monitoring was delivered by Grafana.

And obviously, we dogfood-ed our solution by becoming the first team to onboard. We built the Jenkins solution, within Jenkins! When teams wanted changes, pull requests would be raised in GitHub, all automation checks would run, and our team would have the final say before it was merged ready for a managed release.

Key capabilities included:

Jenkins Configuration as Code

• All instances came from a standard jenkins.yml file that was then forked into a reporting unit instance. Teams were free to update and manage their base configuration.

Docker-based images from the Jenkins project.

• All the primary and agent images came from the Jenkins project, with additions where required.

Plugins:

• Amazon-ecs, Ansicolor, Build-monitor-plugin, Build-timeout, Configuration-as-code, Dependency-check-jenkins-plugin, Docker-slaves, Email-ext, HTML publisher, Lighthouse-report, Matrix-auth, Oic-auth, Pam-auth, Parameterized-scheduler, Role-strategy, Slack, and imestamper

Results included:

• A consistent CI/CD tooling experience for all teams.

• A greater visualization, not only per pipeline but holistically across reporting units.

• Teams are substantially quicker to onboard engineers into the product, since the community within our business, but also the wider Jenkins community is vibrant