VWO's Pipeline for A/B Testing

If there was ever life support for engineers, it would be Jenkins for us.

Background: There were many challenges to this transformation.

• Training/Single Point of Contact: We wanted one place to create builds and deploy those builds to servers. This requires training people and Jenkins had amazing community support.

• Security: Since Jenkins has been used for years by a team of varying sizes, the security of our build pipeline was of utmost importance. We did not want to deal with software that is not maintained/updated regularly. This is a serious concern since forking the software and fixing it takes time.

• Cost: Jenkins was open-source and completely free for us. The only cost that we bear right now is our hardware costs to run Jenkins.

• Integration with Infrastructure: Integration with Github/Bitbucket Server is of primary importance to us.

• Programming Language Support: Node.js support is great with Jenkins and we could freely do a lot of things. Maintain multiple node.js versions, publish different packages through Jenkins builds.

• Hiring Talent: This is of utmost importance for our business. Jenkins is one of the most widely used CI/CD.

Goals:  Enable A/B Testing on any platform/hardware

"Jenkins has helped see us through tough times and good times. It has supported us for years without ever asking for anything in return. One of the best companions ever that you should always have."

Gaurav Nanda, Associate Director, consultant to VWO

Solution & Results:  We set up Jenkins along with an in-house Node.js bot to allow triggering builds remotely. This allowed us to separate how the build was done and create a wrapper around the process that anyone could use.

It also allowed us multiple benefits. One was the ability to abstract the process. The second was that we need not provide Jenkins access to everyone in the team. They could see the input and input via the bot on Slack.

We integrated Jenkins with Bitbucket Server Integration, allowing us to leverage webhooks and trigger builds from within Bitbucket. We were able to see the build status from within Bitbucket and if there were any red flags, they were marked by Danger.js.

Jenkins allows us to control releases, do reverts without ever needing to maintain it time over time.

Jenkins ability to store credentials, create freestyle projects, create pipelines which we later figured out was a great way to version the build changes and anybody could contribute easily. Jenkins Bitbucket server integration was a major factor in further strengthening the visibility around builds.

Results? They couldn’t be better.

• Zero sprint effort to release features

• Manual shipping 2-3 times a day transformed to shipping more than 100 times in a day

• Time to fix bugs reduced by 5x