Jenkins is the way to truly automate everything!
Automating Ml Models with Jenkins
A lesson MlOps: integrating machine learning and DevOps with Jenkins.
Background: _ The LinuxWorld India training program allows students to work on live projects provided by the organization. In this way, they get real exposure to the IT industry. LinuxWorld offers training in the city of Jaipur, with a team of highly experienced and expert trainers. The students can enroll in programs like DevOps, Docker, Splunk, Cloud Computing, Big Data Hadoop, RedHat certified programs, Python, Splunk, python during their training program._
The project for my training at Linux World was to automate machine learning using agile DevOps techniques. The problem statement was that whenever we run an ML model, CNN model, or DL model, it needs our oversight. For example, if we got a 60% accuracy in the first instance, our laptop had to be on during that entire time. Then, if the accuracy is not so great, we would have to continuously change the layers and do some manual tweaking, which can be very time-consuming. We would have to test repeatedly until it crosses 90% to ensure it was a great model.
Goals: With the help of Jenkins and GitHub, automate the model training and accuracy automatically and continuously so that it makes new changes in the NN layers to achieve accuracy of more than 90%.
"Jenkins helped in achieving my automation work of continuously tweaking the Ml model until it reaches 90% accuracy through scripts."
Solution & Results: My project solved this problem statement using various tools: with one click, the entire model will run automatically and help tweak and change the layers until we reach the target accuracy of 90% automatically. Once it achieves the goal, an automatic mail will be triggered with an alert that the model has successfully achieved 90% accuracy.
The full-flow connection here was challenging: it needed to connect the git triggers to jobs and autorun code until it auto-identified it in Redhat Linux, which runs the model.
Development can be tedious and expensive, considering the complexity of BI processes using machine learning. It is hard to choose a small instance once these machine learning processes require big machines to finish a job. We faced a challenge in waking up multiple instances, processing concurrent jobs, and finding the right way to turn off these machines, thus preventing the bleeding in cost.
Jenkins is an excellent tool for DevOps, especially for making CI/CD pipeline and assembly lines and automation part of this feature. I used my Ml part to tune the model automatically, but Jenkins will help trigger the jobs with the script I wrote using Groovy. With this solution, I don’t have to manually check if my model is accurate. After deploying this method, everything runs just once, starting from triggers, jobs, and the tweaking of the pattern — which can be done easily. That’s the power of using Jenkins.
Here are the key capabilities I used:
Github plugin
Scriptwriting using groovy
Jobs
Jenkins password system
Auto triggers system
Auto build
Jenkins pipeline plugin
Auto mail sending plugin
Jenkins assembly lines system
And I got the results I was hoping for:
Faster build times and triggers
Less human interference; more automatic work
Excellent speed of achieving 90% accuracy from this for Ml models
End-to-end automatic with just one click
Easily transferable project