Why Do Enterprises Grow from Jenkins?

Jenkins® is arguably one of the most popular development tools on the planet, with more than 297,000 active installations as of June 2021 and a growing community of millions of users.

If you’re a small, agile development team and you need to integrate code multiple times a day, Jenkins is great. Some reports estimate that up to 71% of all continuous integration pipelines run on it. However, as teams, environments, projects and market pressures increase, some of the maintenance that comes with building and using Jenkins pipelines can quickly get out of hand. For example, you might lose sight of how many teams are on each instance, how many controllers are in use, who has made changes to code and scripts or how many scripts are in use—keeping track of these administrative tasks can be a real challenge.

These growing pains often arise as an organization reaches enterprise-level maturity. At this point, the organization looks to scale its Jenkins usage across the company without affecting the security or stability of its software development lifecycle (SDLC). In working with many of the Fortune 100 enterprises, we’ve found a few common ways that enterprises determine they’ve reached the limits of what Jenkins might provide on its own.

In this series, we’ll walk through these reasons for scaling Jenkins and explain how to recognize them in your own teams, from losing track of plugins to losing control of security and credential management. Let’s start with one of the most common motivations for growth, though—a fragile Jenkins instance.

Jenkins Training | Jenkins Online Training Course & Certification

Avoiding Upgrades and Changes Due to a Fragile Jenkins Instance

When you’re a small team, it doesn’t matter so much that modules need to be configured and installed manually. It’s easier to do it yourself than it is to figure out a way to optimize the process. But as your organization (and your Jenkins environment) grows over time, your needs will evolve and you’ll find yourself constrained by your architecture. You may worry about your plugin versions staying compatible, so you hesitate to upgrade. You start delaying important upgrades and changes, but that only makes things worse or delays the inevitable.

At some point, probably sooner than later, you will need to scale your Jenkins requirements. As a growing organization, you’re also inherently at risk for losing the institutional knowledge that’s essential to maintain your pipelines. It makes sense to start automating certain processes—but if your Jenkins instance is fragile, rearchitecting is an even bigger task than it would be in ideal circumstances. 

A fragile Jenkins instance is often due to the competing needs of an overused and poorly architected controller. Teams depend on a single controller, and the load on it increases as more jobs are configured and more builds are orchestrated more frequently. It’s common for Jenkins usage and benefits to spread through an organization and have user after user ask to be onboarded to the same Jenkins instance. (This is called vertical growth.) The end result is a Jenkins instance in use by multiple teams with conflicting plugin needs and maintenance schedules.

Alternatively, teams may choose horizontal growth when they realize they need to scale. They spin up new Jenkins controllers for different environments, departments or product lines. But this approach can have its drawbacks, too. Teams may not have the resources for regular plugin updates, core upgrades and backups across multiple controllers—not to mention the manual configuration of security settings and roles for each one. This approach may also make it more difficult to create pipelines.

Strengthening Your Jenkins Environment

To scale your Jenkins environment, you need to strengthen it. One of the best ways to do that is to put your Jenkins controller on a Kubernetes cluster. Jenkins already has features for scaling out-of-the-box, but containerizing Jenkins on Kubernetes can help you work smarter. When you scale Jenkins on Kubernetes, Jenkins can automatically remove corrupted builds or agents and spin up new, healthy ones. You can run builds in parallel, so you don’t have to plan and limit executors. The balanced load distribution that Kubernetes offers can also improve your builds’ performance.

When you want to set up Jenkins on Kubernetes, you’ll need to install the Kubernetes plugin on each controller. At scale, however, just like any other plugin, management gets cumbersome. Many teams have multiple Kubernetes clusters for different development environments, and they need a way to provision and manage controllers across clusters.

CloudBees CI allows users to run a “controller of controllers” on a main Kubernetes cluster. It creates a common operations center that manages controllers across other Kubernetes clusters. You can host these clusters on all the major cloud providers, and mix and match providers without losing central visibility and control.

Of course, it’s easy to accumulate too many controllers for the resources you have. CloudBees CI can also help you optimize the resources you use by setting your controllers to hibernate when idle.

To achieve an ideal balance of freedom, security and manageability, many teams choose to assign one controller to each team. While it’s important to closely monitor the financial implications of the compute resources you’ll require, this team-based approach boosts the performance and speed of your build jobs and reduces cross-team friction when managing plugin versions and agents’ needs. Bonus—a “one-controller-per-team” model limits the blast radius when something goes wrong.

Moving to an Enterprise-Grade Jenkins Environment

Enterprises have unique needs when it comes to CI/CD. We’ll discuss more of these in future installments of this blog series, but one of the most critical is the ability to scale CI across multiple teams without wreaking havoc on the Jenkins environment. CloudBees CI runs on Jenkins, and helps teams streamline and automate processes that were once manual. CloudBees CI adds enterprise functionality to Jenkins so you can strengthen as you scale, helping to embed best practices, rapid onboarding, security and compliance into your Jenkins environment.