#cd pipelines

Pipelines are logical constructs in Continuous Delivery that promote a software release across the multiple CI/CD stages like build, test, deploy in a consistent, repeatable, and safe way. Spinnaker supports creation of pipeline-as-code using JSON.

To start with your deployments using Spinnaker CI/CD pipelines you need to have access to working Kubernetes clusters where you can deploy your application. Before configuring your Spinnaker pipeline create your application by logging into the Spinnaker UI.

The Application Management feature in Spinnaker helps you to configure and manage clusters/ server-groups, load balancers, firewalls, functions that are part of your application.

Breaching the siloed operations among Dev, Test, and Ops teams, the CI/CD pipelines are created for the Continuous Delivery of your software releases. Yet due to manual intervention at the end of each pipeline stage to analyze risks, diagnose and troubleshoot issues the pipeline continuity is broken, slowing down the deployment cycles.

Read our blog “Automate decision-making in CD pipelines using Autopilot” to know how the AI/ML-based OpsMx Autopilot, an automated verification solution adds a layer of intelligence in the existing CI/CD pipelines and provides the automated release verification through real-time decisions, governance and compliance of policies, and providing intelligent visibility to the DevOps teams.

Autopilot can be configured to automatically promote a release to the next stage with a high degree of confidence — removing the delay of human evaluation. A use case example for a test run stage cited here explores how Autopilot gathers data from different tools and analyses the risk thereby generating different errors and risk scores that help the DevOps teams to quickly decide whether to promote the release to the next stage or fail it. You can check for specific issues in the generated bubble graph by simply clicking a bubble, and also see the detailed log of that error. 

OpsMx Autopilot is an AI/ML-powered Continuous Verification platform that verifies software updates across different deployment stages using CI/CD pipelines, ensuring their safety and reliability in live/production environment. It automates new release verification, reducing time-consuming and error-prone manual verification processes. Autopilot uses AI and ML technologies to assess the risk of a new release, find the root-cause of issues and abnormalities for instantaneous diagnosis, and provide real-time visibility and insight about the performance and quality of new deployments to avoid business disruption.

Spinnaker is a Continuous Delivery (CD) platform that was developed at Netflix where they used it to perform a high number of deployments ( 8000+/day). Later they made it available as an open-source tool. Previously enterprise release cycles used to be stretched for 7/8 months. But with the availability of the Spinnaker CD tool, enterprises have been able to shorten the release cycles from months to weeks to days (even multiple releases a day).

There are several other CD tools available in the market but what made Spinnaker so special?

Spinnaker Features:

Multicloud Deployments

It includes support of deployment to multiple cloud environments like Kubernetes (K8s), OpenShift, AWS, Azure, GCP, and so on. It abstracts the cloud environment to be worked on and managed easily.

Automated releases

Spinnaker allows you to create and configure CD pipelines that can be triggered manually or by some events. Thus the entire release process is automated end-to-end,  

Safe Deployments

With a  high number of release deployments, it is hard to know if some unwanted or bad release has been deployed into production which otherwise should have been failed. The built-in rollback mechanisms with Spinnaker allow you to test and quickly rollback a deployment and lets the application go back to its earlier state.

Maintain Visibility & Control

This feature in Spinnaker allows you to monitor your application across different cloud providers without needing you to log in to multiple accounts.

So Spinnaker is a foundational platform for Continuous Delivery (CD) that can be quite easily extended to match your deployment requirements.

Overview of Spinnaker’s Application Management & Deployment Pipelines Functionality

Spinnaker supports application management. In the Spinnaker UI, an application is represented as an inventory of all the infrastructure resources – clusters/server-groups, load balancers, firewalls, functions (even serverless functions) that are part of your application.

You can manage the same application deployed to different environments like AWS, GCP, Kubernetes, and so on from the Spinnaker UI itself. Spinnaker supports access control for multiple accounts. For e.g. users like dev or testers with permission can deploy to Dev or Stage environments, where as only the Ops people get to deploy the application into production. You can view and manage the different aspects of the application – like scaling the application, view health of different Kubernetes pods that are running, and see the performance and output of those pods.

Spinnaker pipelines let you have all your application’s infrastructure up and running. You can define your deployment workflow and configure your pipeline-as-a-code (JSON). It enables github-style operations.

Spinnaker pipelines allow you to configure:

Execution options– flexibility to run fully automatically or have manual interventions

Automated triggers– the capability to trigger your workflows through Jenkins jobs, webhooks, etc

Parameters– ability to define parameter which can be also accessed dynamically during pipeline execution

Notifications– to notify stakeholders about the status of pipeline execution

As part of the pipeline, you can configure and create automated triggers. These triggers can be fired based on events like a code check-in to the github repository or a new image being published to a Docker repository. You can have them scheduled to run at frequent intervals. You can pass different parameters to your pipeline so that you can use the same pipeline to deploy to different stages just by varying the parameters. You can set up notifications for integrations with different channels like slack or email.

After configuring the setup you can add different stages each of which is responsible for doing a different set of actions like calling a Jenkins job,  deploying to Kubernetes, and so on.  All these stages are first-class objects or actions that are built-in and that allows you to build a pretty complex pipeline. Spinnaker allows you to extend these pipelines easily and also do release management.

Once you run the Spinnaker pipeline you can monitor the deployment progress. You can view and troubleshoot if somethings go wrong such as Jenkins build failure. After a successful build, the build number is passed and tagged to the build image which is then used in subsequent stages to deploy that image.

You can see the results of deployment like what yaml got deployed. Spinnaker adds a lot of extra annotations to the yaml code so that it can manage the resources. As mentioned earlier, you can check all aspects (status of the deployment, the health of infrastructure, traffic, etc) of the associated application resources from the UI.

So we can summarize that Spinnaker displays the inventory of your application i.e. it shows all the infrastructure behind that application and it has pipelines for you to deploy that application in a continuous fashion.

Problems with other CD tools

Each organization is at a different maturity level for their release cycles. Today’s fast-paced business environment may mandate some of them to push code checked-in by developers to be deployed to production in a matter of hours if not minutes. So the questions that developers or DevOps managers ask themselves are:

What if I want to choose what features to promote to the next stage?

What if I want to plan and schedule a release?

What if I want different stakeholders (product managers/QA leads) to sign off (approve) before I promote?

For all the above use cases, Spinnaker is an ideal CD tool of choice as it does not require lots of custom scripting to orchestrate all these tasks.  Although, there are many solutions in the marketplace that can orchestrate the business processes associated with the software delivery they lack interoperability- the ability to integrate with existing tools in the ecosystem.

Can I include the steps to deploy the software also in the same tool?

Can the same tool be used by the developers, release managers, operations teams to promote the release?

The cost of delivery is pretty high when you have broken releases. Without end-to-end integration of delivery stages, the deployment process often results in broken releases. For e.g. raising a Jira ticket for one stage, letting custom scripting be done for that stage, and passing on to the next stage in a similar fashion.

Use BOM (bill-of-materials) to define what gets released

Integrate with your existing approval process in the delivery pipeline

Do the actual task of delivering the software

Say, your release manager decides that from ten releases, release A and B  (i.e. components of software) will be released. Then it needs all the approvals ( from testers/DevOps/Project managers/release manager) to be integrated into the deployment process of these releases. And, all this can be achieved using a Spinnaker pipeline.

Example of a Spinnaker pipeline

The BOM application configuration ( example below) is managed in some source control repository. Once you make any change and commit, it triggers the pipeline that would deploy the version of the services. Under the hood, Spinnaker would read the file from a repository, and inject it into the pipeline, deploy the different versions of the services, validate the deployment and promote it to the next stage.  

Example of a BOM

A BOM can have a list of services that have been installed. You may not install all services in the release or promote all the services. So you will declare if the service is being released or not, and the version of the release or image that is going to be published. Here in this example, we are doing it with a Kubernetes application.  You can also input different parameters that are going to be part of the release e.g. release it in the US region only.

So the key features of this release process are:

Source Controlled

Versioned (Know what got released and when?)

Approved (Being gated makes the release items become the source of truth. Once it is merged with the main branch it’s ready to get deployed)

Auditable ( Being source-controlled, it will definitely have the audit history about who made the change, and what changes were made)

Some interesting ways to enforce approvals

Integrations with Jira, ServiceNow

Policy checks for release conformance

Manual Judgment

Approvals would include integrations with Jira, ServiceNow, policy checks for release conformance e.g. before you release any release items you need to have their SonarQube coverage for static analysis of code quality and security vulnerabilities at 80%. Finally, if you are not ready to automatically promoting the release to production you can make a manual judgment and promote the same

Spinnaker supports managing releases giving you control over what version of different services would get deployed and released. So all the versions need not have continuous delivery but planned release. It lets you plan releases, determine what releases would get promoted, and promote them through the whole process in an automated manner.

[LINUX ACADEMY] CERTIFIED JENKINS ENGINEER

14:14:10 Length  38 Videos Details

The purpose of this course is to prepare students to pass the Certified Jenkins Engineer 2018 exam. Student will learn CI/CD concepts as well as Jenkins installation and functionality. We will discuss Best practices for CD pipelines as well as taking a look at Jenkins Securtiy.

Training Architect Michael McClaren Size: 1.90G