#aws qualification

What is TestOps? How does it help you continuously qualify AWS/Azure?

"TestOps is a software validation culture and practice which aims at shorter testing and validation cycles to continuously qualify any cloud IaaS/PaaS service, in close alignment with business and regulatory objectives. "

Many life-science companies are asking the question “Can I move my GxP workloads into the public cloud?” Answering “yes” to this question leads to another question: “What toolset do I need to qualify the various IaaS/PaaS services that are constantly changing?”.

Our customers are leveraging our robust framework that we call TestOps. TestOps provides an end to end automation for continuously qualifying any IaaS/PaaS service running on AWS or Azure.

TestOps - How does it work?

"We test "continuously" while you operate your cloud infrastructure!"

In order to illustrate our TestOps framework, a simple use case to continuously qualify AWS DynamoDB is presented. Keep in mind that this framework can be applied to any IaaS/PaaS service or a combination of services.

Step 1 - Define the Intended Use

An IaaS or PaaS service can be configured to be used in many ways to meet varying end user needs. It is important to document the intended use to cover the various scenarios the service will be utilized in. This intended use forms the basis for designing a qualification strategy.

Step 2 - Design the Test Cases in the Coding Framework

Conventional wisdom directs us to develop test cases outside of coding using Microsoft Word as the authoring tool. This approach is surely outdated and does not provide “hard” traceability to your code. xLM’s TestOps supports “Given-When-Then” framework wherein the validation analyst can build test cases within the coding environment (IDE). These test cases are directly linked to the corresponding test automation code by the developers. In other words, both validation analysts and developers are working within the same environment and test case definitions are not separate from the code.

Step 3 - Coding the Test Cases

Once the test case design is complete, the developers code the testing scenarios which typically involves driving the browser for UI Testing or leveraging the APIs. The code that is developed is organized into reusable snippets (features, pages, utilities, etc.) for better maintainability of the test automation model. As mentioned above, the traceability between test case design and code is built into the framework.

Once the code development, code reviews, testing and quality checks are complete, the code is checked into the Code Factory.

Step 4: Build the TestOps Pipeline

Once the code is published, the TestOps Pipeline is configured:

to retrieve the published code

to deploy the qualified code into the test environment

to run the test automation code to qualify the SUT (System Under Test - in this case AWS DyanmoDB)

to generate a test automation report aka “executed protocol”

to archive all the test run artifacts

Step 5: Run the TestOps Pipeline

Once the Pipeline is fully configured and released by Quality, the Pipeline can be run either manually or automatically (scheduled or event based). The Pipeline run-time steps include:

Obtaining the Pre-approval to run the “qualified” Pipeline

Running the test automation code and generating the execution report

Step 6: Review the Results

After the Pipeline run is successful, the final step is to review the test automation results to ensure all the acceptance criteria have been met. This review is documented with the Post-deployment approval.

Conclusion

xLM’s TestOps Framework is designed to continuously qualify any IaaS/PaaS service (AWS, Azure, Google Cloud) cost efficiently utilizing test automation and pipelines. Such a framework can enable you to build a Service Catalog of “qualified” IaaS/PaaS services so that your development teams can consume these services on an on-demand basis. This model completely removes the qualification hurdle each time a project is deployed in the cloud.

This framework utilizes the following key six (6) steps:

Development of the Intended Use that is traceable to all qualification models

Development of the test cases using the Given-When-Then framework that is directly linked to the test automation code

Development of the Test Automation Models.

Configuring the Pipeline

Running the Pipeline

"AWS makes "infrastructure-as-code" a reality.  My blog post will make "continuous-qualification" an absolute reality. "

The million dollar question is "Can AWS be qualified?".  In this blog post I will provide a prescriptive guide on not just "how to qualify", but also how to maintain the qualified state intelligently.   

As we all know, initial qualification of IT Infrastructure is hard but pales in comparison to the effort involved in maintaining the qualified state (QS).   The model presented here ensures QS Integrity so that requalification can be avoided at all costs.   

Step-by-Step Guide

AWS CloudFormation Template Qualification

Step 1:  Transform your infrastructure requirements into code.  This can be done by using AWS CloudFormation.  Using CloudFormation you create a template for your infrastructure.  Then, test your template and hand it over to IT Quality.  

Step 2: Build a "continuous" qualification framework to qualify your template.  This framework can automatically deploy the template in a TEST environment and perform various tests to ensure all the requirements are met.  Also, the test execution reports with all evidence are automatically generated.   IT Quality reviews the results and then certifies the CloudFormation template.  

Step 3: Publish the qualified CloudFormation template to the Service Catalog.   Your global teams can deploy this template any number of times without worrying about qualification (Qualify it once and use it many times!).

AWS Monitoring Toolset Qualification

In order to maintain your AWS Infrastructure in a Qualified State, you need to create a "qualified" catalog of AWS Monitoring Services.   Such services include AWS Config, AWS CloudTrail, AWS Systems Manager, Amazon GuardDuty, Amazon Inspector, etc.

Step 4: Build a "continuous" qualification framework to qualify each monitoring service.  This framework can automatically qualify each service in a TEST environment and perform various tests to ensure all the requirements are met.  IT Quality reviews the results and then releases the service for Production Use.  

Step 5: Now each of the qualified monitoring services can be used globally without worrying about qualification.

Build Qualified AWS Infrastructure and Go Live

Step 6: Now your IT teams can build "qualified" infrastructure using CloudFormation templates from the Service Catalog.  Also, they can monitor them using "qualified" services in order to maintain the "qualified state" of the infrastructure.

Conclusion

By following the above six (6) steps, you can build and maintain a qualified infrastructure that is GxP compliant and always "audit ready".    The above framework uses the mantra "qualify it once and use it many times", thus not only making it cost efficient but also baking in the best practices.  

Frequently Asked Questions (FAQ)

AWS releases changes constantly.  How will I maintain change control? A true "cloud" is designed to constantly release changes so that the end customer can leverage these innovations and thus increase productivity.  AWS is no exception here.  However, this presents a dilemma for the traditional validation folks who are used to reviewing each change and then addressing it one way or the other.   If you want to embrace the Cloud, the compliance perspective must change from examining every change by the Cloud Provider (which is practically impossible considering the velocity of changes) to ensuring your requirements are met constantly.  This can be achieved with the "continuous" validation framework where by you are constantly (for example:  daily) testing to ensure your requirements are met in spite of the changes.    

How does "continuous" qualification really work? Continuous Qualification is GxP compliant and based on a sophisticated Model Based Testing Framework.   Once the Continuous Qualification model is built, it can be used to perform initial qualification of an infrastructure template or a monitoring service, for example.  It can be run at regular intervals to "continuously qualify" with no human intervention ("lights out mode").   This approach enables cost effective testing on a continuous basis thus enabling deployment of GxP workloads in the public cloud.  

What are the real advantages of qualifying an Infrastructure Template? Before the advent of "software defined" data center, the only way to qualify was to build first and then perform the qualification.  Now with AWS Cloud, building infrastructure is akin to writing code.  In other words, you script the infrastructure you want to build and then run it as many times as you want to consistently and within minutes stand up your virtual data center.   Now, this makes qualification cost effective and implementing IT best practices that much easier.  Amazon does provide downloadable templates that you can start with and then modify them to meet your requirements.  For example, you can stand up a complex environment for SAP implementation within minutes using CloudFormation templates.    You build the template using best practices (for regulatory, security compliance, etc..) that meet your specific requirements.  You then qualify the template and make it available in the Service Catalog for consumption throughout your organization.  

Why do we need the monitoring tools after the template is qualified? Once the infrastructure is built and deployed, you need to monitor its health and also ensure that the Qualified State (QS) drift has not occurred.   QS drift can occur when changes to the deployed infrastructure are made (either intentional or unintentional) after it is qualified by bypassing the change control process.   AWS Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations.  This service will enable compliance with the Qualified State (QS).  It will bring to your notice if a QS Drift has occurred. AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure.  This service provides the audit trail transactions that can be used by other AWS Services (e.g. AWS Config, AWS SNS). AWS Systems Manager gives you visibility and control of your infrastructure on AWS. With Systems Manager you can view operational data from multiple AWS services and allows you to automate operational tasks across your AWS resources.  It helps you easily understand and control the current state of your resource groups and view detailed system configurations, operating system patch levels, software installations, application configurations, and other details about your environment. Amazon GuardDuty is a managed threat detection service that continuously monitors for malicious or unauthorized behavior to help you protect your AWS accounts and GxP workloads.  It monitors for activity such as unusual API calls or potentially unauthorized deployments that indicate a possible account compromise thus violating your Qualified State. Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications. Amazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices.  

What other are measures should I consider to ensure Qualified State Integrity? You need to design your infrastructure based on IT best practices that meet your business, security and compliance requirements.  For example, you can define encryption requirements (forcing server side encryption for S3 objects), permissions to resources (which roles apply to certain environments), which compute images are authorized (based on hardened images of servers you have authorized), and what kind of logging needs to be enabled (such as enforcing the use of CloudTrail on applicable resources).  Such security best practices can be enforced by using CloudFormation templates. Our goal is to create a GxP audit-ready environment.  For example, AWS Config allows you to capture the current state of any environment, which can then be compared with your “secure environment” rules. You can ensure that the controls are operating 100 percent at any point in time, versus traditional audit sampling methods or point-in-time reviews.

Further Reading...

GxP Cloud on AWS

Automating Security, Compliance, and Governance in AWS

Service Based Accretive GxP Qualification Model For Cloud Infrastructure

Continuous Qualification for AWS or Microsoft Azure - How is it done?