#service features

INTRODUCTION

The main objective of this post to answer the following questions.  I know a picture is worth a thousand words and so I will do this with illustrations.

Can "Validation" and "DevOps" be used in the same sentence (like vCD - Validated Continuous Delivery)?

Is DevOps a viable model for validated apps in GxP Environments?

How can one ensure the "validation fidelity" in a DevOps environment?

First, let us look at some definitions:

DevOps (a clipped compound of "development" and "operations") is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, releasing to deployment and infrastructure management. DevOps aims at shorter development cycles, increased deployment frequency, and more dependable releases, in close alignment with business objectives. (Source:  Wikipedia)

Validated Continuous Delivery (vCD) is a software engineering culture and practice which aims at shorter development and validation cycles, increased deployment frequency of "compliant" releases, in close alignment with business and regulatory objectives.  (Source: xLM)

MODEL SUMMARY

In order to answer the three questions above, I will present a simple working model.  This model includes the following steps:

Code Development - Code is developed using an Agile Model and committed to the Repository.

Continuous Integration (CI) - The CI Server builds the release.

Code Deployment - The Build is deployment to the Test Cluster.

Build Validation - The Build is validated by Continuous Validation (CV) and Executed Protocols are posted for QA Review.

QA Review - QA reviews the executed protocol and approves the results.

Summary Report - A Summary Report is automatically generated that captures all the steps in the Pipeline with detailed evidences.

Deployment to Production - This activity can be scheduled and managed by the Deployment Team.

The above steps are illustrated in the following diagram:

The above pipeline was built in AWS CodePipeline and is shown in the figure below:

MODEL DETAILS

Code Pipeline - Step 1 - Git Repository

The source code was managed using a Git Repo.  The first step in the pipeline was setup to monitor the Repo and would execute when a commit was made.

Code Pipeline - Step 2 - Jenkins Build

The pipeline automatically retrieved the source code package and sent it to the Jenkins Continuous Integration Server.  The Jenkins server was setup to build the deployment package and deliver it to the Pipeline.

Code Pipeline - Step 3 - Code Deploy

The Pipeline was setup to automatically deploy the Build to a cluster of three (3) test servers using AWS CodeDeploy  

Code Pipeline - Step 4 - Test (Release Validation)

The Pipeline was setup to create a Test (Validation) Session in xLM and this session in turn was setup to launch the xLM Test Automation Model.

The Model would run all the required tests (IQ, OQ, UAT) and generate an Execution Report.

Code Pipeline - Step 5 - QA Approval to Release

The Pipeline was setup to stop execution until the QA reviewed the test results and approved it. 

Code Pipeline - Step 6 - Automatic Summary Report Generation

The Pipeline was setup to create a Summary Report Session in xLM and this session in turn was setup to launch the xLM Report Automation Model.

The Model would automatically capture all the evidence required to generate the Pipeline Validation Summary Report.

CONCLUSION

vCD (Validated Continuous Delivery) is a reality.  The DevOps model can be successfully adopted to continuously deliver GxP Apps.  The above Six Step Model provides the framework along with the toolset to implement "continuous validation" in the enterprise.  Such a model will not only compress deployment timelines but will increase compliance using automation and reduces cost significantly. 

FDA put out its DRAFT guidance on Data Integrity and Compliance with cGMP in April 2016. This guidance is important because we come to understand FDA's thinking on Data Integrity.

"For the purposes of this guidance, data integrity refers to the completeness, consistency, and accuracy of data. Complete, consistent, and accurate data should be Attributable, Legible, Contemporaneously recorded, Original or a true copy, and Accurate (ALCOA)."  -- FDA Guidance on Data Integrity and Compliance With CGMP, April 2016

My focus in this post will be limited to IT applications. Needless to say the guidance has big repercussions for anyone in areas of IT Quality and Computer Systems Validation. Many predicate rules that were written for the paper world still hold good and can be applied to e-records as well. FDA does cite quite a few of them in the guidance.

The two words "Data Integrity" has far reaching implications for computer systems. Any FDA regulated company need to carefully look at their CSV program and revisit, but not limited to, the following areas:

System Design & Implementation

Validation

User Access Controls

Segregation of Duties

Audit Trail Design and Capture

Review of Audit Trails

Data Backup and Archiving

Data Retention

Disaster Recovery & Business Continuity (BCP)

Electronic Signature Design & Implementation

Training

I would like to focus on only one topic in this post: Audit Trail reviews. There can be two (2) types of reviews:

Regular Review: Audit Trails that need to be reviewed with the "parent GxP record".

"FDA recommends that audit trails that capture changes to critical data be reviewed with each record and before final approval of the record. Audit trails subject to regular review should include, but are not limited to, the following: the change history of finished product test results, changes to sample run sequences, changes to sample identification, and changes to critical process parameters."  -- FDA Guidance on Data Integrity and Compliance With CGMP, April 2016

Scheduled Review: Audit Trails that need to be reviewed at regular intervals.

"FDA recommends routine scheduled audit trail review based on the complexity of the system and its intended use."  -- FDA Guidance on Data Integrity and Compliance With CGMP, April 2016

Let us take regular review of audit trails. The expectation here is that the audit trail associated with a "critical GxP record" must be reviewed along with the record itself (e.g.: a batch record). A "data integrity friendly" application must present the audit trail data to the reviewer in a way that is easily understandable and user friendly for this criteria to be met. Otherwise the reviewer will be spending 10x the time reviewing the audit trail rather than the parent record. Most modern and well designed IT apps can accomplish this out of the box (e.g.: Atlassian Jira).

Now let us move on to the trickier "scheduled review". Anyone who is familiar with an Enterprise application realizes that this can be a daunting task. Even if you decide to perform such reviews, you need to ask: How often? What to Review? How to make this a value added exercise? What is the cost of doing such reviews? How many resources do I need?

Some of the larger pharma companies are mandating that SOPs must be developed and audit trail reviews must be conducted on a periodic basis. System owners and administrators are struggling with this mandate. Most of the legacy applications generate logs and audit trails that can be read only by software programmers rather than end users. And also the sheer volume of audit trail records make this an insurmountable task (for example: a cloud enterprise application can generate hundreds of audit trail records for a single workflow execution). Imagine a situation where one has to review a million records generated in one quarter!

Big Data Analytics and Machine Learning can be your savior. A Big Data system can be setup to consume audit trail logs and records and display meaningful information in a dashboard format. Such a system can also send notifications of any unusual activity. Also, such a system can categorize transactions and provide statistical information. Such an Enterprise system can consume logs and records from various apps and perform bulk of the scheduled reviews.

Such a system can:

Look for unusual login activity

Monitor record deletion (if such an activity is not permissible)

Monitor changes to critical system configuration records

Monitor user role changes

Monitor abnormal, disallowed or unusual record state changes

Monitor system logs for critical application errors and correlate them with user activity

And much more...

There are many advantages in designing such a Big Data system. Once a good baseline is established by providing it with historical data, it can learn from it and flag unusual activity in near real-time. For example: such a system can flag a particular sequence of record status change as "unexpected" based on the historical data.

An intelligent Big Data system can be your big friend to handle audit trail reviews. Such a system should be integral part of your data integrity and cyber-compliance toolset.

Scenario: 

Any Track and Trace Enterprise application is a complex implementation which has many interfaces.  What are the requirements of a validation toolset for such an application?  Let us take a look.

A typical architecture diagram is as shown below.  The Pharmaceutical Company manages the Master Data, Serial Number Management, Partner Management, Country Specific Compliance, etc..   This company has to manage interfaces to its Contract Manufacturers & Packagers, 3PLs and Customers.  

The Track and Trace Software App is constantly changing:

Regulatory Changes

Data Schema Changes

Evolving Industry Standards

Country Specific Deadlines

Add/Remove CMOs, 3PLs, CPOs

Software Patches

New Releases

The requirements for the validation toolset are:

Continuous Validation Platform:

The validation platform that is required to meet the qualification challenges of a modern, complex Track and Trace enterprise app is in itself must be sophisticated.  xLM has developed a platform and a managed service that stacks up to this challenge.  Click here for xLM Platform details, click here to see how xLM Continuous Validation Service works and click here for xLM's process flow. 

What is the Bottomline?

Conclusion:

The question is "Can Amazon AWS or Microsoft Azure be qualified so that I Life Science companies can run validated apps on it?"   

Cloud releases updates and changes at such great velocities that leaves all validation specialists continuously wondering on how to qualify it.  Traditional strategies including GAMP 5 was never designed to address continuous changes.  These models were designed for a "waterfall" world and not an "agile" one.  It is not possible to qualify an infrastructure where changes are released without release notes with such an outdated mindset and toolset.

Does this mean that the Life Science world should stay in the dark ages with their paper based, wet signature toolset and a brick and mortar data center?  You need continuous Qualification for Continuous Compliance that works better than ever, just like the Cloud!

Here is a step by step guide to Continuous Qualification:

Step 1: Design your BASE Cloud Infrastructure (combination of IaaS / PaaS)

Step 2: Divide your BASE into logical Services

Core Services (e.g. Networking, Compute)

Management Services (e.g. Auditing, Monitoring)

Step 3: Build your DEV BASE IaaS/PaaS.  Finalize and test your configuration.

Step 4: Develop a Qualification Plan / Governance Framework

Step 5: Subscribe to our IaaS / PaaS Qualification Service.  We have built a sophisticated infrastructure to qualify Microsoft Azure or AWS Services.  Our Service Qualification coverage is unmatched.  We post the following on a "Weekly" basis on our customer portal

Service Qualification Plan

Service Requirements Specification (with Comprehensive Coverage)

Service Risk Assessment

Service Qualification Protocols & Execution Reports

Service Qualification Summary Reports

Service Release Certificates

Step 6:  You don't have to Qualify any Service.  We will do that for you on a continuous basis.  Our "continuous" qualification platform for each Service will provide documented evidence that your Service continuous to work as designed (irrespective of many changes pushed by the Cloud Vendor)

Step 7:  Deploy PROD BASE IaaS/PaaS

This blog post is Part 3 in this series. What are the key process steps involved in continuously validating a Cloud App? (click here for Part 1; click here for Part 2)

STEP 1

Customer will provide the User Requirements (URS) to xLM. 

STEP 2

xLM IT will create a new xLM instance on its qualified infrastructure.  This new instance is validated.

A Validation Plan and Risk Assessments are created.

A new SUT (System Under Test) is created in xLM.  xLM team will

input the requirements

perform risk assessment working with the customer

create specifications (design, configuration, etc..)

create test cases (automated, manual).  All test cases are pre-approved by the customer.

Bi-directional traceability (Requirements <> Specifications <> Test Cases) is automatically generated by the system.

Customer approves all validation deliverables.

STEP 3

Test Scripts (IQ, OQ and/or UAT) are executed in the Cloud App QA Environment.

Customer approves the Validation Summary Report (automatically generated by xLM)

STEP 4

A Release Certificate will be issued by xLM.

Cloud App released to Production

STEP 5

Whenever a patch or upgrade is released by the Cloud App Vendor(s),  xLM will perform Compliance Analysis and a report will be issued to the customer.  Updates to validation deliverables is managed under change control.

Once this report is approved by the customer, xLM will create a test campaign (Regression Testing, New Scripts for New Feature Testing) and will execute this campaign in QA Environment.  This will be performed before the patch/upgrade is deployed in the PROD Environment.

A Release Certificate will be issued by xLM.

STEP 6

Continuous Validation:  Regression/Smoke tests are constantly run to ensure that the cloud app is in a "validated" state.

STEP 7

Customer will have 24/7 access to real-time "Cloud App Validation Health" portal.

Related Links

Why should you select xLM as the Validation Service Provider for your Cloud App?

This blog post is Part 2 in this series. What are the key steps involved in continuously validating a Cloud App? (click here for Part 1)

The above diagram depicts the key elements of a Continuous Validation Program for a Cloud App. One has to bear in mind that the underlying IaaS and PaaS infrastructure is constantly changing. In fact, the Cloud App itself is continuously changing. In the new cloud world very rarely you are given the option of pinning to an "ancient" version. Thus this Continuous Validation Framework is designed to mitigate these risks and ensure that your cloud app is maintained in a validated state.

"Continuous validation is providing documented evidence to certify that a cloud app not only met the pre-established acceptance criteria, but “continuous” to meet thus mitigating the risk of unknown changes."

Requirements Definition: This step provides the foundation for the continuous validation framework. You need to clearly specify your Functional, Non-functional, Regulatory, Performance, Security, Logging, Disaster Recovery, Interface requirements.  

Risk Assessment: Perform a thorough risk assessment to ensure business continuity, regulatory compliance, etc. are considered. In my experience very rarely a logical, useful risk assessment is performed, let alone applying it to testing strategies. The output of the risk assessment must be applied to your testing strategies to determine: What features to test? What Platform combinations to test on? What should be the extent of negative testing? What type of testing strategies to utilize (for eg: datasets to use, N Pair Testing)?  

Specification Definition: Specify IaaS/PaaS, Configuration, Workflow, Interface, Security, Log Management, etc.. specifications to meet your requirements.  

Test Automation Scripts: I recommend using a Model Based Test Automation framework for developing various models to validate your cloud app. Always separate your model from data which will be utilized during runtime. You can use a data designer to generate test data including randomized data. You can also use combinatorial testing strategies to reduce the number of iterations while increasing your test effectiveness.  

Test Model Validation: The test automation model must be validated to ensure that it is meets the specified objectives. This validation effort can be minimized by designed your model to generate good execution reports. A well designed model execution report provides enough evidence so that it can to the extent possible becomes self-validating.  

Test Execution: The model based test automation approach provides you with the flexibility to repurpose the same model to conduct various types of tests (smoke, regression, greedy path, optimal path, load, performance, etc..). Also, such a framework lends itself more conducive to updates (remember your cloud app is constantly changing...so will your test automation framework!). Such framework is well suited for continuously running validation test scripts (say on a daily basis) - you can even randomize your smoke and regression tests continuously!  

If it are looking as long as the superfluity class cleaning services for your soap opera or residential units, you are in the right place, no doubt. Although, there is interest lack of wholly arrayed chare centers in the country inanimate selection speaking of the defeat and definitely useful service provisioner out of a whole range of dealers is really a intricate process. But majestic thing is clear that if you will know utterly that what are the leading service features you have got to check out in the deportment of services autonomous proper to a dealer in this direction, you furlough conclude the best for you. So, have they ever heard aimlessly the janitorial services Miami group? If not, it's time to kill now to discover the word-for-word world in reference to janitorial services here.<\p>

How throne we offer unmarred services? <\p>

The top level janitorial service dealers fashionable US are glaring to understand your question first, and then work finally for prophesied solutions. No matter, what cast in relation to cleaning matters in hand at what level i are unreluctant to enjoy, they can support you present-day an select way with our nonesuch efforts. Moreover, a well oiled customer care staff is animated in their job center all being the split shift to hear from you. Also, their cleaning staff is really efficient to offer ego the aristocracy and satisfactory services. They are highly trained and familiarized to induce out good job for you and wave myself within the short period of keep pace with. All these excellent service features of their divan make alterum unique and greatest service dealers in this direction in the country.<\p>

Where lavatory they help you? <\p>

The highest service provider can meet your requirements every delay whenever inner man want a place to be cleaned commercially. They are lovable all for their flexible assistance and also they are ready always till fill you in cleaning of following buildings-<\p>

Poorhouse and office Departmental closet Parade and shopping secret places Schools and colleges Hospitals etc.<\p>

How to go-between them? <\p>