xLM - Continuous Validation

Background

FDA released its final guidance on “Data Integrity and Compliance with Drug cGMP” in December 2018. The main purpose of this guidance was to clarify the role of data integrity in current good manufacturing practice (cGMP). This guidance provides the Agency’s current thinking on the creation and handling of data in accordance with cGMP requirements.

I wrote a blog post based on the DRAFT guidance in 2017. This post is a followup post based on the final guidance.

Why is this guidance important?

"In recent years, FDA has increasingly observed cGMP violations involving data integrity during cGMP inspections. This is troubling because ensuring data integrity is an important component of industry’s responsibility to ensure the safety, efficacy, and quality of drugs, and of FDA’s ability to protect the public health. These data integrity-related CGMP violations have led to numerous regulatory actions, including warning letters, import alerts, and consent decrees."  -- Data Integrity & Compliance with Drug cGMP - FDA Final Guidance

If you are using a cloud app to manage any GxP function, understanding the current thinking within the FDA is of paramount importance for ensuring regulatory compliance. The expectations of the FDA to ensure data integrity will impact:

Cloud App Design & Implementation

Validation

User Access Controls

Segregation of Duties

Audit Trail / Electronic Signature Implementation

Data Backup, Archiving & Retention

Disaster Recovery & Business Continuity (BCP)

Ongoing Validation & Maintenance

Audit Trail Reviews

What Questions Do You Need to Ask to Ensure Cloud App Compliance?

Am I prepared to keep my Cloud App in a “Validated State” considering the frequency of changes not just at the SaaS level but also at the IaaS/PaaS levels?

Do I have a robust validation process to ensure my Cloud App stays validated in spite of the frequent patches and new releases?

Can I unequivocally state that my cloud app validation meets the current FDA requirements for data integrity?

Do I have a process not to just validate the audit trail transactions but review them per current FDA expectations?

Cloud App Validation Strategies for Data Integrity

"Firms should implement meaningful and effective strategies to manage their data integrity risks based on their process understanding and knowledge management of technologies and business models."  -- Data Integrity & Compliance with Drug cGMP - FDA Final Guidance

Your validation strategy needs to provide assurance that the intended use of the app is met on a “continuous” basis. You need to take a risk based approach to ensure the intended use is not just met during initial validation but continues to meet through the myriad of patches and releases. To ensure data integrity your program must validate that the historical data can be retrieved without any compromises. In addition, your program must ensure that the audit trail transactions function as expected and there is a process for scheduled reviews.

xLM TestOps Framework for FDA Data Integrity Compliance

xLM’s TestOps managed service is designed to meet the FDA’s Data Integrity guidance in the context of cloud app validation. TestOps framework is based on “continuous validation” which is specially designed for GxP compliance in the cloud. Here is how it works:

TestOps is a modern framework based on software test automation. It provides end to end automated validation for any cloud app. The initial validation package is purposefully built for your instance based on your intended use. Read more…

Validation runs can be scheduled or run on-demand (for example: when a patch is released). During any of these subsequent runs, the intended use is not only affirmed but also the integrity of historical data is confirmed.

During the above steps, the audit trail and e-sig transactions are validated for all critical GxP functions.

A Big Data/AI based audit trail log analysis system automates the “scheduled” reviews.

Conclusion

Bridging the Gap between GxP Validation and Assurance from xLM Continuous Validation on Vimeo.

Today, companies use a variety of cloud software to manage regulated data produced in GxP environments (e.g, clinical trials, manufacturing, etc.). Maintaining compliance and validation is a challenge with continuously-updating Software-as-a-Service.

iLink Systems and xLM have joined hands to provide “continuously validated” Microsoft apps and services to Life science customers worldwide. This partnership enables delivery of a range of services that can be a game changer for Life science customers replatforming to Microsoft Azure. Together they have developed a Managed Service Platform designed specifically for life-sciences to “GxP enable” Microsoft Azure implementations.

How does this partnership help Life science companies?

Replatform with built-in GxP frameworks, easing management of GxP workloads on Azure

Maximize Azure cloud-hosted application performance while reducing validation costs

Reduce time and resources spent on separate validation efforts, while keeping up with the rapid changes in Azure services

Leverage automation services eliminating the time and effort between instance spin up and service provisioning

Access GxP Validation Health and continuous automated compliance to 21 CFR Part 11, Annex 11, and GxPs in real-time

What are the “tangible” benefits?

Decrease time to launch qualified environments from 3 months to 3 hours

Reduce time spent on deployments and qualification as much as 90%

Decrease requalification costs by as much as 95%

Increase qualification coverage by more than 500%

About iLink Systems: iLink has over 18 years of providing Infrastructure Services and Solutions. They manage 45+ Data Centers, 65,000 + assets. They have invested in platforms, tools and a robust ecosystem of alliances that helps them deliver differentiated value. They are a Microsoft CSP Tier 1 Partner and a Product Development Partner. They hold ISO 9001 & CMMI Level 3 certifications. They have achieved an overall Microsoft customer satisfaction survey score is 192.31/200 (The average score for Gold Certified partner this year was 168.09).

xLM and AODocs jointly presented a webinar on how a modern cloud native AODocs QMS is delivered validated!  Even companies like Google use this patented AODocs Platform. The following topics are covered in this webinar:

What are the inner workings of a GxP Compliant QMS System that is purposefully built on Google Cloud

How can continuous validation be tightly integrated at the Platform and Application Layers

How this modern, highly scalable QMS leverages G Suite's document level collaboration

Veeva Validation Challenge!

Veeva is a powerful cloud platform that can span the entire drug life-cycle (from R&D to Commercial). It is a multi-tenant platform that is constantly evolving - at least three to four new releases a year. Veeva provides a good vendor validation package that you as a customer can leverage. There is only one caveat - the end-user company is still responsible for Veeva validation and cannot pass the buck to the vendor. In other words, you are responsible for the GxP validation of your instance.

Why is “End-User” validation of Veeva a challenge?

The “last mile” validation of a multi-tenant cloud platform like Veeva at a minimum involves establishing that the “intended use” of your instance was met at initial deployment. This is not too bad, right. A validation plan, user requirements specification, configuration specification, user acceptance testing, and traceability matrix will do the trick! But what about change control of your instance? There are 3 or 4 new releases a year.

"The challenge is how to ensure your intended use is met with every new release?"

The challenge is how to ensure your intended use is met with every new release? There are features that are “auto-enabled” and there are features that can be turned on by you if you choose to. Here is your conventional model to tackle changes that come with every new release:

Perform Compliance Analysis to identify the impact of new features and changes

Based on your risk-based approach, clearly identify the tests that make up the regression suite.

Identify what new test cases have to be created, what existing ones need to be updated.

Update your specifications and other documents.

Manually execute your regression suite along with the new/changed test cases.

Prepare a summary report.

Performing the above-mentioned tasks 3 to 4 times a year to stay in a validated state is not fun and sucks up a lot of resources. You then begin longing for the on-prem version where the upgrade cycle may be a few years (not a few months). Veeva cloud is supposed to bring in a new era of constant innovation to help you constantly increase your productivity! This is not what I am seeing on the “ground” on the validation front.

Why is xLM TestOps an ideal solution to this End-User Validation Challenge?

xLM TestOps was designed for cloud platforms like Veeva from ground up. It incorporates agile frameworks designed for GxP Compliance. Best of all it is provided as a managed service so that you can rest in peace and not worry about the nitty gritty details about test automation.

What is xLM TestOps?

xLM TestOps is built on modern Software Testing frameworks. It leverages both API based as well as UI based test automation. We balance the right mix of API and UI based test automation to optimize on efficiency while not compromising on compliance.

Our initial end-user validation package includes user acceptance testing that is built on DevOps pipelines and 100% test automation.

How is xLM TestOps different from conventional validation?

To begin with, manual validation testing has been removed and is replaced with a continuous validation managed service built on modern software testing frameworks. The mantra is “use software to validate software”!

Running a regression suite takes minutes and not days.

TestOps provides at a minimum 10x the coverage compared to manual testing and incorporates sophisticated techniques: Combinatorial Testing, Multi-Browser/Multi-OS Testing, Randomized Testing, Data Driven Testing, etc..

All this comes at a fraction of the cost (our customers are enjoying over 50% in cost savings) and delivered at speed.

How is it done?

Step 1: As soon as the new version hits the Veeva Sandbox, we run the TestOps pipeline (DRY RUN) which incorporates 100% regression testing. Within minutes we will inform you of the impact of the new release on your intended use. This includes the impact of all auto-enabled new features.

Step 2: Based on the Step 1 results, any adjustments to the existing validation stack including test scripts are made.

Step 3: A formal run which incorporates 100% regression suite. A summary report is generated to validate the new release.

Step 4: Steps 1 to 3 are repeated if needed to validate any new optional features.

Conclusion

"Conventional validation methods based on rusty manual testing is akin to fighting a forest fire with a garden hose."

FOR IMMEDIATE RELEASE: 10/29/2019 Sid Agrawal xLM – Continuous Validation (919) 889 5978 [email protected]

xLM-Sumo Logic Celebrate a Match Made in the Cloud

Innovative companies join forces to accentuate client offering and enable compliance in the cloud-era

Philadelphia, PA: 29-OCT-2019: xLM, the company behind an automated platform that can “validation enable” any cloud App, announced its partnership with Sumo Logic. The partnership will address all validation-related concerns of existing and future Sumo Logic Life Sciences customers.

Of the new strategic alliance, Sumo Logic Vice President of Global Partner Sales and Alliances Jabari Norton said, “We are excited about our strategic partnership with xLM as it will benefit Life Science customers by providing them real-time insights to monitor their GxP workloads across public, private or hybrid cloud environments. The combination of our Continuous Intelligence Platform coupled with xLM’s solution provides the ability to tackle the tsunami of data that businesses are under pressure to understand and act upon to drive great customer experiences in today's cloud era.”

“The Continuous Validated state of the Sumo Logic cloud App will allow its users to “GxP enable” their instances shifting focus to using the App to its full potential – all while considerably reducing resources and compressing timelines,” said Sid Agrawal, Vice President Marketing and Business Engagement at xLM.

Through this partnership, customers will be able to take advantage of xLM’s complete validation life-cycle management related services for the Sumo Logic App making it fit for use as per FDA’s current 21 CFR Part 11 regulation as well as upcoming FDA Computer System Assurance (CSA) guidance.

About xLM: xLM - Continuous Validation is a technology firm based in the Greater Philadelphia Area, PA. Its managed service is governed by a cutting-edge Quality Management System built on ISO 9001:2015, 21 CFR Part 11, Annex 11 and GxPs. xLM is a subsidiary of ValiMation Inc., an established name in the GxP and validation-related consulting space serving big and small Life Sciences companies globally since 1996.

What is the scoop on FDA’s CSA?

FDA’s CDRH is planning on releasing a guidance on Computer Software Assurance (CSA) for Manufacturing, Operations, and Quality System Software in 2020 (applies to non-product CSV only). Currently, it is on their “A” list. There has been a lot of "industry chatter” about this particular guidance related to computer validation. The reason is that FDA has not updated their guidances on computer validation for about two decades (“cloud” did not really exist then!). Everyone is eager to know what their current position is when it comes to computer validation. So much has changed in the last two decades and we still don’t know what FDA’s current thinking is when it comes to the “cloud”. FDA has surely managed to keep us behind a “dark cloud”!

" “the medtech industry’s high focus on meeting regulatory requirements versus adopting best quality practices has the potential to increase risk to patients. This compliance-centric approach has resulted in quality issues and has hampered innovation in manufacturing and product development practices. Additionally, this single-mindedness has led to low rates of investment in automation and digital technologies.”"  -- Francisco (Cisco) Vicenty, Case for Quality at FDA’s CDRH

What are key takeaways?

To begin with, they are getting rid of the word “validation” and replacing it with “assurance”.

The idea here is that the industry needs to use smarter techniques to ensure software assurance is high. They want us to borrow a page or two from the high tech software industry.

They are encouraging the use of automation, not only in automating various processes but also in their validation.

FDA is very openly asking the industry why are you so “paper” heavy! Use automation to drive higher software quality.

They are questioning why the industry is not focused on methods and activities that enhance software assurance. FDA is stating that the industry is bog down in paper while not improving software quality.

FDA is asking us to incorporate “critical thinking” to increase software assurance. They are literally want the industry to turn things upside down when it comes to validation (see Figure below).

FDA is even encouraging the industry to use risk based techniques much more than they do now. They have gone to the extent of advising the industry to use Unscripted Exploratory Testing, Unscripted Error Guessing and Unscripted Adhoc Testing - of course based on a risk based analysis.

It is very clear that FDA is openly nudging the industry to adopt varying degrees of testing based on risk. They are questioning if the current one size fits all paper heavy process brings any value at all.

How do I know if my validation program is in sync with FDA’s CSA?

Let us take a short quiz.

Do you review the Release Notes for every new release and then try to decipher the cryptic notes by the vendor?

Do you selectively run regression tests and not the complete suite of tests with every release or patch?

Do you manually execute your tests?

Do you measure the impact of your validation on increasing software quality?

If your answer is 1-YES, 2-YES, 3-YES and 4-NO, your program is stuck in the past.

What are the best practices to be in sync with FDA CSA?

Embrace a “no paper” approach to validation soaked in automation. This is possible only if you adopt a sensible test automation framework which not only saves time but can increase your software assurance by a factor of 10X or more. With this, you can run your entire regression suite in minutes on a continuous basis.

Build your test cases into the coding environment so that there is a direct traceability between the test scenarios and the test automation code (for example the Gherkins framework).

Adopt a pipeline based test deployment framework with built-in approvals. Such a framework can easily plug into your DevOps and provide all the audit trails to track every step in every run.

Analyze the historical run data to see what the software validation scores are with every release.

What is TestOps? How does it help you continuously qualify AWS/Azure?

"TestOps is a software validation culture and practice which aims at shorter testing and validation cycles to continuously qualify any cloud IaaS/PaaS service, in close alignment with business and regulatory objectives. "

Many life-science companies are asking the question “Can I move my GxP workloads into the public cloud?” Answering “yes” to this question leads to another question: “What toolset do I need to qualify the various IaaS/PaaS services that are constantly changing?”.

Our customers are leveraging our robust framework that we call TestOps. TestOps provides an end to end automation for continuously qualifying any IaaS/PaaS service running on AWS or Azure.

TestOps - How does it work?

"We test "continuously" while you operate your cloud infrastructure!"

In order to illustrate our TestOps framework, a simple use case to continuously qualify AWS DynamoDB is presented. Keep in mind that this framework can be applied to any IaaS/PaaS service or a combination of services.

Step 1 - Define the Intended Use

An IaaS or PaaS service can be configured to be used in many ways to meet varying end user needs. It is important to document the intended use to cover the various scenarios the service will be utilized in. This intended use forms the basis for designing a qualification strategy.

Step 2 - Design the Test Cases in the Coding Framework

Conventional wisdom directs us to develop test cases outside of coding using Microsoft Word as the authoring tool. This approach is surely outdated and does not provide “hard” traceability to your code. xLM’s TestOps supports “Given-When-Then” framework wherein the validation analyst can build test cases within the coding environment (IDE). These test cases are directly linked to the corresponding test automation code by the developers. In other words, both validation analysts and developers are working within the same environment and test case definitions are not separate from the code.

Step 3 - Coding the Test Cases

Once the test case design is complete, the developers code the testing scenarios which typically involves driving the browser for UI Testing or leveraging the APIs. The code that is developed is organized into reusable snippets (features, pages, utilities, etc.) for better maintainability of the test automation model. As mentioned above, the traceability between test case design and code is built into the framework.

Once the code development, code reviews, testing and quality checks are complete, the code is checked into the Code Factory.

Step 4: Build the TestOps Pipeline

Once the code is published, the TestOps Pipeline is configured:

to retrieve the published code

to deploy the qualified code into the test environment

to run the test automation code to qualify the SUT (System Under Test - in this case AWS DyanmoDB)

to generate a test automation report aka “executed protocol”

to archive all the test run artifacts

Step 5: Run the TestOps Pipeline

Once the Pipeline is fully configured and released by Quality, the Pipeline can be run either manually or automatically (scheduled or event based). The Pipeline run-time steps include:

Obtaining the Pre-approval to run the “qualified” Pipeline

Running the test automation code and generating the execution report

Step 6: Review the Results

After the Pipeline run is successful, the final step is to review the test automation results to ensure all the acceptance criteria have been met. This review is documented with the Post-deployment approval.

Conclusion

xLM’s TestOps Framework is designed to continuously qualify any IaaS/PaaS service (AWS, Azure, Google Cloud) cost efficiently utilizing test automation and pipelines. Such a framework can enable you to build a Service Catalog of “qualified” IaaS/PaaS services so that your development teams can consume these services on an on-demand basis. This model completely removes the qualification hurdle each time a project is deployed in the cloud.

This framework utilizes the following key six (6) steps:

Development of the Intended Use that is traceable to all qualification models

Development of the test cases using the Given-When-Then framework that is directly linked to the test automation code

Development of the Test Automation Models.

Configuring the Pipeline

Running the Pipeline