#testops

Background

FDA released its final guidance on “Data Integrity and Compliance with Drug cGMP” in December 2018. The main purpose of this guidance was to clarify the role of data integrity in current good manufacturing practice (cGMP). This guidance provides the Agency’s current thinking on the creation and handling of data in accordance with cGMP requirements.

I wrote a blog post based on the DRAFT guidance in 2017. This post is a followup post based on the final guidance.

Why is this guidance important?

"In recent years, FDA has increasingly observed cGMP violations involving data integrity during cGMP inspections. This is troubling because ensuring data integrity is an important component of industry’s responsibility to ensure the safety, efficacy, and quality of drugs, and of FDA’s ability to protect the public health. These data integrity-related CGMP violations have led to numerous regulatory actions, including warning letters, import alerts, and consent decrees."  -- Data Integrity & Compliance with Drug cGMP - FDA Final Guidance

If you are using a cloud app to manage any GxP function, understanding the current thinking within the FDA is of paramount importance for ensuring regulatory compliance. The expectations of the FDA to ensure data integrity will impact:

Cloud App Design & Implementation

Validation

User Access Controls

Segregation of Duties

Audit Trail / Electronic Signature Implementation

Data Backup, Archiving & Retention

Disaster Recovery & Business Continuity (BCP)

Ongoing Validation & Maintenance

Audit Trail Reviews

What Questions Do You Need to Ask to Ensure Cloud App Compliance?

Am I prepared to keep my Cloud App in a “Validated State” considering the frequency of changes not just at the SaaS level but also at the IaaS/PaaS levels?

Do I have a robust validation process to ensure my Cloud App stays validated in spite of the frequent patches and new releases?

Can I unequivocally state that my cloud app validation meets the current FDA requirements for data integrity?

Do I have a process not to just validate the audit trail transactions but review them per current FDA expectations?

Cloud App Validation Strategies for Data Integrity

"Firms should implement meaningful and effective strategies to manage their data integrity risks based on their process understanding and knowledge management of technologies and business models."  -- Data Integrity & Compliance with Drug cGMP - FDA Final Guidance

Your validation strategy needs to provide assurance that the intended use of the app is met on a “continuous” basis. You need to take a risk based approach to ensure the intended use is not just met during initial validation but continues to meet through the myriad of patches and releases. To ensure data integrity your program must validate that the historical data can be retrieved without any compromises. In addition, your program must ensure that the audit trail transactions function as expected and there is a process for scheduled reviews.

xLM TestOps Framework for FDA Data Integrity Compliance

xLM’s TestOps managed service is designed to meet the FDA’s Data Integrity guidance in the context of cloud app validation. TestOps framework is based on “continuous validation” which is specially designed for GxP compliance in the cloud. Here is how it works:

TestOps is a modern framework based on software test automation. It provides end to end automated validation for any cloud app. The initial validation package is purposefully built for your instance based on your intended use. Read more…

Validation runs can be scheduled or run on-demand (for example: when a patch is released). During any of these subsequent runs, the intended use is not only affirmed but also the integrity of historical data is confirmed.

During the above steps, the audit trail and e-sig transactions are validated for all critical GxP functions.

A Big Data/AI based audit trail log analysis system automates the “scheduled” reviews.

Conclusion

Veeva Validation Challenge!

Veeva is a powerful cloud platform that can span the entire drug life-cycle (from R&D to Commercial). It is a multi-tenant platform that is constantly evolving - at least three to four new releases a year. Veeva provides a good vendor validation package that you as a customer can leverage. There is only one caveat - the end-user company is still responsible for Veeva validation and cannot pass the buck to the vendor. In other words, you are responsible for the GxP validation of your instance.

Why is “End-User” validation of Veeva a challenge?

The “last mile” validation of a multi-tenant cloud platform like Veeva at a minimum involves establishing that the “intended use” of your instance was met at initial deployment. This is not too bad, right. A validation plan, user requirements specification, configuration specification, user acceptance testing, and traceability matrix will do the trick! But what about change control of your instance? There are 3 or 4 new releases a year.

"The challenge is how to ensure your intended use is met with every new release?"

The challenge is how to ensure your intended use is met with every new release? There are features that are “auto-enabled” and there are features that can be turned on by you if you choose to. Here is your conventional model to tackle changes that come with every new release:

Perform Compliance Analysis to identify the impact of new features and changes

Based on your risk-based approach, clearly identify the tests that make up the regression suite.

Identify what new test cases have to be created, what existing ones need to be updated.

Update your specifications and other documents.

Manually execute your regression suite along with the new/changed test cases.

Prepare a summary report.

Performing the above-mentioned tasks 3 to 4 times a year to stay in a validated state is not fun and sucks up a lot of resources. You then begin longing for the on-prem version where the upgrade cycle may be a few years (not a few months). Veeva cloud is supposed to bring in a new era of constant innovation to help you constantly increase your productivity! This is not what I am seeing on the “ground” on the validation front.

Why is xLM TestOps an ideal solution to this End-User Validation Challenge?

xLM TestOps was designed for cloud platforms like Veeva from ground up. It incorporates agile frameworks designed for GxP Compliance. Best of all it is provided as a managed service so that you can rest in peace and not worry about the nitty gritty details about test automation.

What is xLM TestOps?

xLM TestOps is built on modern Software Testing frameworks. It leverages both API based as well as UI based test automation. We balance the right mix of API and UI based test automation to optimize on efficiency while not compromising on compliance.

Our initial end-user validation package includes user acceptance testing that is built on DevOps pipelines and 100% test automation.

How is xLM TestOps different from conventional validation?

To begin with, manual validation testing has been removed and is replaced with a continuous validation managed service built on modern software testing frameworks. The mantra is “use software to validate software”!

Running a regression suite takes minutes and not days.

TestOps provides at a minimum 10x the coverage compared to manual testing and incorporates sophisticated techniques: Combinatorial Testing, Multi-Browser/Multi-OS Testing, Randomized Testing, Data Driven Testing, etc..

All this comes at a fraction of the cost (our customers are enjoying over 50% in cost savings) and delivered at speed.

How is it done?

Step 1: As soon as the new version hits the Veeva Sandbox, we run the TestOps pipeline (DRY RUN) which incorporates 100% regression testing. Within minutes we will inform you of the impact of the new release on your intended use. This includes the impact of all auto-enabled new features.

Step 2: Based on the Step 1 results, any adjustments to the existing validation stack including test scripts are made.

Step 3: A formal run which incorporates 100% regression suite. A summary report is generated to validate the new release.

Step 4: Steps 1 to 3 are repeated if needed to validate any new optional features.

Conclusion

"Conventional validation methods based on rusty manual testing is akin to fighting a forest fire with a garden hose."

TestOnNeed for Entrepreneurs, Startups and Enterprises.

Já não é exclusividade dos software developers o segundo chapéu de Operations, formando assim o papel de DevOp (Developer + Operations). Agora os Testers e QAs também podem receber esse upgrade.

‘TestOps" é o novo termo que define o Analista de Qualidade do futuro, aquele que tem conhecimento e se envolve no processo de deploy,  configura o servidor de integração contínua e monitora o status do processo; aquele que analisa a performance do sistema; instala, configura e otimiza as ferramentas utilizadas pelo time; alem de automatizar seus testes. Ou seja, se aventura por terrenos alem dos limites da tradicional qualidade de software.

Executar queries em SQL para analisar a causa de um bug de inconsistência de dados ou saber como hackers podem explorar as possíveis vulnerabilidades do sistema que você esta testando são exemplos do dia-a-dia de um TestOp. Até mesmo ser um ‘white-hat hacker’ por um dia... Não é empolgante? Mais empolgante que ser QA tradicional certamente é.

Em um post de 10/2011 Simon Munro cunha o termo - caso ninguém tenha feito antes, ele foi o responsável; em seguida Seth Eliot da Microsoft escreveu três vezes sobre o novo termo. Eliot o relaciona com ambientes de computação em nuvem, porem acredito que sua aplicação seja mais ampla.

Imagine em um projeto ao melhor estilo Agile seguindo o whole-team-approach (onde todos na equipe são responsáveis pela qualidade do projeto), nesse cenário os perfis de alguns integrantes se tornam no mínimo duplos, com o skill predominante somado ao skill de operação. Nesse caso o QA também é bem vindo quando tem skills de usabilidade, programação, redes, sistemas operacionais, design, ou quaisquer áreas que forem agregar valor e agilidade ao projeto.