#chaosengineering

I recenti attacchi cinetici condotti tramite droni contro le infrastrutture AWS in Medio Oriente rappresentano un evento critico per l'analisi della sicurezza informatica. Questi episodi dimostrano che l'infrastruttura hardware alla base dei servizi cloud commerciali è fisicamente esposta. Questo solleva interrogativi precisi sui metodi di protezione fisica e sulle strategie di continuità operativa adottate dai grandi fornitori di servizi IT a livello globale.

I Limiti Strutturali delle Strutture Sotterranee

Di fronte a minacce fisiche, l'opzione di collocare i data center all'interno di strutture corazzate sotterranee presenta ostacoli ingegneristici ed economici severi:

- Dissipazione Termica: I server operano a temperature elevate e richiedono complessi impianti di raffreddamento. Sottoterra, il calore ristagna. L'espulsione dell'aria calda richiede condotti voluminosi verso l'esterno, creando punti di accesso vulnerabili. - Dipendenza dalle Reti di Superficie: Un data center necessita di enormi quantità di energia elettrica e di cavi in fibra ottica per la trasmissione dei dati. Poiché queste linee di approvvigionamento corrono in superficie, la loro distruzione isola completamente la struttura sotterranea. - Impulso Elettromagnetico (EMP): Le armi nucleari generano impulsi elettromagnetici in grado di distruggere i circuiti elettronici. In assenza di isolamenti magnetici di standard militare applicati a ogni cavo in ingresso, i server si danneggerebbero irreparabilmente anche se protetti da coperture in cemento armato. Ridondanza Geografica e Disaster Recovery

Per garantire la disponibilità dei servizi, le aziende tecnologiche evitano la concentrazione delle risorse in un'unica struttura fortificata e adottano la ridondanza geografica. I dati e i processi vengono replicati in tempo reale su più data center situati in nazioni o continenti differenti.

In caso di distruzione fisica di un nodo, la rete esegue un reindirizzamento automatico del traffico avvalendosi di tre protocolli primari:

- Protocollo BGP: I router scambiano segnali di stato continui. Se un data center cessa di trasmettere, i router adiacenti aggiornano le tabelle di rete globali per escludere il percorso non più disponibile. - Indirizzamento Anycast: Più data center annunciano lo stesso indirizzo IP. La rete invia i dati alla struttura geograficamente più vicina e funzionante. - Global Server Load Balancing (GSLB): I sistemi DNS centrali monitorano lo stato dei server e forniscono agli utenti finali esclusivamente gli indirizzi IP dei data center operativi. I Blocchi Globali del Sistema per Vulnerabilità Logiche

La ridondanza geografica è concepita per neutralizzare i danni fisici, ma costituisce una debolezza critica nei confronti degli errori software. Quando provider come AWS o Cloudflare subiscono blocchi su scala globale, la causa risiede in vulnerabilità logiche che si propagano attraverso la rete stessa:

- Errori di Configurazione: I sistemi cloud applicano gli aggiornamenti in modo sincronizzato e quasi istantaneo. Un bug inserito nel software di gestione viene replicato in pochi secondi su tutti i server globali, disattivando contemporaneamente le strutture intatte in ogni continente. - Blocco del Piano di Controllo: Le reti separano il transito dei dati dai sistemi di autorizzazione (come la gestione delle identità e dei permessi DNS). Se i server centrali di autenticazione smettono di funzionare, il traffico dati si arresta del tutto, rendendo inutile la disponibilità dei data center secondari. Il Fallimento Sequenziale per Sovraccarico

Un'ulteriore causa di disservizio esteso si verifica quando la caduta di un data center genera un eccesso di traffico sulle strutture rimanenti. Se un nodo principale viene disconnesso, i protocolli di sicurezza deviano automaticamente tutto il suo carico sui data center di backup. Se queste strutture secondarie non dispongono della capacità di calcolo sufficiente per elaborare i nuovi volumi di dati, si sovraccaricano. I sistemi di protezione interni disattivano quindi i server per evitare danni termici all'hardware. Questo riversa un quantitativo di traffico ancora maggiore sui nodi successivi, innescando una serie di spegnimenti sequenziali che portano al collasso dell'intera rete regionale o globale.

Il Rischio Sistemico Ineliminabile

In sintesi, i protocolli di sicurezza e la ridondanza geografica estrema non rendono l'infrastruttura cloud globale invulnerabile. Mentre la protezione contro le distruzioni fisiche localizzate è garantita dalla distribuzione dei nodi, l'interconnessione necessaria per il funzionamento del sistema crea un rischio strutturale sul fronte logico e software. Un singolo errore di configurazione replicato automaticamente su scala globale, o un sovraccarico di traffico durante un reindirizzamento d'emergenza, possono aggirare le difese hardware e innescare un blocco totale. La possibilità di una caduta diffusa dei data center, quindi, rimane un'eventualità concreta e costante nell'ingegneria di rete moderna.

Soluzioni e Strategie di Mitigazione

Per limitare i danni di un blocco globale o di un errore software a cascata, le infrastrutture critiche devono adottare approcci architetturali che riducano la dipendenza da un singolo sistema:

1. Architettura Multi-Cloud L'utilizzo esclusivo di un singolo fornitore espone l'intera infrastruttura ai suoi errori software interni. La mitigazione consiste nel distribuire i servizi su fornitori cloud concorrenti e indipendenti (ad esempio, dividendo il carico tra AWS, Microsoft Azure e Google Cloud). Se un aggiornamento errato disattiva la rete di un provider, i sistemi ospitati sugli altri fornitori continuano a operare normalmente.

2. Modello Ibrido e Backup On-Premise Per i dati di importanza vitale, come i registri bancari o le identità crittografiche, la sola replica sul cloud pubblico non è adeguata. Le aziende devono mantenere server fisici di proprietà (on-premise), fisicamente isolati dalle configurazioni software dei provider cloud. Questo garantisce l'accessibilità ai dati primari anche in caso di compromissione totale dei protocolli di routing BGP o di blackout del fornitore esterno.

3. Chaos Engineering (Ingegneria del Caos) I sistemi di ridondanza falliscono spesso perché non vengono testati in condizioni reali. La pratica del Chaos Engineering prevede lo spegnimento intenzionale e controllato di server e data center in ambiente di produzione. Questo processo forza i sistemi di failover ad attivarsi regolarmente, permettendo agli ingegneri di identificare e correggere i limiti di capacità prima di un disastro reale.

4. Sistemi di Fallback Statici e Indipendenti Quando il database principale o l'applicazione dinamica smettono di funzionare, il traffico deve essere deviato verso server di emergenza esterni alla rete principale. Questi server ospitano versioni statiche e di sola lettura del sito o del servizio. Questo accorgimento previene la visualizzazione di errori totali di connessione e mantiene attiva la comunicazione con gli utenti.

In today's fast-paced and ever-evolving business landscape, organizations are constantly striving to enhance employee productivity and ensure the resilience of their systems and applications. One innovative approach that's gaining momentum in the tech world is "Chaos Engineering." Chaos Engineering is not only about testing the reliability of your systems but can also have a profound impact on employee productivity and collaboration. In this blog, we'll delve into how the principles of Chaos Engineering can be harnessed to boost employee productivity and foster a culture of innovation and resilience within your organization.

Understanding Chaos Engineering

Chaos Engineering is the practice of deliberately injecting controlled and measured forms of chaos, such as system failures or performance bottlenecks, into your applications and infrastructure. The primary objective is to proactively identify weaknesses, vulnerabilities, and dependencies within your systems and fix them before they cause unexpected outages or issues. But how does this relate to employee productivity?

Resilience Breeds Confidence

Chaos Engineering instills a sense of confidence and resilience among employees. When your teams are confident in the system's ability to withstand unexpected failures, they are more likely to focus on their work, knowing that the technology will support their efforts. This boost in confidence leads to increased productivity, as employees spend less time worrying about system failures and more time on their tasks.

Collaboration and Learning

Chaos Engineering practices often involve cross-functional teams working together to design, execute, and learn from experiments. These collaborative efforts break down silos and encourage different departments to share their knowledge and insights. In the process, employees not only become more aware of the system's behavior but also of their colleagues' roles and responsibilities. The knowledge sharing and improved collaboration that stem from Chaos Engineering practices can lead to more effective problem-solving and a better work environment.

Reduction in Downtime

Downtime can be a significant productivity killer. Chaos Engineering helps identify and address weaknesses in your systems before they result in unexpected outages. This reduction in unplanned downtime means employees can carry on with their tasks uninterrupted, resulting in higher productivity and efficiency.

Encouraging Innovation

Chaos Engineering encourages a culture of innovation and continuous improvement. By actively seeking out weaknesses and vulnerabilities, teams become more open to trying new approaches and technologies. This can lead to the development of more efficient processes and solutions, further boosting productivity.

Enhanced Problem-Solving Skills

Chaos Engineering challenges employees to think critically, adapt to rapidly changing situations, and develop their problem-solving skills. These skills not only benefit the Chaos Engineering process but also transfer to other aspects of their work, making employees more adept at handling unexpected challenges and boosting overall productivity.

Conclusion

Introduction to quality assurance

Quality assurance (QA) is an umbrella term which encompasses all software and infrastructure quality tests that are performed to verify software application and infrastructure functional and non-functional attributes, including performance, scalability, availability and security. Azure cloud infrastructure and application testing is a major part of quality assurance. There are QA frameworks to be used for both software application and cloud infrastructure testing. Whether you perform QA using testing frameworks or you manually perform a baseline of tests, you need to have test scripts available. Each test script describes the steps to be carried out to test a certain aspect of an application or infrastructure system, as well as the result of each test and subsequent action items, if needed. This article provides a baseline reference of the steps to be included in each type of infrastructure or application test script alongside with a testing script template available for download.

Azure cloud infrastructure and application testing types

The following types of infrastructure and application test scripts are available: - Application unit testing. A unit test is a test that exercises individual software components or methods, also known as a "unit of work." Unit tests should only test code within the developer's control. They don't test infrastructure concerns. Infrastructure concerns include interacting with databases, file systems, and network resources. xUnit.net is a free, open source, community-focused unit testing tool for the .NET Framework. - Application integration testing. An integration test differs from a unit test in that it exercises two or more software components' ability to function together, also known as their "integration." These tests operate on a broader spectrum of the system under test, whereas unit tests focus on individual components. Often, integration tests do include infrastructure concerns. - Application load (or stress) testing. A load test aims to determine whether or not a system can handle a specified load. For example, the number of concurrent users using an application and the app's ability to handle interactions responsively. Refer to my KB article on Azure App Service load testing (using JUnit) for an example of carrying out load testing for an Azure-based Web application. - Infrastructure functional testing. This aims to test all operating systems, firmware and the functional attributes of on-prem or cloud infrastructure components involved in a solution. - Performance and scalability testing. During performance and scalability testing, you measure the performance metrics of your infrastructure as your application workload increases. This is combined with application software stress/load testing. The performance and scale tests involve setting up and testing autoscale policies for scaling horizontally or vertically when the application load requires it. - Availability testing. Availability tests mainly focus on testing the resiliency of your infrastructure in the event of hardware or software components failures and the associated potential downtime incurred during these failures. This should test your infrastructure and application code high availability, redundancy and resiliency. Resiliency can come in many forms, including load balancing, failover clustering (active-active or active-passive) and always on availability groups, virtual machine availability sets as well as cloud infrastructure fault domains, update domains, cloud zone and cloud region redundancy. The availability tests focus on creating deliberate failure simulation events, such as for example having an Azure VNET or subnet or storage account going down, to check the impact on the overall architecture. - Security testing. Security testing aims to test the attack surface of your architecture, the resiliency against malicious attacks, including ransomware, malware and hacking attempts and to discover potential vulnerabilities and security weaknesses in your security configuration by employing techniques such as penetration testing to evaluate IDS/IPS, Web Application Firewall protection, SIEM/SOAR/XDR systems. Azure Advisor, Azure Security Center, Azure Advisor, Microsoft Defender for Cloud and Azure Sentinel are all good tools to employ for security testing.

Site Reliability Engineering (SRE)

The success of your cloud solution depends on its reliability. Reliability can be broadly defined as the probability that the system functions as expected, under the specified environmental conditions, within a specified time. Site reliability engineering (SRE) is a set of principles and practices for creating scalable and highly reliable software systems. Increasingly, SRE is used during the design of digital services to ensure greater reliability. The degree of reliability that's required for a solution depends on the business context. Reliability is defined and measured using service level objectives (SLOs) that define the target level of reliability for a service. Achieving the target level assures that consumers are satisfied. The SLO goals can evolve or change depending on the demands of the business. Another important term to note is service level indicator (SLI), which is the metric that's used to calculate the SLO. SLIs are based on insights that are derived from data that's captured as the customer consumes the service. SLIs are always measured from a customer's point of view. SLOs and SLIs always go hand in hand, and are usually defined in an iterative manner. SLOs are driven by key business objectives, whereas SLIs are driven by what's possible to be measured while implementing the service. The relationship between the monitored metric, the SLI, and the SLO is depicted below:

Refer to the Reliability and Performance Efficiency pillars of Azure Well Architected Framework for guidance on building scalable and reliable applications.

Chaos Engineering and Azure Chaos Studio

Chaos engineering is a methodology that helps developers attain consistent reliability by hardening services against failures in production. Another way to think about chaos engineering is that it's about embracing the inherent chaos in complex systems and, through experimentation, growing confidence in your solution's ability to handle it. A common way to introduce chaos is to deliberately inject faults that cause system components to fail. The goal is to observe, monitor, respond to, and improve your system's reliability under adverse circumstances. For example, taking dependencies offline (stopping API apps, shutting down VMs, etc.), restricting access (enabling firewall rules, changing connection strings, etc.), or forcing failover (database level, Azure Front Door, etc.), is a good way to validate that the application is able to handle faults gracefully. Azure Chaos Studio is a managed service that uses chaos engineering to help you measure, understand, and improve your cloud application and service resilience. Chaos engineering is a methodology by which you inject real-world faults into your application to run controlled fault injection experiments. Chaos Studio helps you avoid negative consequences by validating that your application responds effectively to disruptions and failures. You can use Chaos Studio to test resilience against real-world incidents, like outages or high CPU utilization on virtual machines (VMs).

In Azure Chaos Studio, you first select and enable your targets (either service-direct targets or agent-based targets, such as VMs and VM Scalability Sets).

Then you create Chaos experiments on selected Azure resources, as shown in the example below. Each experiment can introduce either faults or delays, to test the resiliency of the overall infrastructure in relation to the tested Azure resources.

Infrastructure as Code (IaC) testing frameworks and tools

The most prominent Infrastructure As Code languages in Azure are Azure ARM (JSON), Bicep and Terraform. Each of them has its own methods and tools for testing the IaC configuration. - Terratest is a tool to be used for testing Terraform modules. Terratest is implemented as a Go library. Terratest provides a collection of helper functions and patterns for common infrastructure testing tasks, like making HTTP requests and using SSH to access a specific virtual machine. - Bicep code can be tested by using Azure Pipelines or Powershell PSRule or even combine PSRule in Azure Pipelines. - ARM templates can be tested by employing the ARM template test toolkit. The Azure Resource Manager template (ARM template) test toolkit checks whether your template uses recommended practices. When your template isn't compliant with recommended practices, it returns a list of warnings with the suggested changes. By using the test toolkit, you can learn how to avoid common problems in template development. You can also test ARM Templates by using Pester and Azure DevOps. - Use Pester for Powershell or Powershell DSC scripts. - - Azure Policy Test Framework is a command line tool to test Azure Policy relying on Terraform + Golang.

Testing script template

You can find a free testing script template in the "Free Downloads" section.

References

Robotic Process Automation, or RPA, is increasingly embraced by business enterprises to automate repetitive and mundane activities related to software development and testing. The technology can be categorized as a disruptor given its estimated market value to touch $2,467 million by 2022 at a CAGR of 30.14% from 2017 to 2022 (Source: marketsandmarkets.com). The benefits of RPA technology for the industry are immense and include the ability to reduce operational costs, create business value, increase productivity, enhance quality, and others. However, notwithstanding the benefits that brought the technology to the center stage, it continues to be plagued by a serious problem – the breaking of bots due to change.

For instance, users may seldom notice changes in the software interface, such as login to logon or username to userid. Besides, these changes won’t get in the way of users’ completing specific tasks. However, when it comes to the RPA bots, they won’t budge or execute a particular task on noticing a change. In such a situation, the developers must dig deep into the problem to fix it. If this is the scenario for just one change for one process, then imagine the situation where hundreds of processes are involved.

Developers have to willy-nilly edit all processes to resolve a single change to one object. And should such changes occur frequently because of today’s dynamically changing IT environment, one can only imagine the prospect of making so many changes to the UI or connected APIs, among others. The result of such a situation will be the faltering of automation strategies, downtime, and lost business value. So, what needs to be done? The answer lies in crafting an RPA testing strategy that is resilient to change.

RPA testing strategies to make bots more adaptable?

To mitigate the fragility of RPA technology and drive efficiencies, RPA testing solutions should include the following:

Change management and resilient automation: Enterprise RPA should leverage a model-based intelligent object framework to streamline the maintenance of automation suites. Here, the object action framework, instead of manually updating every instance of a change across applications and processes, applies a one-to-many approach to replicate the change automatically across the software. In the RPA software testing model, a single definition is attached to a particular object. So, applying a fix to that definition can fix every bot or process that uses it. Thus, by fixing the central definition of an object, the effort to find the processes that use the object is eliminated.

Codeless automation: Debugging codes written by others is time-consuming and complex. It’s no wonder codeless technology is gaining popularity across the board for building applications. Also, enterprises do not need to write code to garner the benefits of RPA anymore, for there are codeless solutions to simplify the process of building and maintaining bots. This also enables non-technical users to become involved in bot maintenance.

Scalable automation: The true value of automation lies in reusing and sharing it across the value chain. Further, building an easily maintainable automation process is different from executing a thousand processes in an hour. In other words, automation can only benefit if it is scaled across the enterprise, especially to complex and high-volume processes. So, any RPA testing strategy should check the scalability of automation to processes and whether the outcomes of such processes are accurate and consistent with expectations.  

Integrated automation: Any automation process running in production should not fail, for the consequences can be dire for the entire value chain. Robotic test automation has become an efficient approach to executing testing in production environments. It can validate complex business processes across the production pipeline. Given that most applications and their processes are highly integrated within the production environments, the robotic process automation approach should apply to all applications.  

Conclusion

There should be a synergy between RPA and testing in order to automate testing and apply the same across processes, teams, and projects. It should enhance the capabilities of business enterprises for governance, RPA orchestration, and security. Also, by executing the automated process in testing in the same way you would be running in RPA, you can ensure the bots work properly. Finally, should you want automation to deliver ROI, it is better to implement a change-resilient robotic process software testing solution. This allows you to make the best use of your time while expanding RPA and adding value to your business.

Resource

James Daniel is a software Tech enthusiastic & works at Cigniti Technologies. I'm having a great understanding of today's software testing quality that yields strong results and always happy to create valuable content & share thoughts.

A Practical Guide to Chaos Engineering

Watch this video of cigniti’s blog to learn more about chaos engineering and testing tools for cloud deployment.