#agile test framework

As a Scrum Master you have been leading a team which works on both software and hardware components. Your organisation has decided to move to Agile framework from the pure Waterfall model after looking at the substantial advantages Agile brings to the table. You along with the team have enthusiastically supported this move as it helps you deliver quality software without the bureaucracy of Waterfall model. However, there is one fly in this ointment. What do you do with the phase gate testing strategy that your organisation operates? And mostly manual testing at that! How do you modify or reuse these in order to maximize your product quality? If you want to know of a template for such a transition, read on.

 The Template 

Phase gate test strategy - Waterfall model has strict quality guidelines and metrics which need to be followed in both spirit and letter. If an organisation follows Waterfall model or its multiple variants such as Spiral model, V model etc, it will need to have a test strategy document which gives details regarding how the organisation through its test team plans to implement the quality control it desires.

                                            OR

As can be seen from above diagrams, regardless of which flavor is used, Waterfall has extremely stringent requirements regarding the test strategies to be used. Hence, you would be having an established order of testing activities provided in strategy document which should have:

Test planning

Test Environment setup

Test tools and setup

Test case review

Unit testing

Functional testing

System Testing

Integration testing 

Acceptance testing

Stability testing

Certification testing

Resource planning

 The various levels of testing is targeted at various levels of development. There might be additions to this but in general, these or a combination of these activities would be listed in the strategy document. Now let us add the complexities of phase gate development to above scenario.

Various levels of test cases are designed during the design phase. Validation team comes up with Integration, Acceptance, System and Feature Test cases based on the requirements and developmental activity. Once these have been accepted through design review process, the testing activity starts. Please note that Unit Testing as an activity is retained with development team. Once the development team delivers features in Phase gates, feature testing takes place. In case System testing can start earlier than the Final phase, it is also started early. Integration and Acceptance testing occurs where multiple modules from multiple teams are integrated onto a single Product line. Stability testing starts only when the Product is completed. This then is a short primer about Testing strategies used in Waterfall.

And herein lies our problem. Following Agile principles there are no phase gates. The items delivered by Scrum Teams are expected to be Product increments. Ideally speaking therefore, once Product Backlog Items (PBIs) are delivered by teams, they are expected to be working as a final Product which is shippable. Hence, the delivered PBIs should have been completely tested through all test variants (Functional, System, Integration etc). How do we bridge the gap between this expectation and the existing reality of Phase Gate testing already available before the transition to Agile is completed?

 Implementation

The implementation of our modification needs to be done in 2 phases -

1. Add slices of test variants - Each Sprint consists of PBIs which are READY. To resolve the test strategy dilemma you need to ensure that no PBI is termed READY till it has a thorough test plan. The test plan should be drawn up under the guidance of the validation members who are part of the Scrum team (after all the team is cross-functional). There are 2 very important functions of the Scrum Master here:

Help in guiding the team to include validation members during self-organizing activities to make it truly cross-functional

Ensuring that Scrum principles are adhered to, where, testing is not a separate activity and needs to be carried out by each individual. 

The guidance from validation member is crucial at this juncture. She needs to ensure that all PBIs which are targeted for the Sprint should have been properly tested. Slices of test cases from Functional, System, Integration and if possible Stability testing pertaining to the specific Sprint Backlog needs to be added to the Sprint activities. Although all Scrum members are responsible for carrying out the testing, Validation members should ensure that the Definition of Done (DoD) for each PBI should ensure correct evaluation of test cases. However, there arises a different problem. 

Let us say you are in the Sprint planning session. PBIs 1,2 and 4 are marked as most important by the Product Owner (PO) and are READY. As per prioritization principle, Sprint Backlog should be populated with these 3 items.

However, validation members inform that PBIs 1,2 and 3 mark a block which can be adequately tested (through all variants). PBI 4 is from a different block which cannot be tested as standalone. What should be done in this scenario? The solution is to get PBIs 1,2 and 3 READY and in Sprint backlog. It is more important to provide a shippable increment of correct quality than an increment of low quality. 

Please note however, that this issue can be corrected once you move to the next phase of transition.

There is 1 significant item overlooked in above scenario. Stability and Stress testing! And I have not mentioned those to be added onto the DoD because it might be a bit difficult to do so. Scrum members doing manual stability testing for 24 hours - 48 hours do not tend to generate correct information. Hence, it is up to the Scrum team to decide how to introduce Stability and Stress testing for the PBIs they deliver. It might be that they slice up this variant as well and accept testing for up to 4-8 hours under stress scenarios. Or it might be that they completely give up stability testing to a system team which keeps on doing this activity throughout all Sprints. My suggestion is to go for the first option and introduce a degree of stability for your delivered increments.

2. Automate testing - As a Scrum Master you have introduced mini test planning sessions as part of your Sprint planning. Hence, you are now providing high quality shippable increments as part of your Sprint deliverable. 

There are a couple of problems still seen which is visible through your teams outwards facing artifacts. 

The velocity achieved by the team is generally low or       lower than achievable. 

There are defects which still evade the Scrum teams.       This impacts overall bandwidth and health of Scrum teams.

We need to now move on to the next phase of test strategy modification. Test automation activities need to be supported by the teams to rectify above mentioned problems. The goal of this activity is to establish a process, where the scrum team member should be able to trigger a CI build and automated testing from his work machine after developing code. Once the testing is over, it should give the confidence to all stakeholders that the testing has been thorough and will not cause defects, regressions or performance degradation in the product. Automation is a huge subject, the scope of which cannot be completely dealt with in this blog. Following link for Agile Test Framework gives more details. Suffice to say that Automated testing is essential in releasing the full potential of your Scrum Team.  

However, please note that introducing Automated testing is a decision for the management team and not for Scrum teams. The effort required for setting up Automation in some cases will be equal to the effort for the whole project. Such a large effort may or may not be desirable by all stakeholders. If possible C-level management needs to be involved to understand the utility of such a movement and cost.

One of the biggest arguments in favor of such a move is that without automation any movement towards scaling agile would not be effective.

A mechanism to introduce automation is to also introduce it through a phased manner where initially, Automated UI testing is carried out, which progresses to Automated service testing etc. This reduces the effort used for introducing automation while ensuring the system gradually moves towards a fully automated test strategy.

 Conclusion

​Before we conclude, we need to deal with a remaining aspect of our test strategy we have studiously ignored so far. The subject of Certification testing was left as this almost always assumes a philosophy of its own. Those of you who have handled Certifications by running and passing Certification test suites will accept that this is the proverbial 'Bull-in-a-China-Shop'. It has the power to wreck your whole product if issues are not handled early. However, to simply start such testing requires almost all other aspects of the product to have been assessed for quality. What should the mechanism be for assimilating Certification testing within Sprint planning? 

Certification testing is too large for it to be broken into slices and taken into Sprint planning depending on PBIs taken in Sprint backlog. Instead, during initial stages of Agile transitioning, specific Sprints should be run for Certifications.

However, once Automation is setup, Test Driven Development (TDD) can be used quite effectively to handle Certification test suite within Sprints themselves. Certification test suites come with the advantage that test cases are defined in detail to start with and more often than not, ensure that system is more thoroughly tested than any given test specification.