Software Testing Gallop

The proliferation of the Internet of Things (IoT) and its related technologies and software applications has confronted the testing teams with a number of challenges. With IoT, where most of the everyday objects communicate with each other, and where technologies upgrade every other day, new challenges have emerged for the software testing industry.

For any given software project, QA teams need to ensure that a single app functions as designed and performs as per  the user expectations. As the IoT devices seamlessly connect and communicate with a wide variety of platforms, devices and operating systems, the additional stress on the testing team is enormous. However, such challenges are completely outweighed when it comes to the security challenges posed by the software architecture of the IoT application. With innumerable devices connected to each other in the IoT, there are numerous potential attack vectors that can be exploited by the cyber criminals. If cyber criminals were to tamper with the settings of such devices, the consequences could be highly devastating.

With the amount of detail that needs to be covered in an IoT testing, having a list of critical QA priorities is very important. Such priorities need to cover the length and breadth of the challenges of IoT testing.

The different areas of testing an IoT application are as follows:

-      Functionality: Web/UI, embedded and back-end computing.

-      Compatibility: Multiple Configuration, Protocol and product Versions, Backward Compatibility and Mobile OS.

-      Connectivity: Communication between the devices, third party applications and the infrastructure.

-      Performance of the network, communication and internal computation.

-      Exploratory: Rare scenarios and beyond functional requirements and structured testing.

-      Security: Privacy, Autonomy and Control.

The top challenges and solutions that the QA can enlist are as follows:

-      The convergence of Hardware and Software: The ecosystem of IoT is made of devices, sensors and applications. When it comes to millions of sensors and devices in conjunction with intelligent software, only functionality validation is not enough.  

-      Additional factors along with a working system: IoTworks with sensors and devices communicating with multiple software back-ends with complex algorithms.It is not only enough to ensure a working set of device software. It requires a robust validation process with extremely complex real time scenarios. Obtaining such scenarios is an extreme challenge for the QA team.

-      Sensor Interactions: The biggest challenge lies in the creation of an environment in order to test the real-time implementation of the IoT application. Setting up such an environment requires numerous analytics engine and a significant experience in technical simulation. Although the hardware and the protocols are well tested beforehand, understanding the application intelligence and the device complexity is an extremely big challenge for the QA experts.

-      The device interaction layer: Here the software and hardware components interact in a real-time IoT environment. Their conformance to standards, interoperability, backward compatibility and security are the additional challenges in addition to typical software testing.

-      The user interaction layer: The success of the overall application depends on the user receiving a seamless experience. The main testing challenges include the simulation of network modes along with device level validation, the real time usability along with the back-end IoT environment.

Conclusion

The concept of cloud computing is gaining a lot of popularity due to its ease of access, security and disaster recovery process amongst other things.

With cloud computing, companies can store, manage or process all their files and data using internet servers instead of local storage. Due to a lot of benefits offered by cloud computing, companies have also started using this technology to test performance of software and applications. Cloud testing includes testing of applications using cloud based resources (hardware, infrastructure and software) for a cohesive, end-to-end solution.

Why Use Cloud Testing

Previously, testing of applications was done internally by the testing team of a company. But as more and more services are moving to the cloud, the testing process is becoming a lot better. There are numerous benefits of using cloud software testing compared to regular testing methods.

-  Lesser cost of testing: Due to the flexibility offered by the Cloud, the computing resources can be easily scaled up or down whenever required to provide a cost-effective website and application performance. Companies essentially have to pay for just what they use. Unlike the regular in-house testing process, businesses do not need to invest a lot of money to set up hardware and infrastructure for long term testing. They can go to the cloud provider directly, who takes care of the tools and licenses.

-  Test under different circumstances: Cloud infrastructure is built to handle different testing scenarios for optimum results. Companies can replicate virtual users in various locations across the globe to test how the application responds when used far and wide. Cloud services also support enterprise testing due to its scalability.

- Faster testing: Tests can be run virtually on-demand with scalable Cloud infrastructure. Companies can easily schedule testing times and resources are allocated accordingly. This eliminates long delays as is seen with hardware allocation and deployment procedures of in-house testing.

Challenges in Cloud Testing

Cloud testing can be an effective way to test applications and websites for businesses. But just like every other testing process, it has its own set of challenges.

-  One of the major cloud testing challenges faced by companies is the level of service provided by the vendor. Often after a company has moved a major part of its operations to the Cloud, it cannot be accessed when required. The Cloud service provider can thus cause delays in the testing process.

-   With cloud testing it can be difficult to isolate root causes for multiple problems encountered in load testing. Also, when trying to precisely measure effects of minor changes made to application code or its settings, cloud testing can make it difficult due to differences in internet traffic and bandwidth availability.

-   Sometimes during load testing, one particular IP address can generate a lot of traffic. Apparently, security teams identify the pattern to resemble a DoS attack and shut down the tests. In such cases, the traffic might even be real and lack of a proper testing strategy or schedule can cause undue delays.

Best Practices in Cloud Testing

1. Preparing for a two-stage process: The initial stage should be for internal testing, where minor performance issues can be sorted out under medium load. Companies should proceed to the second cloud testing stage only after preliminary internal tests, by increasing the load gradually for large-scale application testing.

2. Using multiple cloud providers: Using different cloud providers enable companies to test in more geographical locations and get more realistic results. Also, the testing can be more scalable. Network issues can also be sorted at the provider level in this case.

3. Keep your data secure: It is wise to encrypt data transferred between controllers and load generators. This secures both the data sent and received. Also, securing the load generators with their own firewalls prevents external threats.

In a world of Cloud Computing, it can be extremely beneficial for large companies to move their application testing to the cloud.

As Cloud based applications are going mainstream, the quality assurance and the testing teams need to revise and innovate their testing skills and processes in order to ensure a timely and effective application deployment and return on investment. The Cloud technology has already caused disruptive effects on the technical aspect of the business landscape. The most critical change exhibited by the Cloud is the amount of power and flexibility given to the  designers, content creators and the end users. Here, the traditional role of the developers, testers and end users have pushed their limits and are eventually becoming interchangeable in many respects. The multi-tenant architecture helps experience  the full potential of the Cloud. Here, each client shares computing resources provided by the service provider, but with separate data and code bases. The following can be considered as the main steps in the Life Cycle of Cloud Application Testing

-      Requirement Specifications: Specifications, workflow description and security grid reference created by the business analyst

-      Development: Significant Testing is done in this phase before the actual testing cycle starts. Testing performed – Field validations, Formula Validations, Workflow validation, Security validation and web service validation

-      Testing: System testing requires lesser time than usual. Testing performed: Integration Testing, Functional testing, Regression Testing and user acceptance testing.

The Cloud Testing approach: An effective cloud testing approach can be planned through the following steps:

Verification of Field     Properties

Verification of formulas     associated with fields

Verification of triggers and     workflows

Verification of data security

Verification of web services

Many companies are willing to be more adventurous regarding Cloud application testing. Factors which account for such an approach are:

-      Almost half the technology infrastructure for testing remains underutilized, consuming capital, power and space. Testing is a period activity and requires new environments to be set up for each project.

-      Cloud Software Testing can be seen as safe, as it does not involve sensitive corporate data and has minimal impact on the day to day business activities of an organization.

-      Applications are becoming more and more dynamic, complex, distributed and component based, creating multiple challenges for  the testing team.

-      Cost reduction is one of the main factors that is encouraging the companies to move to the Cloud. Cloud Software Testing leverages the Cloud computing infrastructure, which reduces the unit cost of computing and increases effectiveness in testing.

Challenges of Cloud Testing

-      Security is one of the primary concerns in Cloud testing. User privacy protection, security standards on the Cloud, security of applications currently running in the Cloud and the security testing techniques are some of the main issues of Cloud testing.

-      The performance of a shared application is another big challenge. As the application is shared across many users it could lead to delays.

-      Certain configurations may not be supported by the Cloud service provider. This makes it difficult to emulate the customer environment settings.

-      Integration testing is another challenge in the testing process on the Cloud. Interactions between the network, database and servers pose testing challenges under potential risks such as crashes, network breakdown, etc.

Conclusion

Technologies have adopted and aligned themselves to meet the ever-changing business demands and in this process newer techniques and fundamental methods have evolved in software design and development. The evolution of micro-services has been the result of this process and has in turn introduced many challenges in testing applications designed with a micro-service architecture.

Micro-services break down the software architecture of an application into smaller units as per its  business needs. This results in systems which are more flexible, highly resilient, easily scalable and can be developed by separate sets of individual teams. It is a difficult task to formulate effective testing strategies for applications designed with such an architecture.

Challenges of testing API and Microservices

-      As micro-services are developed by teams working on multiple technologies and frameworks, and are integrated over light-weight protocols, the testing team inclines on using the Web API testing tools which are built around SOA (Service Oriented Architecture) testing. This however may not be cost effective as all services have been developed by separate teams and the time available for testing all of them is not guaranteed.

-      Each micro-service is considered to be independent of each other, although they are interconnected to one another. Hence, the main challenge here is to be able to define the right amount of testing required at each point in the testing life-cycle.

-      When these services are integrated with another service or API exposed to the external world, then a simple API testing tool proves ineffective.

-      With micro-services, there is no requirement for a service level aggregator and data storage is managed by individual units. This complicates the extraction of logs during testing and data verification.

In order to overcome the challenges listed above, Mike Cohn’s test pyramid can be used for understanding the amount of testing required.

This pyramid emphasizes on a bottom-up approach to testing, the number of tests and the efforts to be factored in each stage.

Testing between Micro Services and Third Party API

Let us review a scenario where a service in an application consumes or interacts with an external API. The testing strategy in each phase of the test cycle can be defined as follows:

-      Unit Tests cover testing for various functions defined by the service. Including a Test Driven Development here ensures that the requirements are clearly validated during unit testing. Unit Testing also ensures data persistence within the service and its passing on to other interacting services.

-      Contract Testing: This is a contract integration testing and needs to be verified with a live instance of the external service periodically. This testing is done in order to ensure that there is no change in the external service that has been published and used by the user.

-      Integration Testing: Live integration testing is not recommended. However, if API Sandbox is not available for testing, then the integration contract testing needs to be performed thoroughly for verification.

-      E2E Testing is performed in order to ensure that there are no failures in workflows that may be integrated with the internal service. A few monitoring tests can be set up to ensure there are no uncovered inconsistencies. E2E tests can be run at regular and pre-defined intervals to spot any changes.

Conclusion

One of the most important issues faced by the retail industry is digital transformation. The pressure, which has been created by the digital customers, is forcing the brands to adapt to the transformation. It is an intensive transformation of the business activities, processes, and models in order to leverage the advantages of digital technologies in a strategic way. They have revolutionized their engagement procedure with the customers.

Along with the digital transformation of the business, software testing also needs a parallel transformation. Testing and Quality Assurance face many challenges, as it requires tackiling  two divergent demands of the business. The first demand refers to the greater centralization of testing resources, whereas the second demand is to ensure that this centralized testing and QA is able to support the digital agenda of a business.

The changing demands of the customer is intensifying global competition. The reach of social media channels and new trends such as the cloud and mobile are a few challenges, driven by the market.  These need to be dealt with, by an organization, on its road to digital transformation. Customers always expect a good quality and seamless experience. The digital touch points of potential customers help in introducing  the value into the entire process.

To cater to different emerging technologies, the skill sets of an organization requires keeping up in order to leverage the market and and consumer’s behavior;  and resultantly foresee the future possibilities.

The ultimate goal of a business is to ensure a consistent user experience across all channels of interaction. Applications need to be uniformly available across all categories of devices that provide seamless user experience. Hence, the QA team needs to  use optimized test automation tools for testing across all devices, platforms and operating systems. The ability to strategize automation testing and manual testing is a key factor of consideration in the digital transformation of a business.

Higher process maturity is another key challenge in order to adapt to the latest technology. Quality assurance is an inevitable phase for establishing digital transformation more effectively.

Few additional challenges of testing in digital transformation are as follows:

Budget Pressure: There exists the forever requirement of expecting more from the less. Furthermore, as testing is continuous, the investment is required for it is also continuous.

Technical Diversity:  With every release, there are changes in device features, OS support, browser display, etc. Even if the application features are not changed, testing is still required in order to ensure their seamless performance in the newer versions.

Time to Market: It is a constant race against time, to deliver newer features earlier than the others, to deliver extended innovations. This race starts with design phase, then comes development and then testing. Releasing a software application earlier and faster than the others is a constant goal and challenge.

STLC: In order to achieve all the above factors, the software testing life cycle has evolved with testing at different places in each of Waterfall , Agile and DevOps. Being able to adopt and implement the best STLC as per the organization structure is also one of the key testing challenges before a digital transformation.

Conclusion

Not very long ago, testing was not even considered as a separate discipline in software development. One global trend in this era is the increasing qualitative growth. The focus along with production of more products is also on the production of better products. The importance of testing will be increasing remarkably from 2020. This is attributed to the increasing operational risks, increasing connectivity and data fluctuation. The software will become a stronger link between the larger parts of the economic system. Hence systematic and permanent testing will be indispensable for secured functional and operational reliability.

Data protection and security against potential cyber attacks are more important than ever. Hence. 2020 will see an overwhelming demand in security tests. Network and data security testing service will no longer be separated from the quality assurance catalog. Along with the increase in extensive testing services, the number of testers in the Quality Assurance(QA) sector will also increase at a rapid rate. With the increasing variation in testing types, methods and tools, the requirement of specialized testers will rise.

Testing mechanisms such as crowd based testing and classical testing automation are only partially capable of coping up with the growing test needs. Hence, automation testing is sure to grow with standards predefined for testing tools.  The shift in software development methodology  will possibly create a wide space for specialized testing in the testing sector.

The greatest area of concern, digital security, is eventually getting out of control, despite all the preventive measures taken by the state and the private industry. This situation will eventually escalate with the knowledge of new mathematical methods which will allow to break complex cryptographic encryption.

Hence, security testing is looking forward to a substantial boom. This has made the security testing companies the targets of cyber attacks for picking up the know hows. The volume of data in the cloud has made it an important target of cyber attacks.

With all the economic and social development depending completely on the reliability and the security of the software used, the testing sector is set to record an unparalleled boom. Technical systems will be increasingly equipped with Artificial Intelligence. The software test tools keeping pace with this development will also be equipped with AI.

Conclusion

Initially, testing has been just another service provided by a software development company. But the shift in market demands revised the business models and placed testing and QA into the on-demand category. TaaS or Software Testing as a Service is basically an outsourcing concept where testing activities are outsourced to firms who specialize in simulating real world test environments as per project requirements. It thus compartmentalizes the development and maintenance phase of a project. This has provided an advantage to the client to pay for what they need and recurring them as required.

Applications, now a days, have become more complex with complex business processes which call more more complex testing methodologies for software quality assurance without affecting the return on investment . Though automated testing tools are a solution to this, but the investment and skills required to implement it is most of the time unaffordable for some companies. Thus, TaaS solutions are a better bid to overcome these obstacles. It’s a best solution for all types of testing.

Types of Software Testing as a Service(TaaS)

- Functional / FTaaS: This includes graphical user interface, regression, integration and user acceptance testing.

- Performance / PTaaS: Load and Stress testing is performed by creating virtual users to simulate a real world user environment.

- Security / STaaS: The applications are scanned for security loop holes and vulnerabilities.

Features of TaaS

- It is an independent portal which runs an application for functional and performance tests.

- It has a test library will complete security controls and saves all the test assets available to the end users.

- It detects potential performance bottlenecks in the system through load and stress testing.

- Its metering capabilities track the services used by the clients and charges them accordingly.

- It has an on-demand availability of highly functional test labs which have the ability to deploy complex software applications, execute test scripts and run test tools.

- The use of cloud resource pools further maximizes the hardware utilization levels.

An explicit TaaS solution may not be suitable for all kinds of applications as it involves engaging skilled and expensive consultants or sometimes just outsourcing the entire testing phase to a firm. It is most suitable under the following circumstances - 

- An extensive simulation of real time environment is required.

- Requirement of expertise and availability of automated testing tools. 

- Requirement of extensive resources for irregular testing.

Conclusion:

The overall objective of both Quality Assurance and Software Testing is Software Quality. However, the fundamental difference between testing and quality assurance is stated as follows:

Quality assurance defines the appropriate activities that ensure quality is built into the product. It is a set of activities which are followed methodically to ensure that the development and maintenance processes are adequate to meet the objectives of the system efficiently. QA reviews the different elements of a project.

Testing is performed after the entire product is built. It is a process of executing the system with the intent of finding errors and system vulnerabilities. It also includes test planning before test execution. Another term usually confused with this context of discussion is Quality Control. It is a set of activities which are defined to ensure whether  the developed software meets the customer requirements. Testing is considered to be a part of Quality Control. However, not all the activities in QC are testing. Hence, we can say that testing is an activity under QC.

The final objective of Quality Assurance is to provide assurance of the product quality. It consists of activities which usually precede the development stage of the product and continue throughout the development process. QA focuses on prevention of defects, processes and continued improvements of these processes.

There can be a confusion as to who is responsible for QA and who is responsible for Testing. This decision usually will differ as per the organization, its resources and development methodology under practice. Matured organizations implement a segregated Quality Assurance and Testing practice.

As the two concepts differ, so do the responsibilities associated with each of them.

Testing teams document test reviews, plan test, analyze and design tests, verify and validate tests and test the reporting through various test levels. The QA team on the other hand performs the following functions:

• Implements standards and policies for quality • Prepares the project quality solutions • Ensures that the software processes conform to quality plans • Conducts regular process and product audits and presents the assessment report to the senior team. • Assures quality throughout the software development lifecycle

As far as the project documentation is considered, the test team prepares the test documents and plans based on business requirements and design documents. Test documentations cover test processes, test approach, detailed schedules of test, environment requirement, test management and reporting. The QA documentation deals with an extended set of activities. It covers project development methodologies, acceptance criteria, audit processes, configuration management and quality policies.

Conclusion

A test script in Salesforce has two parts - Tests and Incidents, which can be best managed with Google Spreadsheets.  The tests define which process is being tested, the steps to be followed, expected result vs the actual results, the overall result and the incident number in case the test script fails. When a test script fails, an incident is created in the Incidents Tab. It includes the incident no., step no., the description of the actual result compared to the expected result, the cause for the failure and its potential fix. Once the fix has been made, the test script is re-executed. It is appended as a new row in the google spreadsheet. If the test fails again, it is again logged into the Incidents tab and the process repeats itself until the test passes. For a test script to pass, all the steps of the test should be executed as it is expected.

- Purpose of testing

-  To check the functionality of the configuration and the code. -  To ensure that the agreed upon requirements are met by the system's initial build. -  To help control the actual scope of the project. -  To confirm that the business process of the client can be supported by the developed system. -  To gain approval for releasing new functionality. - Types of Testing

Unit Testing: This test is performed by the Apex developers. This process involves writing clauses or conditions in the code that tests its coverage automatically. It tests the threshold limit of the unit code in a given environment. To deploy the Apex code into production, a 78% test coverage is required.

System Testing: This test is performed by the salesforce consultants. It thoroughly checks the technical processes of the system. It is performed by executing a test script based on specific outputs. Functional Testing: This is tested by the users of the application. It tests the ability of the system to support the business processes. This test is intended to achieve the client's confirmation on the system.

Production Testing: This is system test, re-executed in the production environment. It checks if the configuration and the code files have been successfully transferred to the  production environment from the sandbox environment. Also, the User Acceptance Testing is also performed in full scale in the production environment.

Regression Testing

This testing can also be referred to as impact testing by the client. This tests if the newly deployed codes and configurations affect the existing processes of the system in  any way. Many releases of the project may not be intended for all the users. Hence, this test ensures that the process of the users, for whom the new processes have not been made accessible, remain unaffected.

Conclusion:

Software Testing is used to identify the accuracy, completeness and quality of a software application. It executes the software under several positive and negative conditions and checks for its compliance, functionality and performance.

Understanding Error, Bug, Fault and Failure The human action that results in an incorrect output  is an error. The presence of error while software execution is called a bug. The state of software with an error/bug is a fault.This event of deviation in software functionality from its intended result is called a failure.

Test:

A test is an action taken to find out if something is working as it is supposed to work. It involves simulating situations under  which the object is supposed to work.

Test Plan:

It is a descriptive document where the scope of testing, approach, resources, schedule and its related activities are described. The format and the content of a test plan can differ based on the processes, standards and the testing tools being used.

Test Case:

It is a set of conditions or input parameters which is used while execution. The actual output is then compared with the intended output to determine whether the system works as required.

Test Script:

A test script is used in automated testing. It is a set of instructions which are executed in an application under test. Many test automation frameworks can record/generate the test scripts.

Methods of Software Testing

- White Box method: Also known as open box testing, in this method, the internal structure of the application under test is known to the tester.

- Black Box method: This is a testing method where the internal structure of the system is not known to the tester.

- Gray Box method: Being a combination of white box and black box testing, here the internal structure is partially known. It implies black box testing with access to the internal resources.

- Agile method: It follows the principles of the agile methodology and testing is carried out throughout the development process.

- Ad hoc method: This refers to random testing without any documentation or proper planning.

Levels of Software Testing

The different levels of software testing are as follows:

- Unit Testing: Here individual methods of the application are tested. It is performed by using the white box method.

- Integration Testing: Here the individual units are combined and tested as a group. Here, any of the white box, black box or grey box methods can be used for testing.

- System Testing: Here, the complete system is checked. This evaluates the compliance of the system with the requirements specified. Generally, the black box method is used to carry out this test.

- Acceptance Testing: This evaluates the system for its acceptability by checking its compliance level with the system requirements specification. Here, black box method is followed for the testing.

Types of Software Tests:

- Smoke Test - Functional Test - Usability Test - Security Test - Performance Test - Regression Test -Compliance Test

Phases of Software Testing

A software testing life cycle, STLC, defines the different phases in software testing. Although the phases are the same, but their sequence differs as per the Software Development Life Cycle(SDLC) followed and the management decisions. The phases may not be sequential as shown. They may also be carried out in parallel as well. The general phases of any STLC are as follows:

- Requirements Review - Test Planning - Test Designing - Setting up Test Environment - Test Execution - Reporting

Conclusion

Software Quality Assurance ensures that the developed software complies with the specified functional requirements. SQA is a continuous process performed all throughout the Development Life Cycle to check that the software under development meets the standard quality measures. It very clearly and adequately  defines and expresses the problems and their solutions. Its main activities are process definition, process implementation, auditing and training.

Software Quality Assurance is practiced regardless of the development model being followed. It implements testing methods and checks for the quality of the software in each of its development phase. It involves monitoring and improving the development process and makes sure that the problems found have been dealt with.

It addresses the following issues:

- The developed software is a physical realization of the customer requirements.

- The software can be verified as per the physical realization.

- The development costs, including testing, debugging and redevelopment are visible and readily categorized.

- The benefit of the software is visible to its customer/user.

To identify the costs and benefits of a software, a number of software characteristics are defined which have been termed as quality attributes. The common definition of these software quality attributes was developed by Robert Grady at HP. This is known as the FURPS Model. This model has been defined to allow each characteristic of the software to be measurable by Quality Assurance.

The following characteristics have been defined under the FURPS Model.

Functionality: This represents the main features of the software as described in the requirement specification. Usability: This includes factors such as accessibility, consistency and interface aesthetics in the user interface.

Reliability: This considers aspects such as availability, accuracy and recoverability.

Performance: This evaluates aspects such as startup time,  recovery time, system response time and throughput of information.

Supportability:  This supports testability, adaptability, compatibility, scalability etc.

This plan for quality assurance also refers to other development plans of other supporting processes. These different plans include -

1. Documentation Plan 2. Measurement Plan 3. Risk Management Plan 4. Problem Resolution Plan 5. Software Development Plan 6. Configuration Plan 7. Test Plan

A Project Manager may not actually define the project goals, but he ensures that these goals are clearly defined, agreed by the client and included in the Software Requirement Specifications without fail.

Having a predefined policy statement of quality, by the development company, also forms the base for these definitions.  These specifications should be defined in quantifiable terms wherever possible. The role and responsibilities of the organization in achieving these goals should also be well defined.

Conclusion

Software Testing is a process to evaluate a system or its component(s) with the purpose to detect that whether or not it satisfies the specific requirements. Software Testing in simple words is the execution of a system to identify any missing requirements, errors or gaps in contrary to the actual needs or desire. Within the fundamental test process, the activities of analyzing a software item to find the differences between existing and required conditions (i.e. errors/ defects/bugs). In order to achieve Software Quality Assurance to the clients the total testing process goes through the following phases:

Test Cycle:

1. Test Planning

In this phase the Project Manager decides things to be tested along with the budget. Proper planning at this step can greatly lessen the risk of low quality software. This planning is going to be an ongoing procedure with no end point. Activities at this stage include preparation of high level test plan. The Software Test Plan (STP) is intended to set the approach, scope, schedule and resources of all testing activities. The plan should identify the items and features to be tested, the types of testing to be carried out, the personnel in charge of testing, the schedule and resources required to complete testing in addition to the risks associated with the preparation. Almost all of these activities are revolved around a software test plan.

2. Analysis and Design

This phase includes the following activities:

- Reviewing the test basis.

The test basis is the information required to initiate the test analysis and create your own test cases. It is basically documentation and test cases such as requirements, architecture and interfaces, product risk analysis and design specifications are based on it. The test basis documents can be used to understand what the system must do once built.

- Identifying test conditions.

- Designing the tests.

- Evaluating testability of the system and requirements.

- Designing the test environment set-up and identifying required tools and infrastructure.

3. Implementation and Execution

At this stage, the test situation is acquired in order to formulate them to test conditions as well as test software to create the test environment. Test implementation and execution is where the most visible test activities are performed and usually involve the following major responsibilities:

- Building and prioritizing test cases, preparing test data, writing test procedures, and optionally, creating test harnesses as well as writing automated test scripts.

- Accumulating test cases into test suites, where tests are run in sequence for efficiency.

- Checking whether the test environment set-up is accurate.

- Running test cases in the determined order, either manually or by means of test execution tools.

- Keeping a log of testing tasks, including the result (success/failure) along with the versions of data, tools, software, and testware such as scripts.

- Comparing actual outcomes with expected outcomes.

- Reporting inconsistencies as incidents in a detailed manner, including preferably causal analysis (inaccurate test specification, code defect, test execution or test data error).

- Repeating test activities where necessary when changes have been made, for instance, re-executing a test that previously failed so as to confirm a fix (retesting), executing a corrected test as well as previously passed tests to verify that defect have not been raised (regression testing).

4. Assessing Exit Criteria and Reporting

In this phase of the fundamental test process, test execution is evaluated against the defined objectives. The activities include the following:

- Verifying whether the previously decided exit criteria have been met.

- Finding out if more tests are required or if the specific exit criteria need changes.

- Writing up the outcome of the testing tasks for the business sponsors as well as other stakeholders.

5. Test Closure Steps

Software testing or Software Quality Assurance verifies and validates that the application or program fulfills its process requirements. It identifies the severity level of the errors in the system that requires fixing and defines its quality factors in terms of reliability, usability, security, portability and compatibility. Software Testing is generally limited within budget and timeline. Adhering to predefined principles ensures effective testing generating plausible results despite limitations.

Principles of Testing

A principle can be defined as an accepted rule or more appropriately a standard guideline which helps in carrying out an action effectively and consistently. Certain principles can be followed while software testing. In this article we have tried listing as many of them.

1. Test until the program fails – The true value of testing is achieved when errors are detected. Hence, this process should be executed with the intention of exposing failures to the maximum extent.

2. Initiate Testing Early – Initiating the process of testing early in the development cycle fixes huge errors and reduces cascading errors in the later stages. This also reduces the risk of reworking on the same code in the final phase.

3. Evaluate the Context – Testing differs as per the context of the software being developed. Example: the testing of a security software or a banking software will differ completely from that of a regular E-Commerce software. Also, the testing process in a waterfall model differs from the process followed in Agile methodology. The test objective as well differs at different points in a software development life cycle. The testing approach should thus be determined on requirements, risks, objective, knowledge, economy and time.

4. Define the Plan: Software Testing is carried out for achieving a specific purpose that should be documented in quantifiable terms. Hence test plans should be defined in terms of scope, objectives, strategy, environment, risks, schedule, tools, techniques and end deliverable.

5. Design Test Cases: The requirements should be well understood before designing test cases. The test cases should be written before the performing the tests. If there is confusion over the test requirements and product architecture then designing test cases will be ineffective.  A test case definition should describe the input data and the corresponding correct output. This documentation of the expected output is necessary to provide an unambiguous testing result.

6. Review Test Cases: Executing same test cases will not generate new errors after a certain period of time. Hence as debugging is carried out, the test cases should be revised to find more defects.

7. Use Valid as well as Invalid Conditions: Along with valid input values, the testing should also be performed with invalid or absurd input values. Invalid input cases detect errors at a higher rate by triggering failures.

8. Engage different Testers: As already stated, testing is context dependent and so is are its purposes. Hence testing at various levels should be performed by different people/team. This will yield better results.

Example: Acceptance testing should be performed by the users, unit testing should be performed by developers, release testing should be performed by Quality Managers etc.

9. Innovate: It is practically impossible to guarantee the absence of errors. Hence testing should be carried out more on the highly used vulnerable components as compared to the lesser used components.

10. Perform Cross Dimensional Testing: Static testing goes into the depth of the application by revealing the problems in the domain and the data structures. Dynamic testing covers the breadth with extreme input values that are not generally used.

11. Cluster the Defects: Errors keep occurring in the section where errors were already found. Hence such sections should be subjected to more rigorous testing.

12. Evaluate the Tests: Test cases should be evaluated to test its effectiveness. This is determined by its rate of error detection.

13. Perform Negative Testing: Focus is usually put on establishing the fact that the software performs as per the specifications. Another approach to test an application is by ensuring that it does not function the way it is not supposed to.

14. Conclude Realistically: Testing although a continuous process needs to be concluded somewhere. A balance should be reached considering the factors of time, budget and software quality.

While following the various principles, the limiting factors should also be taken into account. Independent Software Testing gives appropriate results if the principles are followed accordingly.

Conclusion:

Software Testing or Software Quality Assurance verifies and validates the software application against its initial technical requirements. It is used for determining software quality factors like reliability, usability, security, integrity, capability, efficiency, portability, compatibility etc. The procedures may differ as per the purpose for which the software is being tested. This also needs to be performed well within its specified budget and time limits.  

Software Testing Goals:

In testing, goals can be defined as intended outputs. It has the following goals:

Verify and Validate: Software testing is a quality control measure. It is used to verify that the application or product works as desired. It reports the status of the actual application in comparison to the requirements. Hence, it should reveal as many inconsistencies as possible, verify if the software is meeting its initial requirement specifications and further validate its quality.

Prioritization of Test coverage: As already stated, exhaustive testing is not possible due to many constraints mainly budgetary and time limitations. Therefore, the test coverage areas should be defined properly. Highly frequent functions should be tested more thoroughly than infrequent functions. For optimized results, highly used components should be isolated and tested more thoroughly.

Balance: Testing needs to be balanced between the negotiated requirements, the practical limitations and user expectations. The end users viewpoints cannot be all that matter as all the needs may not be achievable due to limitations in time, budget and technical feasibility.

Deterministic: While testing, it should be clear as to what is being tested and what will be its possible outcome. All the errors and defects should be exposed while performing a particular testing. This clean insight helps in determining the cost, time and the overall development.

Traceability: Comprehensive documents need to be maintained as testing is being performed. This helps to avoid chaos and hence eliminate duplicate testing.

The Limitations:

The limitation of testing needs to be considered to set a realistic expectation of its outcome. The main limitations of testing can be listed as follows:

- It is done with the intention of finding errors. Exposing failures make this process more effective. Thus it cannot show the absence of errors.

- It only delays the process. It provides no help to decide on the product's immediate deployment or to delay its release.

- It cannot ensure the overall compatibility of the system rather it establishes its compatibility only on a specified combination of the system.

- Software Testing does not enable locating the exact error. Rather it just establishes the existence of one.

- A total or exhaustive software testing is practically not feasible. This leads to compromise and thus it cannot be ensured that the final delivered product is entirely bug-free.

Conclusion:

Ad-Hoc testing is a kind where the tester performs the test without any prior planning. In fact, there is no adequate time for properly designing test scenarios with the desired results. In most of the cases, the tests are carried out without any documentation describing the basic specifications and requirements. While executing such tests the tester can easily adopt this approach to tests any part of the system and find out the hidden bugs, if any. Usually, the tester applies skills, intuition, experience and knowledge for the software testing. As far as applying this process is concerned, it is usually carried out under demanding project schedules. There are times when it is impossible to perform extensive testing. Under such circumstances, ad-hoc testings are adopted in place of conservative processes.

However, before indulging in its practice, it is always advisable to follow certain guidelines that will help the tester to acquire more this type of testing.

Get familiar with the software

Knowledge about the architecture of the software is essential to ensure success in ad-hoc testing. Hence, the trick is to get accustomed with the way the software functions.

Test properly

Merely going through the testing procedure is not enough. The best part of ad-hoc testing is that it allows you to test all the software parts without facing any kind of restrictions. The testers can utilize this factor in their favor. They can perform the tests on various levels of the applications. Moreover, it is equally important to focus on the areas that have more risks. Although these are performed without any planned process, an expert tester will always consider the risk analysis first. Hence, it is advisable to begin with the fundamental risk identification and analysis. Once the tester comprehends the necessary areas, he can move onto the bigger folds.

Compare with a tester who lacks knowledge

An adept tester has the desired opportunity to rely completely on his knowledge and ability. This allows him to think about the specific and related testing situations. Such a scope will enable him to look for more number of software bugs during the process.

Experience matters

If your tester has considerable experience, he will be able to find the bugs as compared to an inexperienced tester.

Never mind the numbers of tests

This is another factor that will have an impact on your success or failure. The number of tests decides how many bugs you come across. Mostly, when there is a tight schedule to maintain during project completion, several rounds of testings cannot be possible. But if you consider the best practices, the more tests you perform, the better will be the visibility of the bugs.

Benefits of Ad-hoc Testing

The ability to learn is the final word on ad-hoc testing. The capability of the tester to learn about the software is critical along with the ability to find out the bugs. While discussing about the best practices, the question might arise as to why is it necessary to opt for ad-hoc testing. Well, there are several advantages associated with this testing process. One of the major reasons is that it can be used as an additional testing layer during the regular testing processes. Secondly, this kind of testing offers quicker results and it can considerably improve the aim that could not be developed during the conservative testing methods.

Software Testing is not only being deployed to find defects in the software code but more importantly to prevent defects from cropping into the application in the first place. This is the primary reason for which software testing needs to be started as early as in the requirement analysis stage, but not at the end of the development. The perspectives resonate the demands of increasing productivity by agile developments, shorter builds and frequent release cycles. This objective of testing also influences the choice of the type of testing approach – manual or automated.

Manual testing plays a critical role where need for testing is exploratory in nature. Alpha and Beta testing are done by users in actual or simulated deployment scenarios.

Automation testing works fine with unit testing, functional testing, story acceptance testing and feature acceptance testing. However, still some part would always be checked manually.

Test automation has limitations when it comes to exploratory testing. Hence, in such cases, manual testing with a blend of automation capabilities delivers value.

  The role of the manual tester is being influenced by the demands of the rapidly evolving applications and IT processes with paradigm changes in the division of the workload into automated, semi-automated and manual. So a manual tester’s inputs go into making the test automation scripts to identify and execute the common defect trends in the system leaving the high priority test areas to the intellect of the testers.

  The software industry changes to adopt things like continuous delivery/deployment and reliance on an increasingly fragmented mobile platform set, the testing industry (i.e. the people that comprise it) likewise has to evolve to meet these new challenges. Initiatives like test automation, dog-fooding, crowd-sourcing, among others, are indeed all having very real effects on the role of the traditional ‘manual’ tester.

Automation Test Tools offer a competitive advantage and faster time to market. Various automation tools, available in the market comprising scriptless automation offer GUI to developers that helps them design automated test scripts for frequently repeated tests. While the features vary from tool to tool in the testing landscape, they can be easily contextualized for the business environment and deployed manually.

Automation testing is to assist the manual tester.

From a business perspective, the combination of Manual and Automation Testing is expected to deliver outcomes with cost-time benefit without compromising on the software quality. This means achieving the balance between time and efforts through a comprehensive testing strategy.

From the testers’ perspective, automation takes on the burden of recurrent test runs and reporting while freeing time to focus on exploring the defects that are unspecified in the automation scripts. More- over, test automation itself needs to be tested for its testability which makes it implicit for the testers to be oriented with automation initiatives.

 From the industry perspective, the adoption of crowd sourcing is providing great alternatives to accommodate increasing workloads and address the requirements of exploratory testing which is possible by manual testing.

The software testing landscape is consistently evolving owing to the complex software projects. In fact, the software industry has undergone several structural changes in the past few years. In this regard, 2014 was considered a dynamic year owing to the commencement of the SOA (Service Oriented Architecture), cloud testing and virtualization. Internet of Things (IoT) is another primary advancement that is gradually altering the ways in which organizations interact with their employees, customers as well as partners. A report released by NelsonHall in 2014 indicated towards a surge in the software testing domain. According to the report from this leading analyst firm, by 2018 the domain is expected to grow every year by 9.5%.

  So which software testing trends are all set to rule in 2015? Here are the top 10 trends that will definitely create an impact on the testing domain –

  Security Testing to become top priority

There are certain basic aspects that security testing covers. These include confidentiality, authentication, integrity, availability, authorization and non-repudiation. These tests are performed to judge the capability of the software or a system to prevent unauthorized access to data as well as resources. An enhanced connectivity of data and resources, rise of cloud computing and IoT will inevitably make security testing a priority for the IT organizations.

  Cloud Testing will be adopted rapidly

Cloud computing is another field that is increasingly becoming a part of the mainstream IT industry. It is estimated that by 2015 around 26% of the software applications will be hosted on the cloud. This is because cloud infrastructure provides suitable solutions that are cost effective and are easy to build and maintain.

  Big Data Testing to become more popular

The era of big data has arrived. This is a trend that is becoming popular rapidly. These tests are no more limited to test data chunks. Rather this test is applied to judge the accuracy of the information obtained. In 2015, the trend for Big Data testing is all set to grow.

  Mobile Testing

‘Go mobile’ is the 2015 mantra. This trend will have a huge impact on the software testing industry. The popularity of the handheld devices has led to a steep rise in the use of mobile apps. It is expected that by the end of 2015, the growth will shoot upto 95%.  

  Agile Testing development

Agile development methodology is applied to ensure a sound testing approach. 2015 will witness continuous testing a lot as organizations aim at reaching the delivery phase without waiting for too long. This will also result in testing in production as testers will be required to provide updates continuously regarding the progress on the software development. 

  IoT Testing to matter

Internet of Things includes everything that is connected to the internet. In the recent times it has witnessed growth as the world becomes interconnected. As more and more users expect hassle free performance of the apps and other technologies, the demand for IoT testing will increase.

  DevOps Movement

Organizations are looking for quick deployment that will be cost effective and of high quality. This is where the DevOps method has a considerable role to play. It belnds the tasks performed by the organizations, developers and professionals. Testing, in this case, plays a significant role. The demand for DevOps professionals will rise in 2015.

  Service Virtualization

It allows the DevOps teams to use the virtual services in place of the traditional production services. This helps in carrying out the tests comprehensively even if the important components are not present in the system architecture. With the DevOps movement gaining ground, this, too, will grow.

  Telematics Testing will ensure better IT systems

As technology keeps improving, it will have a huge impact on the travel industry as well. As a result, the automotive IT industry will grow in 2015.

  Crowdsourced Software testing – how good is it?

This is especially helpful when there is a lack of resources for testing the software in various situations and environments. Owing to its usability, it is gradually gaining popularity.