Integration vs E2E Testing: What worked for me as a charm?
When it comes to testing software applications, various testing techniques can be employed. Three common testing methods are unit testing, integration testing and end-to-end testing. All these different kinds of testing overlap, so you just can’t implement one or two forms of testing and you will be good. Like if you are in the initial stage of development you can write a few unit tests which a developer can run once he makes changes to a function. Further, if we have mocked most of the external dependencies we can check the compatibility between different modules and service layers. At last, we can do an e2e test to ensure the overall outcome of the application for a particular input.
In a traditional development approach, testing activities were often performed towards the end of the SDLC, after the code had been developed. This approach is known as a "right shift" because testing is shifted to the right side of the development timeline.
However, in recent years, there has been a shift towards integrating testing activities earlier in the SDLC, which is referred to as a "left shift." This shift aims to catch defects and ensure quality at an earlier stage, improving overall efficiency and reducing the cost of fixing issues later on.
In this blog, I will share my journey with different kinds of testing approaches and their importance in the software testing cycle. And discuss the differences between them. At last, I will talk about what can be a much more productive approach that helped me get rid of writing tests at all !!
When to do Integration testing?
So basically, as we know Integration testing is a testing technique that focuses on testing the integration of various components or modules of a system. This testing is performed after unit testing and before system testing. The primary goal of integration testing is to ensure that the different modules or components of a system work together as intended. Since this approach involves syncing different modules, instead of testing a particular functionality in isolation like in unit testing. In other words, integration is the sum of all the units that make a certain API work.
Years back Guillermo Rauch (founder of socket.io) who is quite active in the domain also tweeted that we should mainly write integration tests as helps to it ensure that all the components of the backend system work as intended and are compatible with each other. It's the whole application that needs to be tested not just a bunch of packages.
When to do End-to-End testing?
So, E2E is a testing technique that focuses on testing an entire system, from the start to the end. This testing can be performed after integration testing and before user acceptance testing. The primary goal of these tests is to ensure that the entire system behaves as expected and meets the user's requirements by simulating a real-world situation from the perspective of the end user. We can consider e2e testing as part of BDD. Usually in most organizations, QA and development teams collaborate to define features using BDD-style specifications, create automated E2E test scenarios, execute them against the application, and report results. This can be a cumbersome process as QA are usually not code-oriented persons. And many developers face a hard time whenever a regression is tested against a feature. As I promised we'll later In this section get to know how I generated my e2e test suite with minimal effort.
While E2E testing is typically performed towards the later stages of the SDLC, it is still considered a "right shift" activity. E2E testing involves testing the entire system or application as a whole, ensuring that all components and interactions work correctly from end to end. It often requires a stable, functioning system before it can be effectively conducted.
That being said, it's important to note that there is no strict boundary between left-shift and right-shift activities. The goal is to find the right balance between early testing activities, such as unit testing and integration testing (left shift), and later-stage activities like E2E testing (right shift), to achieve comprehensive testing and deliver high-quality software.
Priorities for Backend Testing
In the current scenario, where backend testing is concerned, integration testing should be given more priority. This is because integration testing focuses on testing the integration of different modules or components of a system, which is critical for the back end of an application.
Additionally, integration testing can help identify issues early in the development cycle, which can save time and money in the long run. It also helps to ensure that all the components of the backend system work as intended and are compatible with each other.
In my opinion, we should focus on writing more controller-level tests instead of internals like authentication because they are not exposed to the consumer of the app or library. As we are interested in behaviour that can happen right? It can be considered that if the product primarily focuses on the end-user experience and can't compensate on the response the user will get. Then E2E testing should be given more priority cause at last we have to run the application not just tests to meet the requirements of the user
You might have heard of two famous testing models Pyramid model and the Trophy model. The testing pyramid suggests that you should write more unit tests than integration tests, and more integration tests than UI tests. This is because unit tests are the most efficient type of test, and they can catch a lot of bugs. Integration tests and e2e tests are less efficient, but they can catch bugs that unit tests miss.
The Testing Trophy is a more comprehensive and effective model for software testing than the Testing Pyramid. The pyramid model does not take into account the return on investment (ROI) of different types of tests. If you are looking for a model that focuses on ROI and catches bugs early on, then the Testing Trophy is a good option for your team.
But, what if I tell you that there exists a tool that can invert the pyramid by making a test case of your end-to-end API flow, hardly in minutes and you don't even have to mock the dependencies or create a test environment as well? The problem of syncing the db state before performing testing queries is no longer a trouble for testers.
- reference from LUCA ROSSI blog
Key differences between integration vs E2E testing
Writing integration vs E2E testing are both important aspects of software testing, but they differ in scope and complexity. Here's my comparison of the time and effort spent on writing integration vs E2E testing, along with some relevant facts and considerations:
E2E tests: E2E tests cover the entire application or a significant part of it, simulating real user interactions and workflows across multiple components and layers.
Integration tests: Integration tests focus on testing the interaction between different components or services within the application, ensuring their proper integration and communication.
E2E tests: E2E tests tend to be more complex and involve multiple layers, user interfaces, APIs, and databases. They often require setting up test environments, configuring data, and handling various edge cases.
Integration tests: Integration tests are generally less complex and focus on verifying the interactions between specific components or services. They may involve mocking or stubbing external dependencies to isolate the tested components.3) Execution time:
E2E tests: Due to their broader scope, E2E tests often take longer to execute compared to integration tests. They may involve starting up the entire application and simulating user interactions, which can result in slower test feedback cycles.
Integration tests: Integration tests typically have shorter execution times as they focus on specific components or services. They can be executed more frequently during the development and integration phases.4) Maintenance effort:
E2E tests: E2E tests are generally flaky and more prone to breakage as they cover larger portions of the application and are sensitive to changes in user interfaces, workflows, or external services. Maintaining E2E tests often requires updates when the application undergoes significant changes.
Integration tests: Integration tests are usually more stable and less affected by changes in user interfaces. They primarily focus on the interaction between components, making them more resilient to UI changes. However, changes to integration points or APIs may still require test updates.
While E2E tests can provide a comprehensive check of the system, they can also miss some bugs, especially ones involving edge cases. Integration tests, with their more focused scope, are better suited to covering these cases.
On the other hand, integration tests may miss bugs that only appear when the entire system is working together, which E2E tests can catch.
That being said, the generally accepted view in the software testing industry is that E2E testing tends to be more time-consuming and resource-intensive than integration testing. But it's also considered a more thorough method that checks the system as a whole.
How you can reduce your testing efforts with Keploy ??
Now we got to know that e2e gives more confidence but writing it is hard, And if we can perform API calls (which we usually do before pushing code though) and get some sort of tests bound along with their mock data. This can be run along with your native testing frameworks like Junit, Jest, and Gotest and logs the result which shows the difference between the actual and expected responses. This can be pretty useful stuff for all the teams which want to deliver fast and are focused on the end responses for the user.
So in a nutshell, Keploy simplifies the process of creating realistic E2E tests by capturing actual outcomes from your real environment, whether it's a testing or production environment. This means that you can leverage the power of real data mocks, replicating the scenarios that your application interacts with, be it infrastructure calls like HTTP or database operations.
One of the major hurdles in E2E testing was the effort required to write and maintain the tests. Keploy addresses this pain point head-on by automatically capturing the interactions and outcomes of your API calls. These tests are then stored in a readable YAML file format, making them easily manageable with popular version control systems like Git. With Keploy, you have a clear history of your tests, making collaboration and tracking changes a breeze.
When running your application, instead of making actual outgoing calls, your application retrieves responses from the mock files generated by Keploy. This allows you to create a self-contained E2E test suite that accurately mimics how your application should behave after a successful release. By setting up these tests in your continuous integration (CI) pipelines, such as GitHub Actions or Jenkins, and executing them using Keploy's simple commands, you can ensure the reliability of your application with minimal effort.
Some glimpses of the tests and mock data generated by keploy can be shown in the form of YAML as -version: api.keploy.io/v1beta2 kind: Http name: test-1 spec: metadata: {} req: method: POST proto_major: 1 proto_minor: 1 url: http://localhost:8082/url header: Accept: '*/*' Content-Length: "33" Content-Type: application/json Host: localhost:8082 User-Agent: curl/7.81.0 body: |- { "url": "https://google.com" } body_type: "" resp: status_code: 200 header: Content-Length: "66" Content-Type: application/json; charset=UTF-8 Date: Wed, 27 Sep 2023 02:17:20 GMT body: | {"ts":1695781040077972081,"url":"http://localhost:8082/Lhr4BWAi"} body_type: "" status_message: "" proto_major: 0 proto_minor: 0 objects: [] assertions: noise: - header.Date created: 1695781040
And mock.yaml which comprises all buffer-decoded outgoing requests and responses looks like this -
So just perform a record session from real traffic and replay them exactly as they occurred. This makes it an invaluable tool for debugging hard-to-replicate issues that may be missed by traditional QA or even developers themselves. By leveraging real traffic data, you can gain deeper insights into your application's behaviour and identify potential issues more effectively. This empowers developers to release code efficiently as regression can be caught hand-to-hand. They just need to write some unit tests for some of their core methods and the rest can be covered through API flow via keploy. A collective coverage can give a big boost to the team for maintaining code coverage which can be considered a healthy dev practice.
In conclusion, both E2E and integration testing have their roles and neither can fully replace the other. They are both important parts of a comprehensive testing strategy and should be used in tandem to ensure a well-tested application. The time and effort invested in each type of testing can vary greatly depending on the specifics of the project and the team's expertise.
So when we have tools like keploy why we should not encourage them to perform a thorough e2e testing as we already "got everything right on as a test" ?? In my opinion, keploy is a boon to developers and testing teams as they can just simply run their application the same way they expected to and can see the code path covered for a particular test case.
In the next blog, we will discuss and showcase a sample application in which I created test cases and mocks without writing a single line of code with the help of Keploy V2. Check Keploy's GitHub repo here - https://github.com/keploy/keploy .
In a traditional development approach, testing activities were often performed towards the end of the SDLC, after the code had been developed. This approach is known as a "right shift" because testing is shifted to the right side of the development timeline.
However, in recent years, there has been a shift towards integrating testing activities earlier in the SDLC, which is referred to as a "left shift." This shift aims to catch defects and ensure quality at an earlier stage, improving overall efficiency and reducing the cost of fixing issues later on.
In this blog, I will share my journey with different kinds of testing approaches and their importance in the software testing cycle. And discuss the differences between them. At last, I will talk about what can be a much more productive approach that helped me get rid of writing tests at all !!
When to do Integration testing?
So basically, as we know Integration testing is a testing technique that focuses on testing the integration of various components or modules of a system. This testing is performed after unit testing and before system testing. The primary goal of integration testing is to ensure that the different modules or components of a system work together as intended. Since this approach involves syncing different modules, instead of testing a particular functionality in isolation like in unit testing. In other words, integration is the sum of all the units that make a certain API work.
Years back Guillermo Rauch (founder of socket.io) who is quite active in the domain also tweeted that we should mainly write integration tests as helps to it ensure that all the components of the backend system work as intended and are compatible with each other. It's the whole application that needs to be tested not just a bunch of packages.
When to do End-to-End testing?
So, E2E is a testing technique that focuses on testing an entire system, from the start to the end. This testing can be performed after integration testing and before user acceptance testing. The primary goal of these tests is to ensure that the entire system behaves as expected and meets the user's requirements by simulating a real-world situation from the perspective of the end user. We can consider e2e testing as part of BDD. Usually in most organizations, QA and development teams collaborate to define features using BDD-style specifications, create automated E2E test scenarios, execute them against the application, and report results. This can be a cumbersome process as QA are usually not code-oriented persons. And many developers face a hard time whenever a regression is tested against a feature. As I promised we'll later In this section get to know how I generated my e2e test suite with minimal effort.
While E2E testing is typically performed towards the later stages of the SDLC, it is still considered a "right shift" activity. E2E testing involves testing the entire system or application as a whole, ensuring that all components and interactions work correctly from end to end. It often requires a stable, functioning system before it can be effectively conducted.
That being said, it's important to note that there is no strict boundary between left-shift and right-shift activities. The goal is to find the right balance between early testing activities, such as unit testing and integration testing (left shift), and later-stage activities like E2E testing (right shift), to achieve comprehensive testing and deliver high-quality software.
Priorities for Backend Testing
In the current scenario, where backend testing is concerned, integration testing should be given more priority. This is because integration testing focuses on testing the integration of different modules or components of a system, which is critical for the back end of an application.
Additionally, integration testing can help identify issues early in the development cycle, which can save time and money in the long run. It also helps to ensure that all the components of the backend system work as intended and are compatible with each other.
In my opinion, we should focus on writing more controller-level tests instead of internals like authentication because they are not exposed to the consumer of the app or library. As we are interested in behaviour that can happen right? It can be considered that if the product primarily focuses on the end-user experience and can't compensate on the response the user will get. Then E2E testing should be given more priority cause at last we have to run the application not just tests to meet the requirements of the user
You might have heard of two famous testing models Pyramid model and the Trophy model. The testing pyramid suggests that you should write more unit tests than integration tests, and more integration tests than UI tests. This is because unit tests are the most efficient type of test, and they can catch a lot of bugs. Integration tests and e2e tests are less efficient, but they can catch bugs that unit tests miss.
The Testing Trophy is a more comprehensive and effective model for software testing than the Testing Pyramid. The pyramid model does not take into account the return on investment (ROI) of different types of tests. If you are looking for a model that focuses on ROI and catches bugs early on, then the Testing Trophy is a good option for your team.
But, what if I tell you that there exists a tool that can invert the pyramid by making a test case of your end-to-end API flow, hardly in minutes and you don't even have to mock the dependencies or create a test environment as well? The problem of syncing the db state before performing testing queries is no longer a trouble for testers.
- reference from LUCA ROSSI blog
Key differences between integration and E2E testing
Writing end-to-end (E2E) tests and integration tests are both important aspects of software testing, but they differ in scope and complexity. Here's my comparison of the time and effort spent on writing E2E tests vs. integration tests, along with some relevant facts and considerations:
E2E tests: E2E tests cover the entire application or a significant part of it, simulating real user interactions and workflows across multiple components and layers.
Integration tests: Integration tests focus on testing the interaction between different components or services within the application, ensuring their proper integration and communication.
E2E tests: E2E tests tend to be more complex and involve multiple layers, user interfaces, APIs, and databases. They often require setting up test environments, configuring data, and handling various edge cases.
Integration tests: Integration tests are generally less complex and focus on verifying the interactions between specific components or services. They may involve mocking or stubbing external dependencies to isolate the tested components.3) Execution time:
E2E tests: Due to their broader scope, E2E tests often take longer to execute compared to integration tests. They may involve starting up the entire application and simulating user interactions, which can result in slower test feedback cycles.
Integration tests: Integration tests typically have shorter execution times as they focus on specific components or services. They can be executed more frequently during the development and integration phases.4) Maintenance effort:
E2E tests: E2E tests are generally flaky and more prone to breakage as they cover larger portions of the application and are sensitive to changes in user interfaces, workflows, or external services. Maintaining E2E tests often requires updates when the application undergoes significant changes.
Integration tests: Integration tests are usually more stable and less affected by changes in user interfaces. They primarily focus on the interaction between components, making them more resilient to UI changes. However, changes to integration points or APIs may still require test updates.
While E2E tests can provide a comprehensive check of the system, they can also miss some bugs, especially ones involving edge cases. Integration tests, with their more focused scope, are better suited to covering these cases.
On the other hand, integration tests may miss bugs that only appear when the entire system is working together, which E2E tests can catch.
That being said, the generally accepted view in the software testing industry is that E2E testing tends to be more time-consuming and resource-intensive than integration testing. But it's also considered a more thorough method that checks the system as a whole.
How you can reduce your testing efforts with Keploy ??
Now we got to know that e2e gives more confidence but writing it is hard, And if we can perform API calls (which we usually do before pushing code though) and get some sort of tests bound along with their mock data. This can be run along with your native testing frameworks like Junit, Jest, and Gotest and logs the result which shows the difference between the actual and expected responses. This can be pretty useful stuff for all the teams which want to deliver fast and are focused on the end responses for the user.
So in a nutshell, Keploy simplifies the process of creating realistic E2E tests by capturing actual outcomes from your real environment, whether it's a testing or production environment. This means that you can leverage the power of real data mocks, replicating the scenarios that your application interacts with, be it infrastructure calls like HTTP or database operations.
One of the major hurdles in E2E testing was the effort required to write and maintain the tests. Keploy addresses this pain point head-on by automatically capturing the interactions and outcomes of your API calls. These tests are then stored in a readable YAML file format, making them easily manageable with popular version control systems like Git. With Keploy, you have a clear history of your tests, making collaboration and tracking changes a breeze.
https://cdn.hashnode.com/res/hashnode/image/upload/v1695924849087/93f6087b-27a6-47c7-83fb-4cbf640d0fba.gif?auto=format,compress&gif-q=60&format=webm
When running your application, instead of making actual outgoing calls, your application retrieves responses from the mock files generated by Keploy. This allows you to create a self-contained E2E test suite that accurately mimics how your application should behave after a successful release. By setting up these tests in your continuous integration (CI) pipelines, such as GitHub Actions or Jenkins, and executing them using Keploy's simple commands, you can ensure the reliability of your application with minimal effort.
Some glimpses of the tests and mock data generated by keploy can be shown in the form of YAML as -COPYversion: api.keploy.io/v1beta2 kind: Http name: test-1 spec: metadata: {} req: method: POST proto_major: 1 proto_minor: 1 url: http://localhost:8082/url header: Accept: '*/*' Content-Length: "33" Content-Type: application/json Host: localhost:8082 User-Agent: curl/7.81.0 body: |- { "url": "https://google.com" } body_type: "" resp: status_code: 200 header: Content-Length: "66" Content-Type: application/json; charset=UTF-8 Date: Wed, 27 Sep 2023 02:17:20 GMT body: | {"ts":1695781040077972081,"url":"http://localhost:8082/Lhr4BWAi"} body_type: "" status_message: "" proto_major: 0 proto_minor: 0 objects: [] assertions: noise: - header.Date created: 1695781040
And mock.yaml which comprises all buffer-decoded outgoing requests and responses looks like this -
So just perform a record session from real traffic and replay them exactly as they occurred. This makes it an invaluable tool for debugging hard-to-replicate issues that may be missed by traditional QA or even developers themselves. By leveraging real traffic data, you can gain deeper insights into your application's behaviour and identify potential issues more effectively. This empowers developers to release code efficiently as regression can be caught hand-to-hand. They just need to write some unit tests for some of their core methods and the rest can be covered through API flow via keploy. A collective coverage can give a big boost to the team for maintaining code coverage which can be considered a healthy dev practice.
In conclusion, both E2E and integration testing have their roles and neither can fully replace the other. They are both important parts of a comprehensive testing strategy and should be used in tandem to ensure a well-tested application. The time and effort invested in each type of testing can vary greatly depending on the specifics of the project and the team's expertise.
So when we have tools like keploy why we should not encourage them to perform a thorough e2e testing as we already "got everything right on as a test" ?? In my opinion, keploy is a boon to developers and testing teams as they can just simply run their application the same way they expected to and can see the code path covered for a particular test case.
In the next blog, we will discuss and showcase a sample application in which I created test cases and mocks without writing a single line of code with the help of Keploy V2. Check Keploy's GitHub repo here - https://github.com/keploy/keploy .
