JUnit,Integration,End to End Tests

During my experience I meet many definition of Test Types. I know them by many names : end to end tests, functional test, system test, UI test, Selenium Tests, Soap-UI test, Integration tests,Acceptance tests. 

Although there are great articles out there I believe still some questions remain : What really is an integration test? Can it be a JUnit test? Are Integration Test duplicates of JUnit? What should any of the test types cover and what not? What kind of tests does an system need?

In this post we are not going to invent other definitions test types but rather hopefully come to a common understanding which also would help to use them correctly.

Unit Tests

In previous post we explored this type of tests by an example and come to conclusion that is essential to avoid Testing Implementation Details and not follow the rule “Test each method and each Class“. Here comes first tentative to describe Unit Tests:

A Unit tests is a simple test that runs on Isolation from other Unit Tests(not from his internals) aiming to cover only a small part of our system.

We cannot have a database in a JUnit test since is shared resource and tests can interfere to each other business.

What if we create a new database in the beginning of each test and than drop it, so each tests has its own database? We certainly follow the “Run in Isolation Rule”! This good question reveals the fact that our definition is not complete. We mention the word “Simple” there but simplicity is very relative and it is not enough. Additionally a Unit Test needs to be also very fast. The reason is well know: on this phase of development we need very fast feedback loop. A fast feedback loop helps us to develop faster and understand better the code. In a matter of seconds we know if our code is in the good path or not and change in the very earlier phase of development where the code added is not big and complete. So lets try again:

A Unit tests is a very fast response test that runs in Isolation from other Unit Tests(not from his internals) aiming to cover only a small part of our system.

There is a still an ambiguous variable in the definition and that is “very fast”.Very Fast could be relative depending what is fast means on different system so I wold say that means “As Fast as Possible“. In a few feel free to mock every shared resource like database and every thing that slows down the test like : Web Services, Heavily Network Access, third party Systems… But is important as we say  in this post to not mock Internals or Implementation Details as this leads to awful problems and takes agility from your hand without even noticing it.

Integration Tests

Is worth to mention that this type of tests are the most confused one.Usually I heard people call them End to End Test or even Functional Test.As we will see this kind of test are not End to End as they test the system on entirely different conditions.

As we saw above JUnit Tests tend to test the system in state fare away from Production. Many things are mocked there and small parts are tested in isolation. So we do not have the confidence that the this small well tested parts of system are working correctly together. Below problems may emerge:

• Another part of system is not working as per our understanding. So also problems inside the application parts may occur or maybe the way the parts interact with each other.
• SQL may not return the data we assumed or SQL Syntax is incorrect.
• Web Service is not called correctly or not returning structure we hoped.
• Performance problems and multi threading issues may come up.

Integration Tests are testing parts of the system in interaction with each other as consequence testing the system in a state more similar to Production.

Here it starts the confusion…

• How similar to Production should the system be under Integration Tests?
• What parts should I mock? Should I include thirst party web services?
• Are Integration Test duplicate of JUnits? If yes can I remove JUnits?
• How much to cover?
• Are there any time constrains here? How fast should they run?

Lets try to give answer to this questions.

How similar to Production?

Integration Tests still are not testing a  production system state.

1. Anyway the first thing which is a must is avoiding the production infrastructure overhead and complexity. So if we have an application run on several servers or maybe a distributed micro-service application running in cloud we would need to put everything in one server.
2. Network access must be also avoided so everything runs on localhost.
3. Shared Resources maybe be shared between own system parts but not from other systems.So system under test has the exclusivity of accessing the DB or particular Queue or Web Service and no other system.
4. Usually every Integration Test runs in isolation from other Integration Tests. All Environment is on localhost and a fresh copy of this local host environment is created each time a test runs and drooped when finish. So every test at the beginning creates Fresh DB, Fresh Queue , reset Web-Services and their resources.
5. Above point says “Usually” because for performance reasons we may group Integration Test together and make this tests share the same environment. I would suggest to avoid as much as possible since this solution is hard to maintain and may increase the number of tests failing because of bad data instead of a bug.
6. Although not exactly as Production we include here configuration properties files as much as possible similar to PROD.

What parts should I mock?

1. From point 2,3 and 4 above we immediately exclude and mock third party systems that we do not own by our self.
2. Feel free to mock if particular system we are using and own is difficult to deploy locally or hard to reset its very state-full state. Usually we should not test other systems but instead only test how we are using them.
3. Is important to include every framework,tool or platform we are going to use in Production. Also third parties technology like DB, Queue systems.

Are IT duplicate of JUnits?

Depending on the application type, YES sometime they are.

• If the application under test has a very thin layer of logic and a very thick DB layer(most of the systems have) than yes Integration Test duplicate JUnits. By DB layer is meant DAOs or Repositories so the layer which executes queries and returns Beans or data in particular structure. The Logic layer in this case it just gives the returned data to the caller and maybe sometimes just converts the data to other Beans or slightly different.structure.Some java REST application out there may be doing exactly this: through HTTP URL are getting data from DB and showing those data in some format(JSON format usually). In this case I would avoid having many JUnit tests and add them only in the case Logic or Service layer is doing some more complex logic there. In this case JUnit are not useful and offer little value because most of the application main functionality is depending on DB.Is worth to mention that if you try to do JUnit tests in this kind of application you will find yourself probably doing more mocking things than testing them(testing internals). Here Integration Tests play a crucial role as we will have many of them testing different queries scenarios so one should be carefully with the performance and how the DB is shared.
• Regardless of the DB layer if the application under test has thick layer of logic than Integration Tests are not duplicates. The reason is that JUnit are covering that thick layer of complex logic and Integration Test are covering the part they are missing , for example DB queries. We can have a Service queering all possible flights from DB between two points and than run a very complex logic to choose and suggest only the best options or relevant ones to the particular user. So we will have many Junit Test covering that complex logic with different supposed data or users or states or points…. and just one or depending on where condition very few Integration Test which makes sure the query returns the data in the intended form and content. Same thinking can be easily extended to Queues or other systems services.

How much to cover?

An integration tests should always created to test what JUnit is missing under the conditions we explained at the first point. So basically is not meant to test third parties or over test things that JUnit already covers like NullPointerExceptions checks or empty fields, formats… This hardly depends on the system but if we think in terms of what new missing paths from JUnit our test is covering than we are fine. Is worth to mention that Integration Tests take more time to execute so we should be more carefully in added them and really think about the value they add or quality they increase.

How fast should they run?

As usually as fast as possible with a system that looks more close to Production. So in theory we should expect our Integration Tests to execute more slowly. We should look carefully how long all Integration Tests take to execute and this should not be too long as this decreases our ability to validate changes therefore slow down response to change. It greatly increase the time to go to production and providing value for the customers. Here is an example how it can go:

For example if all Integration Tests take 2 hours to execute than a developer needs 2 hours and to find out if the code is working or not which is 1/4 of working hours. During this time a developer can be blocked and wait with doing almost nothing. Or even worst it can switch to another different task and loose the focus completely and make it hard to return later and maybe introduce another bug. It is also possible that during the 2 hour time and the time needed to fix the bug found by Integration Tests some other developers pushed code and when the new code with bug fix is pushed it is still not working because of changes. So here it starts again you wait 2 hours….

By now it should be clear that Integration Tests should run as fast as possible by following the first point suggestions and avoiding adding duplicates to JUnit tests already coverage.I would say 20-30 minutes is acceptable time.

When is not possible to achieve acceptable times than I would say the system is to big and monolithic and is urgently needed to separate in smaller systems or modules which can easily tested and maintained. Is better to slow down and split the system than continue that way until one day it will not be even possible to go to PROD(or it may take months).

End to End Tests

In principle this kind of tests cover what is left from JUnit and Integration Tests.

Ideally End to End Tests test a system exactly as Production one.

We say “Ideally” because is not always possible to have a system that looks exactly as Production.Than the questions arrive:

• What is the minimum requirement that  qualifies a test to a End to End test?
• What should they cover?
• How Fast should they run?

What qualifies a e2e test?

• Firs I would say everything that Integration Test has included is definitely included in here like DB, Queue,Other systems…
• What is a must for e2e test is having a system on the same infrastructure as production one. In the case we have really heavy production infrastructure like tens of hundreds of servers we can of course put the system on a smaller scale. For example if in production we have 1+9 load balance servers with each of them routing to 10 servers we can of course limit to 1 load balance and two servers in test as anyway even if the scale is smaller the infrastructure is similar to PROD in principle.
• Preferable also third parties should be included here. Some times is not possible because this system may be expensive to use for test or maybe is not even possible for security reasons.In this cases is fine to mock third parties service.
• End to End tests share everything together so here is not like Integration Tests with each of them having a clean state.Usually in e2e tests we have DB with big data that ideally will look like PROD. Of course sometimes is not possible for privacy reason but still we can use hashed information or anonymous one. Even if that is not possible than probably we should put a lot of attention to create meaningful data for this tests.

What should they cover?

Usually anything that JUnit and Integration Test are missing should cover here like production like infrastructure,third parties, all other slow systems we mocked previously, frameworks,platforms,technology,tools… We should be carefully to not duplicate tests in here as e2e test are very slow and expensive to maintain. For example if our application has only DB and not calling any web service or we did not mock other systems on Integration Test than probably adding e2e test makes no sense as Integration Test covered already the functionality. Probably is highly suggested to execute some few Smoke Tests to see if the all infrastructure is working as intended like load balance is working, DB is accessible or information is coming to Queues and so on.

How fast should they run?

Same also here as fast as possible with a system preferably exactly as production. This kind of test are really hard to maintain and slow. Most of the time this kind of tests tend to fail for non functional reasons like network problems, other systems issues, credential problems and so on. In a few words generally problems we do not have in production. This problems take long time to investigate as we go through logs, travel through different systems and in the end we find out that connectivity is not working. The test itself here can take more time to execute but as always we should notice carefully how long they need to run total and how often they fail for non functional reasons. With end to end tests is hard to give any total time as the system is like production and it really depends on the system nature but I would really try to have very few of this tests. Is better to have longer running integration tests than end to end tests, this tests really can harm the productivity.

Conclusion

• Have many JUnit tests as they are faster and give fast feedback, less integration test and even less or even no end to end tests to cover prod infrastructure and third parities. So in a few rods we should avoid Ice Cream Anti Pattern but rather have Pyramid of Testing.
• In principle all tests types are classified based on how much similar is the system under test with the production system.
• Feel free to skip JUnit tests when your logic is thin and DB layer is thick. Is good to cover with JUnit maybe some of thicker complex part that do more than fetching data from DB.
• Avoid testing duplicates in Integration Tests or End to End test. If certain logic is covered by JUnit like non empty fields ,wrong formats than we need only for example to add test that check the SQL query fetching correct data. Same for End to End test if database layer is checked than we need only to check maybe some third parties and the load balance and infrastructure is working.
• Is fine to not have end to end tests and in this case we do not need to name Integration Tests to End to End tests as they are not testing a system production like.