There is no denial that Continuous Testing (CI) growth is so much faster and on-demand in software testing. A bunch of technologies, supporting tools and handicrafts are upscale to embrace faster and quick turnaround CI results to the team
This is not a typical article will go through all aspects of CI, but rather than a selection of what should be focusing so far in CI
What are CI focuses
CI aims to get fast feedback to the team for the smallest change pushed to the development repository as a commit. Once commit is made, a proper CI process will prevent bad things to deliver to respective persons by making checks on it. The checks must be:
All check attributes served to be the main focus of a CI process, of course, it depends on how the maturity established in your current project which can be less or more. These checks span across all testing activities in the project except Exploratory Testing.
Let’s talk about each check-in details
There will be no mention regards to other checks from the developers’ side such as automatic code reviews as it is part of the development pipeline.
Fast is the main core focus. Scheduling, selecting tests, and refine the CI tool for faster test preparation is the main point here.
The first thing in mind is your test script design, it must be atomic, independent of each other, and pinpoints to the absolute validations you need to check. Don’t spend time writing unnecessary steps or tests which are already done in the other tests. If your application under test has a shortcut to access the current state you want to test and this access point is already tested, then just use it without doubts.
When a test is executed, no matter what build you are using such as Gradle or Maven, etc…, they all need to download dependencies. Ensure you utilizing cache dependencies in your pipeline file/settings to avoid the same dependencies version to be re-downloaded
In a practical test pyramid, the very first thing in the gateway testing is unit test as it runs pretty fast. No need to select which unit tests to be executed as these are a very fast test which should not consume so much on the whole CI run
After unit tests are completed, the gateway will continue to execute integration tests to establish the linked component tests. This also happens in code logic so it will be fast also.
After both unit tests & integrations are completed, the next thing in line is the simple check on the build to validate it must pass smoke tests. Smoke test here is another gate for build validation against a commit or a change even small. You must not select smoke tests that spend too much time for execution and avoid actions on the interface as much as possible when they can be done from back-end services as they are cleared tested in the unit & integration-test phase.
When scheduling a test execution, always achieve parallelization in the test executions. Parallelization comes up as to server different tests can happen at the same time on different threads. Test execution time will be greatly reduced
One final thing to affect how fast of a CI process is a CI tool itself. Selecting a primary CI tool based on the fit to the whole application architecture. A CI tool is supposed to do many things at once besides the test stage. So make sure that the CI tool is maintained weekly and consider unnecessary stages elimination to avoid bottlenecks can happen. This mainly is a job of DevOps guy
CI Metrics: Average test executions time, Test environment/ data preparations, cost
As stated previously, all of the tests should not depend on each other, otherwise, there will be a queue time waiting for a required test to be completed before actual tests happen no matter what
Isolation in CI also comes from how you wrap up your project in a container image so that it is isolated and can be portable to different CIs.
CI metrics: Number of automated tests executed
As long as you can trigger many desired tests at the same time on the same pipeline’s stage, the gateway must report back what is tested, what is passed/failed/skipped, environment coverage, and so on. Reporting capabilities outside of test status scope are supported from the CI side, such as environment/device/browser being tested. For test details such as status, test description, etc… then CI tools will utilize the classic power of jUnit report to parse its content for general report overview. That’s not enough if you want more comprehensive reports
Roll up a centralized report dash to consolidate reports is another focus in the CI process. To achieve this, the framework must accommodate to the extension point exposed from the report dashboard to hook its execution results there. Quite many report tools have this kind of capability such as ReportPortal.io or Cube.js, etc…
Reporting comes as an important decision-maker to determine the health of the current build for immediate awareness, and decide is the release ready to roll up to production in the long term. Be sure to add this kind of consolidation reports after executions to track the build’s health and other testing metrics
CI metrics: N/A
The final step in the gateway is to notify the committer about the results through emails or external communication tools such as Slack. They will be notified of how their commit’s heath against the current application’s version and make immediate actions accordingly. Without notifications then who knows how our commits are
CI metrics: N/A
Above are basic checks of a gateway(s) in a CI pipeline should have. There can be more complex stages in the CI pipeline such as service virtualization or seed data for test data or clone different snapshots for a current test environment to justify more testing needs. As either more CI stages and steps are added, the CI pipeline will become more complex and more mature.
CI Mature Levels
Per my experiences a CI pipeline maturity is broken down into 3 levels:
- Level 1: Start-up
- A pipeline is initialized
- Add a stage to trigger tests
- Send emails to notify about tests results
- Level 2: Grow up
- Dependencies caching step is added
- Tests are executed in parallel on different stages against a specific environment
- Tests are executed on local browsers/devices on a host machine
- Report executions result with JUnit reporting XML files
- Send emails to notify about tests results
- Level 3: Tune-up
- Seed test data into tests
- Different test environments (based on product compatibility) are rolled up automatically
- Executed tests are against each deployed environments, specified cloud browsers/devices
- Local executed tests are distributed equally
- Test executions results are centralized and analyzed for short and long term future
- Notify results through Slack
Of course, most of the things say here you can treat them as theories for references and can refine them accordingly if you see they are not fit in your current project.
I hope this article provides helpful information for your upcoming or existing CI process.
I’ve created a not-fancy framework with some of the above implemented here. You can refer it or not if you want