Automated testing is considered by companies as an easy way out to deliver high quality software and products, reducing testing costs and increase efficiency. But the truth is, not all bugs are found by automated testing, so it is very important to realize what can go undetected.
1. Over indulging on propriety testing tools
Many commercial testing tools provide simple features for automating the capture and replay of manual test cases. While this approach seems sound, it encourages testing through the user-interface and results in inherently brittle and difficult to maintain tests. Additionally, the cost and restrictions that licensed tools place on who can access the test cases is an overhead that tends to prevent collaboration and team work. Furthermore, storing test cases outside the version control system creates unnecessary complexity. As an alternative, open source test tools can usually solve most automated testing problems and the test cases can be easily included in the version control system.
2. Too lazy to setup CI server to execute tests
The cost and rapid feedback benefits of automated tests are best realised when the tests are regularly executed. If your automated tests are initiated manually rather than through the CI continuous integration system then there is significant risk that they are not being run regularly and therefore may in fact be failing. Make the effort to ensure automated tests are executed through the CI system.
3. Loving the UI so much that all tests are executed through the UI
Although automated UI tests provide a high level of confidence, they are expensive to build, slow to execute and fragile to maintain. Testing at the lowest possible level is a practice that encourages collaboration between developers and testers, increases the execution speed for tests and reduces the test implementation costs. Automated unit tests should be doing a majority of the test effort followed by integration, functional, system and acceptance tests. UI based tests should only be used when the UI is actually being tested or there is no practical alternative.
4. Jealously creating tests and not collaborating
Test driven development is an approach to development that is as much a design activity as it is a testing practice. The process of defining test cases (or executable specifications) is an excellent way ensuring that there is a shared understanding between all involved as to the actual requirement being developed and tested. The practice is often associated with unit testing but can be equally applied to other test types including acceptance testing.
5. Frustration with fragile tests that break intermittently
Unreliable tests are a major cause for teams ignoring or losing confidence in automated tests. Once confidence is lost the value initially invested in automated tests is dramatically reduced. Fixing failing tests and resolving issues associated with brittle tests should be a priority to eliminate false positives.
6. Thinking automated tests will replace all manual testing
Automated tests are not a replacement for manual exploratory testing. A mixture of testing types and levels is needed to achieve the desired quality mitigate the risk associated with defects. The automated testing triangle originally described by Mike Cohn explains the investment profile in tests should focus at the unit level and then reduce up through the application layers.
7. Too much automated testing not matched to defined system quality
Testing effort needs to match the desired system quality otherwise there is a risk that too-much, too-little or not the right things will be tested. It is important to define the required quality and then match the testing effort to suit. This approach can be done in collaboration with business and technical stakeholders to ensure there is a shared understanding of risks and potential technical debt implications.
No comments:
Post a Comment