Question: Test-Driven Design/Development has been talked about a lot, what about Automated Test-Driven Design/Development? Is that possible?
Response: I expect someone will market something someday that claims to automate TDD. But a human being at some point will still select variables or name the function or supply a requirement formatted in a tool-recognizable way, and I suspect that we will see limitations similar to CASE tools. You can also automate the production of employee performance reviews, but I wouldn’t recommend doing it if you want people to take interest in personal and professional development, the supposed purpose of the exercise.
Question: How applicable are these principles to AGILE environments vs. Waterfall?
Response: I have seen a very wide range of software production management schemes called “Agile”, most of them quite far from the Agile Manifesto, so it is sort of difficult to answer questions about “Agile environments”, the assumption being that there is a class of environments that can justifiably called Agile. In my view, the emphasis in Agile is on delivering working code through continuous integration; on max bandwidth communication up to and including collocation of dev and test; and on requesting inputs from the software’s intended consumers early and often. The study of test automation ROI is not inherent in any of this. The people who form Agile teams bring with them, however, a vast amount of learning and judgment formed through experience. There is nothing to recommend against including, in that learning and measured judgment, the knowledge derived from an analysis of ROI. In other words I think attention to test automation ROI – whether in pre-sprint planning or in parallel non-functional testing — can only enhance the productivity of Agile teams. When taking Agile principles to the Enterprise, test automation ROI would help optimize outcomes.
Question: We experience delays because the high demand people that do manual tests are not available for tests. Then when a retest needs to happen, it goes untested completely or is delayed?
Response: This doesn’t sound like something that can be resolved by test automation. It sounds like a systemic problem in the organization structure, perhaps in group-wide practices that have become persistent by habit. Test automation requires solid process as a foundation if it is to succeed. It won’t correct poor planning.
Question: I’m interested in the comment you made about the reviews of the test automation. How do you approach reviews?
Response: I think I mentioned peer and expert reviews of test automation in the context of cutting costs. Since reviews take time they cost money. But they eventually reduce costs because through expert reviews you have a way of propagating best practices that does not require that your engineers read a book or take a class, and the advice is up front and personal. Automation engineers who acquire bad habits are the last people to discover mistakes in coding that results from them, so reviews help. Some advice – I think I mentioned implementation patterns in the webinar – may be available only from experts, who could be the automation test lead or a senior engineer. Peer reviews have a leveling effect, raising the group’s median quality of automation.
Some of the stuff to include in test automation review: each test case verifies only one requirement; expected test result is unambiguous; all conditions required for test case execution are fulfilled by automated setup (may include disk imaging or installation of software); automation includes clean-up at end of suite (or test case if necessary); documentation up to standards; all variables declared; algorithms as simple as possible; new functions not included when available from common function library; metatags etc.
Question: During the test script planning phase, are there specific questions that we’ll need to ask whether a particular test case should be manual or automated?
Response: Yes. Among the questions you’ll want to ask are:
– Will automating this test case give me any of the benefits of test automation listed on Slides 17 and 18 of the Mindtree webinar?
– Can I afford to develop and maintain automation that would include this test case?
– Will I or other testers become so bored by repetitive execution of this manual test case that we will become incompetent judges of its outcome?
– Is this a test case capable of manual execution? Or does it need to be executed by automation to minimize the variations in human motor skills, eye/hand coordination, and sequential execution of code?