Automated Testing

The principle of automated testing is that there is a program (which could be a job stream) that runs the program being tested, feeding it the proper input, and checking the output against the output that was expected. Once the test suite is written, no human intervention is needed, either to run the program or to look to see if it worked; the test suite does all that, and somehow indicates (say, by a :TELL message and a results file) whether the program's output was as expected. We, for instance, have over two hundred test suites, all of which can be run overnight by executing one job stream submission command; after they run, another command can show which test suites succeeded and which failed.

These test suites can help in many ways:

1] As discussed above, the test suites should always be run before a
new version is released, no matter how trivial the modifications
to the program.

2] If the software is internally different for different
environments (e.g. MPE/V vs. MPE/XL), but should have the same
external behavior, the test suites should be run on both
environments.

3] As you're making serious changes to the software, you might want
to run the test suites even before the release, since they can
tell you what still needs to be fixed.

4] If you have the discipline to -- believe it or not -- write the
test suite before you've written your program, you can even use
the test suite to do the initial testing of your code. After all,
you'd have to initially test the code anyway; you might as well
use your test suites to do that initial testing as well as all
subsequent tests.

Note also that the test suites not only run the program, but set up
the proper environment for the program; this might mean filling up a
test database, building necessary files, etc.