Mime-Version: 1.0
Message-Id: <p04330103b6de53e26a2e@[193.112.141.252]>
Date: Wed, 21 Mar 2001 13:17:41 +0000
To: Perforce Defect Tracking Integration Project staff <p4dti-staff@ravenbrook.com>
From: Richard Brooksby <rb@ravenbrook.com>
Subject: Draft release test procedure
Content-Type: text/plain; charset="us-ascii" ; format="flowed"

Here is a rough draft of a test procedure for releases.  This is 
about the minimum I think we should do for a customer visible 
release, and it should be applied to 1.0.5, which we will be making 
shortly.

This should be turned into a maintained document (or set of 
documents) in master/test at some point.

We also need to create procedures for modifying certain things to 
make sure that test procedures are maintained in parallel with things 
like the documentation and the code.

We also need to spend time working out preventions for all jobs from 
now on and incorporating them into tests.  Perhaps we shouldn't 
"close" a job until we have a change to a test to prevent the defect.

Comments welcome.  This is very rough, but it will cover a lot more 
that we have been doing.

---

1. Clean your machine.  Remove previous installations of P4DTI, 
Bugzilla, TeamTrack, and other software.

For a really thorough test, remove all prerequisite software, or 
start with a completely virgin machine with a freshly installed 
operating system.

2. Obtain the release.

   - Check that links to the release are correct.

   - Check that the release is correctly described.

   - Read and follow the instructions that accompany the release (the 
readme.txt) file.

3. Verify that all the prerequisites are met.

   - Ask Perforce for its version and check it against the manual.

   - Ask Python what version it is and check it against the manual.

   - Inspect the TeamTrack build number.

   - Make sure you're running an _unpatched_ Bugzilla.

4. Install test repository, databases, etc. from the test materials 
for the relevant version (e.g. teamtrack-test.mdb).  These should be 
constructed to meet the various "procedural prerequisites".

5. Copy the configuration for the test machine from the test 
materials for the relevant version.  Check it over by eye.

6. Follow the AG instructions for patching Bugzilla, updating the 
registry, etc.

7. Start the replicator as described in the manual and compare the 
output to that in the manual.  Stop it as described.  Start it again. 
On Linux, test the startup script for starting and stopping the 
replicator.  Reboot the machine to make sure it starts automatically 
as described.

8. Follow the instructions in the manual for testing the 
configuration.  (i.e. take an issue through a complete life cycle as 
described in the UG, from various Perforce interfaces, and run 
"check.py").  Deliberately simulate various user errors and 
violations of permissions to check e-mail delivery and accuracy. 
Look up error messages in the AG.

9. Run any automatic tests to stress the system.

10. Change the set of users and a user's e-mail address and re-start 
the replicator to make sure it picks up the new configuration, as 
described in "Maintaining the P4DTI".

11. Change the set of replicated fields and refresh the Perforce jobs 
as described in "Maintaining the P4DTI".

12. Leave it running for as long as possible.

13. Uninstall the integration, following the instructions in the AG. 
Make sure that Perforce and the DT are still functioning.