RELEASE TEST PROCEDURE Richard Brooksby, Ravenbrook Limited, 2001-03-21 1. INTRODUCTION This is the basic minimum test procedure to be applied to all releases of the P4DTI software. The readership of this document is project engineers. This document is not confidential. 2. PROCEDURE 2.1. Clean your machine. Remove previous installations of P4DTI, Bugzilla, TeamTrack, and other software. For a really thorough test, remove all prerequisite software, or start with a completely virgin machine with a freshly installed operating system. 2.2. Obtain the release. - Check that links to the release are correct. - Check that the release is correctly described. - Read and follow the instructions that accompany the release (the readme.txt) file. 2.3. Verify that all the prerequisites are met. - Ask Perforce for its version and check it against the manual. - Ask Python what version it is and check it against the manual. - Inspect the TeamTrack build number. - Make sure you're running an _unpatched_ Bugzilla. 2.4. Install test repository, databases, etc. from the test materials for the relevant version (e.g. teamtrack-test.mdb). These should be constructed to meet the various "procedural prerequisites". 2.5. Copy the configuration for the test machine from the test materials for the relevant version. Check it over by eye. 2.6. Follow the AG instructions for patching Bugzilla, updating the registry, etc. 2.7. Start the replicator as described in the manual and compare the output to that in the manual. Stop it as described. Start it again. On Linux, test the startup script for starting and stopping the replicator. On Windows, install the NT service for managing the replicator and test it. In both cases log out and make sure the replicator keeps running, then reboot the machine to make sure that the system starts automatically as described. 2.8. Follow the instructions in the manual for testing the configuration. (i.e. take an issue through a complete life cycle as described in the UG, from various Perforce interfaces, and run "check.py"). Deliberately simulate various user errors and violations of permissions to check e-mail delivery and accuracy. Look up error messages in the AG. 2.9. Run any automatic tests to stress the system. 2.10. Change the set of users and a user's e-mail address and re-start the replicator to make sure it picks up the new configuration, as described in "Maintaining the P4DTI". 2.11. Change the set of replicated fields and refresh the Perforce jobs as described in "Maintaining the P4DTI". 2.12. Leave it running for as long as possible. 2.13. Uninstall the integration, following the instructions in the AG. Make sure that Perforce and the DT are still functioning. 2.14. Test the TeamTrack integration in the three configurations we support: TeamTrack 4.5, TeamTrack 5.0, and TeamTrack 5.0 with a database upgraded from TeamTrack 4.5. 3. TESTING WITH MULTIPLE PERFORCE SERVERS 3.1. Follow the instructions in [GDR 2001-11-14, 5] to set up a system with several Perforce servers, replicators and P4Web instances. 3.2. Start all the P4DTI replicators. 3.3. For each project, go through an issue lifecycle: create it in TeamTrack, assign it to someone, check that it is replicated to Perforce, close it in Perforce by fixing it with a changelist, check that the appropriate transition happens in TeamTrack. 3.4. Check that issues get replicated only to one Perforce server, and to the right Perforce server. 3.5. Check that the changelist link in TeamTrack goes to the right instance of P4Web. Check that when two changelists have the same number in two different Perforce servers, theie descriptions show up properly in TeamTrack (that is, TeamTrack doesn't confuse changelist 17 on Perforce server A with changelist 17 on Perforce server B). 3.6. Try simultaneous changes in two or more Perforce servers. 3.7. Try stopping one of the P4DTI replicators. Make some changes while the replicator is stopped. Restart the replicator. Does it pick up the changes that happened while it was stopped? 3.8. Try other user activity that you think might break the system. A. REFERENCES [GDR 2001-06-29] "Testing the P4DTI with multiple Perforce servers" (e-mail message); Gareth Rees; Ravenbrook Limited; 2001-06-29; . [GDR 2001-11-14] "Perforce Defect Tracking Integration Advanced Administrator's Guide"; Gareth Rees; Ravenbrook Limited; 2001-11-14; . B. DOCUMENT HISTORY 2001-03-21 RB Drafted in e-mail. 2001-03-26 RB Edited into shape as text document in master sources. 2001-07-05 GDR Test TeamTrack in all three configurations. 2002-01-31 GDR Added section on testing with multiple Perforce servers, based on [GDR 2001-06-29]. C. COPYRIGHT AND LICENCE This document is copyright (C) 2001 Perforce Software, Inc. All rights reserved. Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer. THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. $Id: //info.ravenbrook.com/project/p4dti/version/1.4/procedure/release-test/index.txt#2 $