Test design

1. Introduction

This document describes the automated test cases for the P4DTI, how they are designed, and how to run them. The purpose of the document is to allow P4DTI developers to collaborate on test case development and so improve the quality of the P4DTI.

The intended readership is P4DTI developers and people developing new integrations using the integration kit.

This document is not confidential.

2. Test framework

The automated tests use the Python unit test framework [PyUnit]. To use the framework with these tests, put the module unittest.py from the PyUnit distribution on Python's library path. New unit test scripts should be called test_module.py, where module.py is the module being tested. Each test script should define a function called tests, which returns an object belonging to the unittest.TestSuite class. The run_tests.py script finds all tests in the current directory and runs them.

To run the unit test for the message module, run the command python test_message.py.

To run the "wrapping" test case (only) from the unit test for the message module, run the command python test_message.py wrapping.

To run all the tests in this directory, run the command python run_tests.py.

3. Test database

This section specifies test databases. The test cases rely on specific features of the test databases (e.g., presence of certain fields) in order to test features of the P4DTI.

3.1. TeamTrack test database specification

1. In TeamTrack 4.5, select "tTrack workflow". In TeamTrack 5.0 create a "tTrack solution" once the database is created.

2. Choose a standard workflow with 5 states.

3. Make a project tree like this:

   Base project
     -- Group 1
        -- Elaborate editor
        -- Clever compiler
     -- Group 2
        -- Test 1
        -- Test 2
     -- Group 3
        -- Test 1
        -- Test 2
   

   (We need projects with identical names for the job000311 regression test; the names of the projects are used in test_p4dti.py, section 9.)

4. In TeamTrack 5.0, create all the groups.

5. Add licences from [Shaw 2001-06-27].

6. Create five users, all with no password:

   Id	Name	Groups/Privileges
   p4dti-replicator0	Replicator	Replicator privileges as specified by [RB 2000-08-10a, 5.3.2].
   gdr	Gareth Rees	Everyone, Developer, "Connect using the API"
   nb	Nick Barnes	Everyone, SQA, "Connect using the API"
   rb	Richard Brooksby	Everyone, Manager, "Connect using the API"
   user with space	User with space	Everyone

   (The user rb is named in various tests in test_p4dti.py.)

7. Make the "Additional Notes" field an append-only journal field.

8. Add five fields to the "Not used" section of the issue workflow:

   1. Logical name "code", database name "CODE", any supported type. This supports a regression test for job000140 (test_p4dti.py, section 6.3).

   2. Database name "MULTISELECT", multiple selection type. This allows us to test that a correct error message is produced when someone wants to replicate a field whose type is not supported (test_p4dti.py, section 6.3).

   3. Database name "BINARY", binary type.

   4. Database name "INTEGER", integer numeric type.

   5. Database name "FLOATING_POINT", floating-point numeric type.

   6. Database name "FIXED_POINT", fixed-point numeric type, with 2 digits of precision after the decimal point.

   7. Database name "SINGLE_LINE", fixed-length text field, with length 80 characters (this supports a regression test for job000442).

9. Specify default values for the Manager, Tester and Engineer fields (this allows the test cases to submit issues without having to fiddle about with users).

10. Edit the base project and set "Zero fill to" some value other than 5 (this allows us to run a regression test for job000426). Make sure that this applies to all other projects (by setting "Use parent project's sequence numbers").

11. Specify a default value for the Issue type (I suggest Bug).

12. In TeamTrack 4.5, edit the Issue workflow and specify an Owner for each state (I suggest Manager for states New, Verified and Deferred, Engineer for Assigned, and Tester for Resolved). This allows the test case to check that "behind-the-scenes" changes made by the defect tracker get replicated back to Perforce.

13. Restart the TeamTrack web server. Submit issues to the projects in Group 1. Make sure at least one issue is assigned to "user with space" (regression test for job000289).

14. Now your database is ready for testing. However, it's best not to test with this database but instead to use the TeamTrack Administrator to copy the database before testing with it. That way you always have a pristine database to go back to (and copy again).

3.2. Bugzilla test database

The Bugzilla test databases are stored as bugzilla-VERSION-mysqldump, where VERSION is the Bugzilla version (2.12, 2.14, etc). Each is produced by running Bugzilla, building up a bugs database, and using mysqldump to generate the file.

See [NB 2001-10-17] for the design, noting the following additions:

1. A user with spaces in their login name, to whom at least one bug is assigned (regression test for job000289).

4. Testing the TeamTrack integration

TeamTrack test procedure

Gareth Rees, Ravenbrook Limited, 2001-02-26


1. Introduction

This document explains how to carry out a test procedure for the
Perforce/TeamTrack integration on Windows NT or Windows 2000.

The intended audience is P4DTI project staff.


2. Setting up a testing session

Use this procedure to start a fresh testing session with a new TeamTrack
server, Perforce server and P4DTI (but don't run the P4DTI yet).

  1. Stop any running Perforce servers and TeamTrack servers.

  2. Make a new dated directory like d:\p4dti-test\YYYY-MM-DD\p4d.

  3. Start a Perforce server in that directory (p4d -p 0.0.0.0:1666 -r
     d:\p4dti-test\YYYY-MM-DD\p4d).  Add a license file.

  4. Start P4Web in browse mode (p4web -p 127.0.0.1:1666 -b).

  5. Copy teamtrack-test.mdb into d:\p4dti-test\YYYY-MM-DD, using the
     TeamTrack Administrator.  Call the new ODBC database called
     something like "P4DTI YYYY-MM-DD"; connect to that database; start
     the TeamTrack web server.

  6. Configure and run the perforce_sample_data.py script.

  7. Unpack the P4DTI release into d:\p4dti-test\YYYY-MM-DD\p4dti.

  8. Copy the appropriate configuration file from this directory to
     d:\p4dti-test\YYYY-MM-DD\p4dti\config.py.


3. Initial installation tests

  1. Normal installation.  Follow the setup procedure; start the
     replicator.  It should replicate all issues successfully.

  2. Recent start date.  Follow the setup procedure; set the start_date
     parameter to the current time; start the replicator.  The
     replicator should replicate no issues.

     Set the start_date parameter to an old date; restart the
     replicator.  The replicator should replicate the old issues.

  job000112. Can't easily replicate by project

     Set replicate_p to "return self['PROJECTID'] in [3,4]".  Check
     that only the programming projects are replicated.


4. Use cases and regression tests (users)

  1. Follow the setup procedure; start the replicator.  Check the
     initial e-mail message to the administrator.

  2. Run perforce_sample_data.py to generate some activity in Perforce.

  3. Follow use cases in the User Guide.

  job000006. TeamShare API error reporting is inadequate

     Make an edit you don't have permission to.  Make a transition you
     don't have permission to.

  job000024. "\012" in fixes table in TeamTrack

  job000053. Implicit changes made by the DT don't get replicated back

  job000063. Dates aren't replicated with fixes

  job000087. Replicator doesn't enforce licences in TeamTrack

     Make a edit in Perforce as a user who has no corresponding user
     in TeamTrack.

  job000107. You can't replicate an issue's project

  job000109. Can't get at Perforce information from TeamTrack

  job000133. You can't close a job by fixing it

  job000146. Empty date fields in TeamTrack give the replicator a headache

     Add "ESTTIMETOFIX" and "ACTTIMETOFIX" to the replicated_fields.

  job000156. Pending changelists aren't clearly indicated as such

  job000169. Change numbers are links in TeamTrack even when no
  changelist URL has been specified

  job000182. Elapsed time fields aren't replicated properly

  job000188. tTrack integration fails with non-ASCII characters

  job000223. Quote in change comment terminates display in TeamTrack

  job000226. Newlines don't show up in change descriptions in TeamTrack

  job000233. When you submit a new issue to TeamTrack it overwrites the issue


5. Use cases and regression tests (administrator)

  job000108. Userids can't differ between TeamTrack and Perforce

  job000140. Logical field name "code" not allowed in TeamTrack

  job000141. Can't add to replicated_fields list

A. References

B. Document History

Copyright © 2001 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.