Perforce Defect Tracking Integration Advanced Administrator's Guide

Contents

• 1. Introduction
  • 1.1. Required experience
• 2. Choosing which issues to replicate
  • 2.1. TeamTrack examples
  • 2.2. Bugzilla examples
• 3. Allowing users to create issues in Perforce
  • 3.1. The replicate_job_p function
  • 3.2. The prepare_issue function
• 4. Migrating to the defect tracker from Perforce jobs
  • 4.1. Planning for migration
  • 4.2. How to migrate
  • 4.3. Providing names and e-mail addresses for Perforce users
  • 4.4. Making defect tracker user accounts
  • 4.5. Making Perforce jobs consistent with the jobspec
  • 4.6. Choosing which jobs to migrate
  • 4.7. Translating jobs to the new jobspec
  • 4.8. Transforming jobs to issues
  • 4.9. Summary of translation, transformation, and preparation steps
  • 4.10. Running migration
• 5. Integrating with multiple Perforce servers
• 6. Querying the TeamTrack database
• A. References
• B. Document History

1. Introduction

This manual is the Perforce Defect Tracking Integration Advanced Administrator's Guide. It describes various advanced methods of administering the P4DTI that are outside the scope of the Perforce Defect Tracking Integration Administrator's Guide because they require a higher level of expertise (see section 1.1, "Required experience").

This document is intended for P4DTI administrators. Ordinary users of the defect tracker or Perforce should read the Perforce Defect Tracking Integration User's Guide.

This guide does not describe how to install, maintain, or perform ordinary configuration and administration of the P4DTI. Read the Perforce Defect Tracking Integration Administrator's Guide.

1.1. Required experience

To do the things described in this guide, you should have the following experience:

1. Perforce: You must be expert with jobspecs and jobs, confident about backing up and restoring from checkpoints, able to set up a duplicate or test Perforce server. You need good working knowledge of the relevant sections of the Perforce System Administrator's Guide.
2. The Python programming language: You must be reasonably competent with Python; in particular, able to write functions that transform dictionaries.
3. The defect tracker: You must know the defect tracker's database schema. You must have detailed knowledge of your workflow and how the workflow puts constraints on the issue data. You must able to make backups, and able to set up a duplicate or test defect tracker.

2. Choosing which issues to replicate

The P4DTI has a general way to select which issues get replicated from the defect tracker: the replicate_p function. To select which issues get replicated you need to edit this function in the file config.py in the installation directory.

Normally, the P4DTI replicates all issues created or modified after the start_date, but you can modify this function to further restrict the issues. See section 5.1, "P4DTI configuration", of the Perforce Defect Tracking Integration Administrator's Guide for more details of the start_date parameter.

Note: once an issue starts being replicated it remains replicated, even if it no longer matches the criteria.

The function is passed one argument, the issue, in the form of a Python dictionary mapping field name (string) to field value. The function must return 1 if the issue should be replicated, or 0 if it should not.

The replicate_p function does not have to check the start_date configuration parameter. The start_date is always honored by the P4DTI, in addition to any test made by replicate_p.

3. Allowing users to create issues in Perforce

Normally, the P4DTI requires users to create new issues using the defect tracker. It ignores new jobs created in Perforce. The P4DTI can be configured to transform new jobs created in Perforce into new issues in the defect tracker.

You need to add two functions to the end of config.py: replicate_job_p and prepare_issue.

3.1. The replicate_job_p function

The replicate_job_p function selects which newly-created jobs in Perforce should be translated and added as new issues in the defect tracker. It takes the newly-created job as its argument, in the form of a Python dictionary mapping field name (string) to field value (string). It must return 1 if the job should be translated and added to the defect tracker; 0 if not.

Note: once a job starts being replicated it remains replicated, even if it no longer matches the criteria.

If you want all newly-created jobs to be translated and added to the defect tracker, you should write a replicate_job_p function as follows:

def replicate_job_p(job):
    return 1

3.2. The prepare_issue function

The P4DTI calls prepare_issue to complete the transformation of a new job in Perforce to an issue in the defect tracker. The P4DTI does its best to build the new issue, but the defect tracker may require fields that can't automatically be deduced from fields in the job. This function must supply values for these fields.

The function takes two arguments (issue, job) as follows:

1. issue is the partly-constructed issue, in the form of a Python dictionary mapping field name (string) to field value.
2. job is the newly-created job, in the form of a Python dictionary mapping field name (string) to field value (string).

The function must modify issue so that it is a valid issue for creation in the defect tracker. Its return value is ignored.

4. Migrating to the defect tracker from Perforce jobs

You must delete all jobs from your Perforce installation before running the P4DTI for the first time. The P4DTI takes over the jobs subsystem of Perforce and rewrites the Perforce jobspec.

Migration allows you to keep existing issues that are stored in Perforce jobs. Migration submits all the Perforce job data to the defect tracker, so that the issues can be replicated back to Perforce, and so appear in both systems.

4.1. Planning for migration

You will need a plan for installing and testing the P4DTI and migrating your Perforce jobs. In particular, you will need to have configured the P4DTI replicator before you migrate, and to have a good understanding of what it does. This makes the migration process quite complicated.

This section describes a plan which will allow your users to continue to work with Perforce and your defect tracker until you're ready to go live.

4.1.1. Configuring and testing the P4DTI and migration

The first stage is to configure and test both the P4DTI replicator and migration.

1. Create an empty Perforce server for testing.

2. Create a defect tracker for testing. If you have existing defect tracking data, this should be a copy that includes your workflow and at least a representative sample of your existing issues.

3. Set up and test the P4DTI replicator (without migration) on the test Perforce server and test defect tracker, following the instructions in the Perforce Defect Tracking Integration Administrator's Guide. Ask your users to try it out, and make sure you are happy with the workflow and the P4DTI configuration.

4. Make a note of the jobspec created by the P4DTI replicator. This will be used in configuring the translation step of the migrator. (See section 4.7, "Translating to the new jobspec".)

5. Replace the test Perforce server with one which has a copy of all your jobs and fixes. (You can do this by creating a checkpoint of your main Perforce server and restoring it onto a new empty Perforce server. See chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.)

6. Set up and test migration from the test Perforce server to the test defect tracker, as described in section 4.2, "How to migrate".

7. Apply any workflow changes you made on your test defect tracker to your main defect tracker server.

When you are confident that you have the correct configuration, proceed to section 4.1.2, "Going live with the P4DTI with migration".

4.1.2. Going live with the P4DTI with migration

Once you have configured and tested the P4DTI with migration, follow these steps to make the system available to your users:

1. Ask your users to stop using the defect tracker and Perforce server.

2. Make a backup up of your Perforce repository and your defect tracker database, in case there are any serious problems.

3. Edit your config.py file to point to your main Perforce and defect tracker servers, but don't start the P4DTI replicator yet.

4. Run the migration and check the results (4.10).

5. Start the P4DTI replicator.

You should now have a functioning integration with your old issue data intact.

4.2. How to migrate

Migration cannot be done automatically by the P4DTI, because:

1. The data in Perforce is in a very different format from the defect tracker.
2. The way the your workflow is implemented in Perforce is likely to be different from the way it's implemented in the defect tracker.

Your existing issues must therefore be translated in order to migrate them from Perforce jobs to the defect tracker.

Here is an overview of preparing for migration. The details of each step are covered in the following sections.

1. Prepare the Perforce data for migration:

   1. Ensure that users in Perforce match users in the defect tracker (4.3 and 4.4).

   2. Make sure that all Perforce jobs are consistent with the current jobspec so that the P4DTI can transform them systematically (4.5).

2. Write Python functions to your config.py to do the following:

   1. Choose which jobs to migrate (4.6).

   2. Translate these jobs so that they match jobs that would be replicated from the defect tracker (4.7).

   3. Fix up the issues transformed from jobs so they satisfy any defect tracker constraints on submitted issues (4.8).

3. Run the migration to transform all the jobs and submit them to the defect tracker (4.10).

To migrate your jobs, work through these subsections.

4.3. Providing names and e-mail addresses for Perforce users

Make sure that every Perforce user has a name and e-mail address.

4.4. Making defect tracker user accounts

Make a defect tracker user account for each Perforce user who needs to do defect tracking.

4.5. Making Perforce jobs consistent with the jobspec

When you change the Perforce jobspec Perforce doesn't update its jobs to match, so if you've changed jobspecs in the past your jobs may be inconsistent with the current jobspec. For example, it may be that you have a "required" field which is missing from some jobs, or a "select" field for which some jobs have an illegal value. These problems will prevent the P4DTI from being able to read the jobs.

To find inconsistencies, go to the P4DTI installation directory and run the command python check_jobs.py (it takes about a minute for every thousand jobs on a typical system). For example:

$ python check_jobs.py
(P4DTI-10081)  Job 'job000024' doesn't match the jobspec:
(P4DTI-7087)  Error detected at line 11.
Value for field 'Status' must be one of open/suspended/closed.

Fix these jobs so that they match the jobspec. When this is done, you'll get the following output:

$ python check_jobs.py
(P4DTI-10092)  All jobs match the jobspec.

4.6. Choosing which jobs to migrate

Choose which jobs to migrate by adding the function migrate_p to the end of config.py.

The migrate_p(job) function determines whether to migrate a job to the defect tracker. The job argument is the job being considered for migration (in the old jobspec), in the form of a dictionary mapping field name to field value. The function must return 1 if the job should be migrated, 0 otherwise.

4.6.1. Examples of migrate_p

1. To migrate all jobs:

   
   def migrate_p(job):
       return 1
   

2. To migrate open jobs only:

   
   def migrate_p(job):
       return job["Status"] == "open"
   

3. To migrate jobs in the "horseshoe" and "nail" projects only:

   
   def migrate_p(job):
       return job["Project"] in ["horseshoe", "nail"]
   

4.7. Translating jobs to the new jobspec

Write a function to translate jobs from the old jobspec to the new jobspec. Add the function translate_jobspec to the end of config.py.

The translate_jobspec function translates a job from the old jobspec (before migration) to the new jobspec (after migration). The old job is passed as the single argument, in the form of a dictionary mapping field name to field value. The new job must be returned in the same form (it's OK to update the argument and return that).

The P4DTI replicator takes over the jobs subsystem of Perforce and rewrites the Perforce jobspec according to the replicated_fields configuration parameter. You should have a copy of the new jobspec, made after testing the P4DTI replicator as described in section 4.1.1, "Configuring and testing the P4DTI and migration".

Job: job000123
Status: verified
User: gdr
Date: 2001/10/11
Title:
        The widgets are specified in imperial units (ft/lb/s).
Description:
        But 80% of customers use ISO 31, so we should use m/kg/s.
Priority: high
Project: widget

Job: job000123
Status: closed
Assigned_To: gdr
Date: 2001/10/11
Summary:
        The widgets are specified in imperial units (ft/lb/s).
Description:
        But 80% of customers use ISO 31, so we should use m/kg/s.
Resolution: FIXED
Priority: P1
Product: widget

4.8. Transforming jobs to issues

Jobs translated to the new jobspec may still not yet be suitable for submission to the defect tracker. The defect tracker may have additional constraints, or require fields that can't be translated from fields in the job. You must therefore write a prepare_issue function as documented in section 3.2, "The prepare_issue function".

4.9. Summary of translation, transformation, and preparation steps

Figure 1 shows a summary of the translate, transform, and prepare steps of the migration process:

1. The job is translated from the old jobspec to the new jobspec by the translate_jobspec function.

2. The P4DTI transforms the job to an issue by converting each field in replicated_fields and any system fields.

3. The issue is prepared for submission to the defect tracker by the prepare_issue function.

Figure 1. The translate, transform, and prepare steps. The example data in the figure is for TeamTrack.

4.10. Running migration

Make sure that you have migrated users as described in section 4.4.

Make sure you have the functions migrate_p, translate_jobspec, and prepare_issue in your config.py, as described in the sections above.

Make a checkpoint of the Perforce server as described in chapter 2, "Supporting Perforce: Backup and Recovery", of the Perforce 2001.1 System Administrator's Guide.

Go to the P4DTI installation directory and run the command python migrate.py. If it succeeds, you'll see output like this. (If you see no output, then your migrate_p function isn't matching any jobs.)

2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000001' to issue 'BUG00001'.
2001-11-14 12:34:13  (P4DTI-8920)  Migrated job 'job000002' to issue 'BUG00002'.
2001-11-14 12:34:13  (P4DTI-8206)  -- Added fix for change 3326 with status closed.
...
2001-11-14 12:35:02  (P4DTI-8953)  Migration completed.

If migration fails, you'll typically get one of these kinds of error:

1. The replicator can't transform a job to an issue; for example:

   (P4DTI-6288)  No TeamTrack state in project 'Horseshoe' corresponding to Perforce state 'suspended'.

   This means that the translate_jobspec function needs to consider more possibilities (in the example, it needs to translate the Perforce status suspended into a state in TeamTrack.

2. The replicator can't create an issue in the defect tracker; for example:

   (P4DTI-522X)  Can't create Bugzilla bug without component field.

   This means that the prepare_issue function has failed to meet all the defect tracker constraints (in the example, it needs to provide a value for the component field).

When you've fixed the problem, run python migrate.py again. Provide the command-line option -s JOBNAME to start migrating at a particular job. For example, if migration failed because of a problem job000123, then you might continue migrating with python migrate.py -s job000123.

Check the migrated issues. Are they as you expected? If not, fix the problem and run the migrator again.

Replace the existing jobs in Perforce with jobs replicated from the defect tracker's database by refreshing them as described in section 9.2, "Refreshing jobs in Perforce", of the Perforce Defect Tracking Integration Administrator's Guide.

Check the consistency of the replicated data by running the consistency checker as described in section 7.3, "Checking data consistency", of the Perforce Defect Tracking Integration Administrator's Guide. If this fails you may be able to fix up problems by hand. If not, you can restore your Perforce server from the checkpoint you made earlier.

Both the defect tracker and the Perforce server now contain transformed issues, and the system is prepared for starting the P4DTI replicator.

5. Integrating with multiple Perforce servers

The P4DTI can be set up to replicate defects from a single defect tracker to multiple Perforce servers. For example, if you have several development groups, each with their own Perforce server, you can set up the P4DTI to replicate the issues for each group to the appropriate Perforce server.

To do so:

1. Install a copy of the P4DTI for each Perforce server, as described in readme.txt.

2. Configure each copy of the P4DTI. These configuration parameters must be different for each copy:

   1. rid, so that the replicators can distinguish the issues they are responsible for.
   2. sid and p4_server_description, so that the defect tracker can report which Perforce server the issue is replicated to.
   3. p4_port, so each replicate can find its Perforce server (depending on how you have the Perforce servers set up, you may need to change p4_user and p4_password as well).
   4. replicate_p, so that the replicators don't try to replicate the same issues.

3. Start and manage each replicator as described in the Perforce Defect Tracking Integration Administrator's Guide.

Example. You have Perforce servers for your development groups in France and Germany and a single TeamTrack server. Projects 4, 7 and 11 are worked on in France and projects 5, 6, 9 and 10 are worked on in Germany. So you set up one P4DTI copy with the following in config.py:

rid = "replicator_fr"
sid = "france"
p4_port = "perforce.company.fr:1666"
p4_server_description = "French development group"
def replicate_p(issue):
    return issue["PROJECTID"] in [4,7,11]

and the other with the following:

rid = "replicator_de"
sid = "germany"
p4_port = "perforce.company.de:1666"
p4_server_description = "German development group"
def replicate_p(issue):
    return issue["PROJECTID"] in [5,6,9,10]

Notes:

1. The P4DTI doesn't support integration between one Perforce server and multiple defect tracking servers.

2. There's no way to move issues from one Perforce server to another. Once an issue is replicated to one Perforce server then it will always be replicated to that server, even if it later matches the replicate_p function for another replicator. If it's going to be doubtful which development group is going to work on an issue, then you need to design your workflow so that issues don't get replicated until it's clear who's going to work on them.

   For example, you might have a replicate_p function like this:

   def replicate_p(issue):
       return (issue["PROJECTID"] in [4,7,11]
               and issue["STATE"] == 19)
   

   (where state 19 is "Assigned"), so that you get an opportunity to take a decision on the issue and possibly move it to a different project, before it gets replicated.

6. Querying the TeamTrack database

When integrating with TeamTrack, in order to write replicate_p and prepare_issue functions, you need to know the key values for the tables in the TeamTrack database.

You can do this by looking directly at the TeamTrack database using your database tool. Or you can use the teamtrack_query.py script that comes with the P4DTI.

Use it as follows:

1. Provide values in config.py for the teamtrack_server, teamtrack_user and teamtrack_password configuration parameters.

2. Run the command python teamtrack_query.py TABLE to query a table and show all fields for all records.

3. Run the command python teamtrack_query.py TABLE FIELD1 FIELD2 ... to find the contents of a table and restrict the output to the named fields.

4. Add the option -q QUERY to restrict the output to the rows matching the query.

Use the TeamTrack database schema [TeamShare 2001-06-19a] to work out which table you need to query and how to understand the output.

Example 1. You want to replicate issues from projects called "Horseshoe" and "Nail". The TeamTrack database schema says that the value in an issue's PROJECTID field is the ID field of a record in the PROJECTS table. So you run this command:

> python teamtrack_query.py PROJECTS ID NAME
+------+----------------+
|  ID  |  NAME          |
+------+----------------+
|  1   |  Base project  |
|  2   |  Horseshoe     |
|  3   |  Cart          |
|  4   |  Nail          |
|  ... |  ...           |
+------+----------------+

So you can write:

def replicate_p(issue):
    return issue["PROJECTID"] in [2,4]

Example 2. When a new job is created in Perforce, you want to create an issue in TeamTrack with issue type "BUG". The TeamTrack database schema says that an issue type is the ID field of a record in the SELECTIONS table where the PREFIX field gives the prefix for the issue id. So you run the command:

> python teamtrack_query.py SELECTIONS ID PREFIX NAME
+------+----------+------------------------+
|  ID  |  PREFIX  |  NAME                  |
+------+----------+------------------------+
|  0   |          |  (None)                |
|  1   |  BUG     |  Bug report            |
|  2   |  ENH     |  Enhancement proposal  |
|  ... |  ...     |  ...                   |
+------+----------+------------------------+

So you can write:

def prepare_issue(issue):
    issue["ISSUETYPE"] = 1

A. References

B. Document History

This document is copyright © 2001 Perforce Software, Inc. All rights reserved.

Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer.

This document is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the copyright holders and contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this document, even if advised of the possibility of such damage.