Perforce Defect Tracking Integration Project

Perforce Defect Tracking Integration Integrator's Guide

Gareth Rees, Ravenbrook Limited, 2000-10-16

Contents

1. Introduction

This is the Perforce Defect Tracking Integration (P4DTI) 1.1 Integrator's Guide. It explains how a developer could extend the P4DTI to work with defect tracking systems that aren't supported by the standard distribution, or adapt the P4DTI to work with a supported defect tracker but in some way that isn't supported.

The intended readership is developers adapting or extending the P4DTI, and project staff.

This manual is not confidential.

I use some words in a precise way to express the importance of an instruction.

  1. I say "must" when the instruction is critical. This means that the integration will fail if the instruction is not followed.

  2. I say "should" when the instruction is essential. This means that integration will be of noticeably lower quality than the supported integrations if the instruction is not followed. However, it won't fail.

  3. I say "may" when the instruction is optional. This means that the integration will not suffer much if you don't follow the instruction.

2. Understanding the P4DTI

This section covers the knowledge that you will need in order to extend or adapt the P4DTI. It gives an overview of the requirements, architecture and design of the P4DTI, with references to the documents that provide more detail. You must have a good overall understanding of the P4DTI in order to extend or adapt it.

2.1. Prerequisites

This manual assumes you are familiar with the following subjects:

  1. The jobs subsystem of Perforce, and the relationship between jobs, fixes and changelists [Perforce 2000-12-22a, 10].

  2. How the P4DTI works, from the administrator's point of view. I strongly recommend that you download, install, configure and run one of the supported integrations, following the Perforce Defect Tracking Integration Administrator's Guide [RB 2000-08-10a], so that you know what the administrator has to know and do, where the data is stored, what problems can occur.

  3. How the P4DTI works, from the user's point of view. I strongly recommend that you try out one of the supported integrations, carrying out all the tasks in the Perforce Defect Tracking Integration User's Guide [RB 2000-08-10b], so that you know what it's like to use, and what benefit the users get.

  4. The programming language Python. See the Python web site <http://www.python.org/> for downloads and documentation. If you're new to Python, try the tutorial [van Rossum 2000-10-16], or the book Programming Python [Lutz 1996].

2.2. Requirements overview

The five most important requirements are these [GDR 2000-05-24, 1-5]:

  1. Defect tracker state is consistent with the state of the product sources.

  2. The defect tracking integration makes the jobs of the developers and managers easier (i.e. make it easier for them to produce a quality product etc.).

  3. It is easy to discover why the product sources are the way they are, and why they have changed, in terms of the customer requirements.

  4. The interface that allows Perforce to be integrated with defect tracking systems is public, documented, and maintained.

  5. The integration provides the ability to ask questions involving both the defect tracking system and the SCM system.

The P4DTI meets requirement 1 and requirement 5 by replicating data between the defect tracker and Perforce (see section 2.3). It meets requirement 2 by making possible for developers to do their routine defect tracking activity entirely from Perforce (by making the defects available through Perforce's jobs interface by replicating them). It meets requirement 3 by supplying a user guide [RB 2000-08-10b] that describes a development process in which issues are linked to changes by making fixes in Perforce. It meets requirement 4 by making the project sources and documents available to the public.

See the Perforce Defect Tracking Integration Project Requirements [GDR 2000-05-24] for a full and maintained set of requirements and their original sources.

2.3. Architecture overview

The P4DTI meets these requirements using a replication architecture [RB 2000-08-10c]. A replicator process repeatedly polls two databases (Perforce and the defect tracker) and copies entities from one to the other. This makes and keep them consistent, to meet requirement 1; it makes them available to users of both systems, to meet requirement 2; and it makes them available for queries combining data from both systems, to meet requirement 5.

The replicator replicates four relations:

  1. Issues are replicated from the defect tracker to Perforce jobs and back again. The replication goes in both directions, but the Perforce jobs should be considered as a subsidiary copy of the real data in the defect tracker. This means that when the two databases differ (for example, because they have been changed simultaneously) the defect tracker is considered to be definitive.

  2. Changelist descriptions are replicated from Perforce to the defect tracker.

  3. Fixes (links between issues and changelists) are replicated in both directions.

  4. Filespecs (links between issues and files) are replicated in both directions.

The P4DTI replicates the filespec relation in order to support use cases like "Associate revisions of documents with task" [GDR 2000-05-03, 6.2] and "Check out copies of revisions of documents associated with task" [GDR 2000-05-03, 6.3] and to support defect trackers like DevTrack by TechExcel that provide a revision control interface based on associating documents with an issue. However, because the supported defect trackers (TeamTrack and Bugzilla) have no such feature, and because alpha and beta testing showed no demand for use cases involving associating documents with tasks, we haven't made any use of this relation (for example, it's not documented in the user's guide). However, it's there if you need it for integrating with your defect tracker.

2.4. Design overview

The replicator is designed to be highly independent of both Perforce and the defect tracker, using public interfaces wherever possible, so that the integration doesn't have to change frequently to keep up with the systems it integrates (requirement 27) and the cost of maintenance is low (requirement 30). It runs as a separate process and uses public protocols to access both databases. It doesn't require any special support from either system (though users will benefit if the defect tracker provides an interface to Perforce fixes; (see section 10).

The replicator is written in the interpreted programming language Python, a portable, stable, readable and open programming language (to meet requirement 21, requirement 24, requirement 25, and requirement 26.).

Figure 1 below shows the broad outlines of how the replicator is constructed. Parts in black are shared by the integrations with all defect trackers. The components in red are the components that you must write in order to integrate with your defect tracker.

If you need to modify any other components to integrate with your defect tracker, that's a defect in the integration kit. Please report it (see section 12.1) or make the necessary modifications and submit them as a contribution (see section 12.2).

Figure 1. Replicator block diagram

Replicator block diagram

The replicator is divided into these components:

  1. replicator.py, a module that runs the replication algorithm and reports failures by e-mail [GDR 2000-09-13].

  2. A defect tracker module, called dt_defect_tracker.py (dt_abc.py in figure 1). This defines a subclass of defect_tracker. It is responsible for fetching, translating and updating entities in the defect tracker database at a high level of abstraction (see section 7). For each new defect tracker, you need to write such a module.

  3. A defect tracker interface. This provides a low-level interface to the defect tracker, allowing entities to be fetched from and stored in records in the defect tracker's database (see section 6). For each new defect tracker, you need to write such a module.

  4. A configuration generator. This is responsible for taking the raw configuration supplied by the P4DTI administrator and building a detailed configuration suitable for the replicator (see section 8).

  5. Supporting components, including an interface to Perforce, a message system, and configuration checking functions.

3. What you need to do

This section gives an overview of the work required in adapting an existing integration or developing a new integration.

3.1. Before you start work

Someone might already have developed the integration or adaption that you plan to work on. Take a look at the P4DTI contributions page <http://www.ravenbrook.com/project/p4dti/contrib/>.

Someone might be currently be working on the integration or adaption that you plan to work on. If so, Perforce support may know about them.

The feature you want may in fact be part of the supported P4DTI product and it is missing from the manuals or the manuals unclear. If so, Perforce support can tell you. And if the manuals are unclear or missing information, then please submit a defect report (see section 12.1).

3.2. Getting help

The Perforce Defect Tracking Integration Kit is a supported product. If you have trouble adapting the P4DTI or developing an integration after following the instructions in here, you should contact Perforce support for help.

Ravenbrook Limited may be able to develop or consult on adaptions and extensions to the P4DTI.

3.3. Adapting an existing integration

This section gives an overview of what you should do if you want to adapt the P4DTI to work with a supported defect tracker but in some way that isn't supported. For example:

  1. You want the replicator to leave your Perforce jobspec unchanged.
  2. You want more control over which defect tracker issues are replicated to Perforce.
  3. You want to change the names of the fields in Perforce.
  4. You want to match up the fields in an unusual way, for example you have a single field in the defect tracker that ought to be two fields in Perforce (or vice versa).
  5. You want to translate values in a field in a different way.
  6. You want to use a locally customized version of Bugzilla.

In these and many similar cases, you should make the P4DTI do what you want by writing a "configuration generator" (see section 8.6).

But don't skip straight to that section. At least skim the rest of the manual. You'll need to understand many of the details in order to write a configuration generator, especially how to write translator classes (see section 7.5) and how the configuration works (see section 8).

3.4. Developing a new integration

This section gives an outline of the steps you need to take to integrate Perforce with a new defect tracker.

  1. You must choose a name for the integration. This should be the name of the defect tracker, for example "TeamTrack" or "Bugzilla". This name (when converted to lower case) must be used as part of the names of modules making up the integration (see section 7 and section 8).

  2. You must provide full implementations of these components:

    1. A documented design for extensions for the defect tracker database schema (see section 4);

    2. A Python interface to the defect tracker (see section 6);

    3. A defect tracker module (see section 7);

    4. A configuration generator (see section 8).

  3. You should develop and apply tests (both automated and manual) of your integration (see section 9).

  4. You should provide a defect tracker interface to the Perforce relations, if possible (see section 10).

  5. You must adapt or extend these components:

    1. The configuration module config.py (see section 8.5).

    2. The Administrator's Guide (see section 11);

    3. The User's Guide (see section 11);

    All other components are designed to be portable between defect trackers. If your integration cannot be made to work without changing the portable components, then there is a defect in the P4DTI Integration Kit. Please report this (see section 12.1).

  6. Once all the work outlined above is completed and tested to your satisfaction, you should make your work available to the community so that others can benefit from your efforts (see section 12.2).

I estimate that at least 10 weeks of effort are required to develop, test, document and release a new integration [GDR 2000-05-30]

4. Defect tracker database schema extensions

You must extend the database schema by adding new fields to the issue relation (see section 4.1), and adding three new relations: the changelist relation (see section 4.2), the fixes relation (see section 4.3), and the filespecs relation (see section 4.4). You should add another relation to the database, to store the replicator state and configuration (see section 4.5). These schema extensions must be documented so that users of your integration can implement database queries and reports that use this data, to meet requirement 5.

These relations should be stored in separate tables if possible, to most easily support queries and reporting using standard database tools. However, some defect trackers do not support this.

Example. TeamTrack release 4.5 doesn't support the addition of tables to its database schema, so the TeamTrack schema extensions squash these relations into a single table, using a type field to distinguish them [GDR 2000-09-04, 2.1].

The design must support multiple replicators replicating from a single defect tracker, and support a single replicator replicating to multiple Perforce servers from one defect tracker, in order to meet requirement 96. To support this, each relation includes a replicator identifier which identifies the replicator which is handling replication for that record, and a Perforce server identifier, which identifies the Perforce server that the record is replicated to.

Examples. The TeamTrack database schema extensions [GDR 2000-09-04] and the Bugzilla database schema extensions [NB 2000-11-14b].

4.1. Issues

The issue relation must be extended with these fields:

Field contents Field type
Replicator identifier of the replicator that is in charge of replicating this issue, or the empty string or NULL if the issue is not replicated. char(32)
Server identifier of the Perforce server to which this issue is replicated, or the empty string or NULL if the issue is not replicated. char(32)
Name of Perforce job to which this issue is replicated, or the empty string or NULL if the issue is not replicated. char(1024) (or varchar(1024) since most jobnames are short).

You may add these fields to the defect tracker's issue table, or you may store them in a separate table and use the issue key to relate the two tables.

Examples. The TeamTrack integration adds the new fields to the existing TS_CASES table [GDR 2000-09-04, 3.1]. The Bugzilla integration creates a table p4dti_bugs containing the new fields and associates them with the bugs table using the bug_id field [NB 2000-11-14b].

4.2. Changelists

The changelist relation has these fields:

Field contents Field type
Replicator identifier. char(32)
Perforce server identifier. char(32)
Change number. int
User who created the change. A foreign key reference to the defect tracker's user relation giving the user who created or submitted the change.
Change status. An enumeration with two values: pending or submitted.
Date the change was last modified. A date and time.
Change description. Text, unlimited in length.
Client from which the change was submitted. char(1024) (or varchar(1024) since most client names are short).

The combination of (change number, Perforce server identifier) is the primary key for this relation: there can only be one change with a particular number on a Perforce server.

4.3. Fixes

The fixes relation has these fields:

Field contents Field type
Replicator identifier. char(32)
Perforce server identifier. char(32)
Issue. A foreign key reference to the defect tracker's issue relation, giving the issue which is fixed by the change.
Change number int
Date the fix was last modified. A date and time.
User who created the fix. A foreign key reference to the defect tracker's user relation, giving the user who last modified the fix.
Status the job was/will be fixed to. char(1024) (or varchar(1024) since most job statuses are short).
Client from which the fix was made. char(1024) (or varchar(1024) since most client names are short).

The combination of (issue, change number, Perforce server identifier) is the primary key for this relation: there can only be one fix between a change and an issue on a Perforce server.

4.4. Filespecs

The associated filespecs relation has these fields:

Field contents Field type
Replicator identifier. char(32)
Perforce server identifier. char(32)
Issue. A foreign key reference to the defect tracker's issue relation, giving the issue which is fixed by the change.
Filespec. text

4.5. Replicator configuration and state

By design, the replicator has no internal state. This is to make the replicator robust against losing a network connection, or the machine it's running on crashing in the middle of a replication: when the network comes back up or it starts again, it tries the replication again [GDR 2000-09-13, 2.9]. This design principle helps to meet requirement 1 (consistency between databases).

This means that if you need to store information, such as a record of which changes have been replicated (see section 4.6) you must store it in the defect tracker's database.

The replicator also needs to pass information to the defect tracker, to support an interface from the defect tracker to Perforce (see section 10). There are two configuration parameters which should be communicated to the defect tracker by storing them in a configuration table: changelist_url and p4_server_description.

4.6. Discovering what's changed

The replicator works by repeatedly polling the databases, so you must provide a way to tell it which issues have changed since the last time it polled. Here are some strategies:

  1. If the defect tracker has a changes table which records the history of changes to issues, then store a record number in the replicator state that gives the last record in the changes table that has been replicated.

    Example. The TeamTrack integration uses this approach [GDR 2000-09-04, 3.5].

  2. If the defect tracker has a last modified date field in the issue table, store the value of this field at the point when the replicator was last replicated. Then you can fetch the changed issues by looking for issues whose last modified date is greater than the last replicated date. This is likely to be less efficient than solution 1.

  3. Modify the defect tracker so that it supports solution 1 or 2.

  4. If all else fails, you could store a "shadow" table of issues, containing copies of the issue records as they were when last modified. Then you can find changed issues by finding differing corresponding records. This is likely to be very inefficient.

4.7. Distinguishing replicated changes from other changes

The replicator needs to distinguish the changes it made from changes made by other users of the defect tracker. Otherwise it will attempt to replicate its own changes back to Perforce. This won't actually end up in an infinite loop of replication, since when it replicates back it will discover that there are no changes to be made, and so not actually do anything. However, this double replication gives twice the opportunity for conflicts, and hence annoying e-mail messages for the users of the P4DTI (see Ravenbrook issue job000016).

Here are some strategies:

  1. Suppose that the defect tracker has separate concepts of "logged in user" and "user who is making the change". In this case, make a special user to represent the replicator and have the replicator log in as that user. The replicator's changes show up with logged in user being the replicator user; all other changes need to be replicated.

    Example. The TeamTrack integration uses this approach [GDR 2000-09-04, 5].

  2. You could store a table listing the changes that were made by the replicator. Any other changes need to be replicated.

  3. If the defect tracker has a last modified date field in the issue table, store the value of this field at the point when the replicator was last replicated. Then an issue has been changed by someone else if its last modified date differs from the last replicated date.

4.8. Perforce users who don't have licences in the defect tracker

The replicator replicates user fields in issues, changelists and fixes (for example, the owner of an issue or the user who submitted a changelist) by applying a user translation function (see section 7.5.4). When a defect tracker user has no licence in Perforce, the translation function can simply use that user's defect tracker login name, since Perforce doesn't validate user fields in jobs. But if a Perforce user has no licence in the defect tracker, the translator needs to do something with them. For issues your defect tracker interface should simply refuse to replicate when a Perforce user has no licence in the defect tracker. But you should be less strict when replicating fixes and changelists: while it is a sensible policy (and required by some defect tracker vendors) to require the current owner of a job in Perforce to have a licence, it is not sensible to require every user in Perforce who ever submitted a changelist to have a licence in the defect tracker.

Example. The TeamTrack integration maps unknown users in changelists and fixes to the special TeamTrack user 0 (representing "no user"). When there's an unknown user in an issue, the integration rejects the attempt to replicate it by raising an error.

5. Coding conventions

This section covers coding conventions followed in the P4DTI. You should follow these conventions in your adaptions and extensions. They will make your code more reliable and easier to debug. It will be easier for users to diagnose problems and fix them. If you contribute your code for inclusion in the P4DTI (see section 12.2) then it will be easier for us to integrate your contribution.

Examples. Look at the TeamTrack module, dt_teamtrack.py, and the Bugzilla module, dt_bugzilla.py for uses of all conventions and features covered in this section.

5.1. Messages

The message.py module defines a class of messages. You must use this class when writing messages to the replicator's log (see section 5.3). You should use this class when raising errors (see section 5.4).

You create a message like this:

import message
id = 123
text = "Constructed a test message."
priority = message.DEBUG
product = "Test"
msg = message.message(id, text, priority, product)

The four arguments to the constructor are as follows:

  1. The id argument is a message identifier (an integer). This must be unique among all messages generated by the product.

  2. The text argument is the text of the message (a string).

  3. The priority argument is the level of importance of the message. This must be one of the constants in the following table:

    message.CRIT A critical error: the replicator will stop immediately. Use this priority for errors in configuration discovered by your configuration generator or by the __init__() and init() methods of your defect tracker interface.
    message.ERR An error. The replicator can't complete some operation. Use this priority for errors discovered during replication, such as untranslatable fields or permission failures.
    message.WARNING

    A warning. The replicator can continue, but the administrator may want to take some action.

    Example. The TeamTrack integration issues a warning when it finds two entities with the same name in the database (for example, two projects, or two resolutions). The integration can continue (it just uses the first entity it finds), but it may not do what users expect.

    message.NOTICE

    A significant but expected condition.

    Example. The replicator uses this priority when it has to overwrite a Perforce job with a defect tracker issue. This is the correct thing to do, and is the documented behaviour [UG, 2.2] but it is still a significant event.

    message.INFO For information only.
    message.DEBUG Unlikely to be useful except for debugging.

  4. The product argument is the name of software product which generated the message. For the supported P4DTI, this must be "P4DTI".

You can format a message as text by converting the message object to a string:

>>> str(msg)
"(Test-123X)  Constructed a test message."

Note that a check digit has been appended to the message identifier. (The check digit uses a mod-11 algorithm similar to that used in ISBNs, so the check digit can be 0-9 or X.) The idea of the check digit is so that Perforce support can ask users for the message identifier of the error that they are reporting. The check digit makes it very likely likely that if the error is misreported or misheard the problem will be detected.

You can wrap a message to some number of columns by calling its wrap() method:

>>> print msg.wrap(25)
(Test-123X)  Constructed
a test message.

5.2. Message catalogs

You may create each message when you need it, but you should use a message catalog. A catalog helps you keep message identifiers distinct and internationalizes your code.

A message catalog is a dictionary that maps message identifier to a tuple of two elements: the message priority, and a format string that can be used to build the message text. For example:

# Test catalog in English
test_en_catalog = {
    123: (message.DEBUG, "Constructed a test message."),
    124: (message.CRIT, "Couldn't connect to defect tracker on host '%s'."),
    125: (message.ERR, "User '%s' has no permission to edit issue '%s'."),
    126: (message.INFO, "Replicated issue %d."),
    # ...
}

Once you have a message catalog for a product, you should build a message factory that will dispense messages from that catalog, like this:

import message
product = "Test"
factory = message.catalog_factory(test_en_catalog, product)

Now you can construct a message by calling the factory's new() method and passing the message identifier, and the arguments for the format string:

msg1 = factory.new(124, 'dt.ravenbrook.com')
msg2 = factory.new(125, ('gdr', 'BUG00123'))

See the catalog.py module for the P4DTI catalog and message factory.

5.3. Logging

The P4DTI logs its progress and errors by creating messages (see section 5.1) and sending them to a "logger": that is, an instance of the logger class defined by the logger.py module.

The logger module defines classes for logging to files, to standard output, and to the system log on Unix. The multi_logger class directs a single message to several loggers. Each logger class takes a priority argument on instantiation: only messages with this priority, or a higher priority, will appear in the log.

You should log as many debugging messages as you like (by default the log_level configuration parameter is message.INFO so these messages won't appear). You should log informational messages sparingly, and only when you actually make a change in a database. You should not log error messages, but should raise them as exceptions instead (see section 5.4); the replicator will log them for you when it catches them.

To add a message to a log, create a message object (see section 5.1) and pass it to the logger's log() message:

import logger
# Log messages of priority INFO and higher to test.log:
logger_object = logger.file_logger("test.log", message.INFO)
msg = factory.new(126, issue_id)
logger_object.log(msg)

The configuration generator (see section 8) must construct a logger object for use by the replicator. The same logger object should be used by the defect tracker module (see section 7) as well, so that all messages are collected in the same place. You must allow the P4DTI administrator to control the volume of log messages by setting the log_level configuration parameter.

5.4. Errors

In the P4DTI, errors are indicated by raising a Python exception, not by returning an exceptional value.

Raise an error using a string as the exception object, and a message object (see section 5.1) as the message. For example:

error = "Example error"
# ...
raise error, factory.new(124, 'dt.ravenbrook.com')

It doesn't make any difference what priority you give to the message, but it is conventional to use message.CRIT when the replicator will stop (for example, configuration errors), and message.ERR when the replicator will continue (for example, untranslatable fields or permission failures).

5.5. Source code layout

You should include in each file of source code:

  1. The author.

  2. An introduction explaining what the file is intended to achieve: for example, which requirements does it help to meet?

  3. A references section listing the sources you've used in preparing the code and which other people should read in order to understand it. You should certainly refer to the appropriate sections in this manual.

  4. A history section listing the changes made to the code, with the date and the person who change it.

  5. A statement of copyright.

  6. A licence giving people permission to copy the file under certain conditions (or denying them permission if that's what you intend).

Example. The logger.py module displays all these features.

6. The Python interface to the defect tracker

You'll need a way for Python to read and write defect tracking records. If the defect tracker has an API of some sort, you'll need to use that; if not, you'll have to read and write the database directly, using one of the Python database interfaces. Your defect tracker interface will need to support these kinds of operations:

  1. Get an issue record.

  2. Update an issue record.

  3. Get all the issues needing replication.

  4. Get all the fixes for an issue.

  5. Add/update/delete a fix.

  6. Create a table.

  7. Add a field to a table.

  8. Get a list of the fields that make up the issue relation, together with the field types, lengths, legal values, etc.

  9. Get a list of users, with names, userids, e-mail addresses.

I can't give you a complete or precise list of operations here; you'll have to see what's required as you implement your schema extensions (see section 4) and defect tracker module (see section 7).

Example. The TeamTrack integration uses the TeamShare API to connect to the defect tracker, because the API provides methods that apply TeamTrack's privilege system and database validation. The integration uses a Python extension module that provides an interface to the parts of the TeamShare API that it needs (only a small part of the whole API, as it happened). See the Python interface to TeamTrack and its design [GDR 2000-08-08].

Example. Bugzilla has no API: you have to understand the Bugzilla database schema [NB 2000-11-14a] and connect directly to the MySQL database. The Bugzilla integration uses a wrapper module that encapsulates the direct database operations as defect tracker oriented functions like update_bug. See bugzilla.py and its design [NB 2000-11-14c].

7. The defect tracker module

You must create a module called dt_defect_tracker.py (where defect_tracker is the lower-case form of the name your chose for your defect tracker (see section 3)) that implements these classes:

  1. The defect tracker interface itself: a subclass of dt_interface.defect_tracker (see section 7.1).

  2. Defect tracker issue: a subclass of dt_interface.defect_tracker_issue (see section 7.2).

  3. Defect tracker fix: a subclass of dt_interface.defect_tracker_fix (see section 7.3).

  4. Defect tracker filespec: a subclass of dt_interface.defect_tracker_filespec (see section 7.4).

  5. A translator between dates in the defect tracker and Perforce: a subclass of translator.translator (see section 7.5.1).

  6. A translator between multi-line text fields in the defect tracker and Perforce: a subclass of translator.translator (see section 7.5.3).

  7. A translator between users in the defect tracker and Perforce: a subclass of translator.user_translator (see section 7.5.4).

  8. Any other translator classes that will be needed to translate fields in the issue relation (see section 7.5).

Examples. The TeamTrack module, dt_teamtrack.py, and the Bugzilla module, dt_bugzilla.py.

7.1. The dt_interface.defect_tracker class

A subclass of dt_interface.defect_tracker implements the replicator's interface to a defect tracker.

Examples. The dt_teamtrack.py module defines a class dt_teamtrack. The dt_bugzilla.py module defines a class dt_bugzilla.

A subclass of dt_interface.defect_tracker must define the following methods:

__init__(self, config)

This is called when the defect tracker object is created. The config parameter is an object whose attributes are the configuration parameters for the defect tracker. See section 8 for the details of how configuration parameters end up in this object.

This method should check that all configuration parameters are supplied and have valid values. Use the methods in check_config.py for basic checks.

The required parameters should certainly include changelist_url, p4_server_description, rid, sid, and start_date, but may include others, either supplied by the P4DTI administrator in config.py or generated by the configuration generator.

all_issues(self)

Return a list of all defect tracking issues that have been modified since the starting point for replication (that is, the date given by the P4DTI administrator in the start_date parameter).

Include in this list:

  1. Issues replicated by this replicator (that is, the replicator identifier for those issues matches the rid configuration parameter);

  2. Issues not replicated by any replicator (that is, the replicator identifier for those issues is blank).

Omit from this list:

  1. Issues replicated by a different replicator (that is, the replicator identifier for those issues differs from the rid configuration parameter);

  2. Issues unchanged since the start date.

Each element of the returned list must belong to your subclass of the dt_interface.defect_tracker_issue class (see section 7.2).

changed_entities(self)

This method is called at the start of each replication cycle to determine what work there is to do.

It must return a tuple of three elements:

  1. A list of defect tracking issues that require replication.

    Each element of the returned list must belong to your subclass of the dt_interface.defect_tracker_issue class (see section 7.2).

    Include in this list:

    1. Issues replicated by this replicator (that is, the replicator identifier for those issues matches the rid configuration parameter);

    2. Issues not replicated by any replicator (that is, the replicator identifier for those issues is blank). The replicator will want to consider these issues as candidates for replication.

    Omit from this list:

    1. Issues replicated by a different replicator (that is, the replicator identifier for those issues differs from the rid configuration parameter).

    2. Issues known to be up to date with Perforce; either because they are unchanged since they were last replicated, or because they have only been changed by the replicator (4.6 and 4.7).

  2. The empty list [ ]. (This is for symmetry with the Perforce interface, which returns a list of changelists. Since changelists are not editable in the defect tracker, there's nothing that can be returned here, hence the empty list.)

  3. A marker. This must be some token that identifies what has been done on this poll. At the end of the replication cycle it will be passed to mark_changes_done().

    Example. The TeamTrack integration uses the record number of the last record in the TS_CHANGES table that the replicator looked at as the marker indicating what it's done. See dt_teamtrack.py and the design [GDR 2000-09-04, 3.5].

This method must not record that the issues it returns have been considered for replication or replicated. The replicator can encounter an error during the course of replication that prevents it from making any progress (Perforce can go down, the defect tracker can go down, the replicator can crash). When the system comes back up, the replicator must re-consider these issues and possibly replicate them again. This helps keep the databases consistent (requirement 1) and is consistent with the design principle that the replicator must have no internal state.

Recording that issues have been replicated must be left for the end of each replication cycle, when the marker (item 3 in the tuple) will be passed to mark_changes_done().

init(self)

This method is called each time the replicator starts.

The method must initialize the defect tracking database so that it is ready to start replication. The tables and fields in your schema extensions (see section 4) must be added if they are not yet present.

issue(self, issue_id)

Return the defect tracking issue identified by the issue_id argument, or None if there is no such issue. The returned issue (if any) must belong to your subclass of the dt_interface.defect_tracker_issue class (see section 7.2).

The issue_id argument is a string identifying the issue (see section 7.2.1).

mark_changes_done(self, marker)

This method is called at the end of each replication cycle, when all issues have been replicated.

The marker argument is the third item in the tuple returned by the changed_entities() method at the start of the replication cycle.

This method must now record that it has considered all changes up to the start of this replication cycle and replicated them successfully, so that at the next replication cycle it can ignore these changes and consider a new set of changes (see section 4.6).

replicate_changelist(self, change, client, date, description, status, user)

Replicate a changelist to the defect tracker database (see section 4.2).

The arguments specify the changelist; these arguments correspond to a subset of the fields in the changelist relation in the Perforce database (the names of the actual files changed, and their new revision numbers, are not replicated).

  1. The change argument, an integer, is the change number.

  2. The client argument, a string, is the client on which the change was last modified.

  3. The date argument is the date and time at which the change was last modified. It has been converted by the date translator (see section 7.5.1).

  4. The description argument is the change comment. It has been converted by the text translator (see section 7.5.3).

  5. The status argument is a string. It is "pending" if the changelist is pending, "submitted" otherwise.

  6. The user argument is the user who last modified the changelist. It has been converted by the user translator (see section 7.5.4).

This method must return 1 if the changelist was new or changed, or 0 if it was unchanged.

7.2. The dt_interface.defect_tracker_issue class

A subclass of dt_interface.defect_tracker_issue implements the replicator's interface to the issues in a defect tracker.

Examples. The dt_teamtrack.py module defines a class teamtrack_case (issues are called "cases" in TeamTrack). The dt_bugzilla.py module defines a class bugzilla_bug (issues are called "bugs" in Bugzilla).

7.2.1. Issue identifiers

The replicator needs a unique identifier for each issue in the defect tracker. This must be a string, so that it can be stored in the P4DTI-issue-id field in the Perforce jobspec [GDR 2000-09-13, 4.2]. The replicator gets the identifier from an issue's id() method. Later, it may pass the identifier to the defect tracker's issue() method.

Example. TeamTrack uniquely identifies issues by their record number in the database. So in dt_teamtrack module, the issue identifier is the string conversion of the record number.

7.2.2. Issues are dictionaries

The replicator considers an issue to consist of a collection of named fields, with a value for each field. Instances of the defect_tracker_issue subclass must support at least the __getitem__() method, so that the replicator can get the value for a field in an issue using the expression issue["fieldname"].

You may want to implement the whole of the Python dictionary interface for your own use, but the replicator only uses __getitem__().

A subclass of dt_interface.defect_tracker_issue must define the following methods:

__getitem__(self, field)

Return the value of the field named by the field argument. Raise KeyError if the issue has no such field.

__str__(self)

Return a string describing the issue, suitable for presentation to a user or administrator in a report. Having several lines of the form "field name: value" should be fine.

add_fix(self, change, client, date, status, user)

Add a fix to the issue (see section 4.3).

The arguments specify the fix; these arguments correspond to the fields in the fix relation in the Perforce database.

  1. The change argument, an integer, is the Perforce change number.

  2. The client argument, a string, is the Perforce client name from which the fix was made.

  3. The date argument is the date the fix was made. It has been converted by the date translator (see section 7.5.1).

  4. The status argument, a string, is the effect of the fix. It is the status the job was changed to when the fix was made (or if the fix is to a pending changelist, then this is the status the job will be changed to when the changelist is submitted).

    The status is also known as the "effect" (for example, in the defect tracker's interface to fixes (see section 9)) because it gives the effect on the job when the fix is submitted.

  5. The user argument is the user who made the fix. It has been converted by the user translator (see section 7.5.4).

corresponding_id(self)

If this issue has been replicated, return the name of the Perforce job to which this issue is replicated.

If this issue has not yet been replicated, return the name for the Perforce job to which this issue will be replicated. The returned value must be legal as the name of a Perforce job. You may want to use the result of the readable_name() method if that is suitable.

filespecs(self)

Return a list of the filespecs associated with this issue. Each item in the list belongs to your subclass of the defect_tracker_filespec class (see section 7.4).

fixes(self)

Return a list of the fixes for this issue. Each item in the list belongs to your subclass of the defect_tracker_fix class (see section 7.3).

id(self)

Return a string that can be used to uniquely identify this issue among all the issues in the defect tracker and to fetch it in future (see section 7.2.1).

readable_name(self)

Return a string giving a human-readable name for the issue, as a string. This name is only used in logs and e-mail messages.

replicate_p(self)

A function that decides, for an issue that is not yet replicated, whether it should be replicated by the running replicator. It must return 1 if the issue should be replicated, or 0 if it should not.

The replicator calls this method for each issue returned by the defect tracker's changed_entities() method which is not replicated: that is, its rid() method returns the empty string.

If this method returns 1, then the next thing the replicator will do is to call the issue's setup_for_replication() method.

The configuration parameter replicate_p can be used by the P4DTI administrator to set this method, for example if the organization only wants to replicate issues for a selected set of projects.

rid(self)

Return the replicator identifier of the replicator that is in charge of replicating this issue, or the empty string if the issue is not being replicated.

setup_for_replication(self, jobname)

Set up the issue for replication. That is, record that the issue is replicated by this replicator and record any other information in the database that is needed to replicate this issue.

You must do at least these three steps:

  1. Record that the issue is replicated by this replicator, so that in the future its rid() method returns the correct replicator identifier (this is the rid parameter in the configuration passed to the defect tracker class when it was instantiated).

  2. Record the Perforce server identifier of the Perforce server it is replicated to (this is the sid parameter in the configuration passed to the defect tracker class when it was instantiated).

  3. Record that the issue is replicated to the Perforce job named by the jobname argument, so that in future its corresponding_id() method returns jobname.

See section 4.1.

update(self, user, changes)

Update the issue in the defect tracker's database.

The user argument is the user who made the change. It has been converted by the user translator (see section 7.5.4).

The changes argument is a dictionary of the changes that must be applied to the issue. The keys of the dictionary are the names of the fields that have changed; the values are the new values for those fields. Each value in the dictionary has been converted by the appropriate translator. If changes is the empty dictionary, then do nothing.

If the defect tracker supports transitions in a workflow, then this method should deduce the transition to apply (if any) based on the old and new values for the issue fields.

Example. The TeamTrack integration attempts to find and apply a transition when the STATE field changes. It looks at all the available transitions for the issue and selects the transition that would cause result in the correct new state.

Example. Bugzilla doesn't have transitions, so there's no need for the Bugzilla integration to deduce one.

This method must check that the proposed change to the issue is legal in the defect tracker. (The changed fields have been converted by their translators, so each is legal individually, but the defect tracker may be more stringent, for example it may require a field not to have a value when the issue is in a particular state.) It must also check that the user has permission to make the proposed change. It's best if you can call a function in the defect tracker's API to apply the defect tracker's own rules (this is likely to be robust and maintainable), but if there's no such function, then you must do your best to emulate the defect tracker's checks.

If the issue can't be updated (for example, because the user doesn't have permission to make the change, or because no workflow transition can be discovered, or because the proposed change is illegal in some way) then this method must raise an error.

Example. The TeamTrack integration calls the TSServer::Transition method in the TeamShare API, which checks the issue for correctness and checks that the user has the correct privilege. All the integration needs to do is raise an error when the function rejects the transition.

Example. Bugzilla has no API, so the Bugzilla integration must emulate Bugzilla's checking. The dt_bugzilla.py module defines three checking methods: restrict_fields(), enforce_invariants(), and check_permissions().

7.3. The dt_interface.defect_tracker_fix class

A subclass of dt_interface.defect_tracker_fix implements the replicator's interface to a fix record in a defect tracker (see section 4.3).

Examples. The dt_teamtrack.py module defines a class teamtrack_fix. The dt_bugzilla.py module defines a class bugzilla_fix.

A subclass of dt_interface.defect_tracker_fix must define the following methods:

change(self)

Return the change number for the fix, an integer.

delete(self)

Delete the fix in the defect tracker so that the change is no longer linked to the issue.

status(self)

Return the status of the fix, a string.

update(self, change, client, date, status, user)

Update this fix in the defect tracker so that has the given fields. If the fields are unchanged, do nothing.

This method is called when someone makes a new fix between the change and issue of an existing fix (for example, the status used to be "open", but now is "closed"). Since there can be only one fix for a given change and issue, the replicator updates the fix rather than creating a new fix.

The arguments specify the fix; these arguments correspond to the fields in the fix relation in the Perforce database.

  1. The change argument, an integer, is the Perforce change number. This is never changed.

  2. The client argument, a string, is the Perforce client name from which the fix was made.

  3. The date argument is the date the fix was made. It has been converted by the date translator (see section 7.5.1).

  4. The status argument, a string, is the effect of the fix. It is the status the job was changed to when the fix was made (or if the fix is to a pending changelist, then this is the status the job will be changed to when the changelist is submitted).

    The status is also known as the "effect" (for example, in the defect tracker's interface to fixes (see section 9)) because it gives the effect on the job when the fix is submitted.

  5. The user argument is the user who made the fix. It has been converted by the user translator (see section 7.5.4).

7.4. The dt_interface.defect_tracker_filespec class

A subclass of dt_interface.defect_tracker_filespec implements the replicator's interface to a filespec record in a defect tracker (see section 4.4).

Examples. The dt_teamtrack.py module defines a class teamtrack_filespec. The dt_bugzilla.py module defines a class bugzilla_filespec.

A subclass of dt_interface.defect_tracker_filespec must define the following methods:

delete(self)

Delete the filespec record so that the issue is no longer associated with the filespec.

name(self)

Return the filespec, a string.

7.5. The translator.translator class

A subclass of translator.translator translates values of a particular type between the defect tracker and Perforce. You should define a translator for each field type in the defect tracker that you want the P4DTI administrator to be able to replicate. You must define translators for dates (see section 7.5.1), multi-line text fields (see section 7.5.3), and users (see section 7.5.4). If your defect tracker has any concept of the state of an issue, then you must define a translator for states (see section 7.5.2).

Example. The TeamTrack integration defines, in addition to the three required translators, translators for: fields that cross-reference an auxiliary table like TS_PROJECTS; elapsed time fields; selection fields; and the STATE field.

The translator base class doesn't know anything about Perforce; all it knows is that it is translating between two defect trackers, called 0 and 1. In the P4DTI, defect tracker 1 is always Perforce, but we haven't limited the design of the translator class by requiring that it is.

Each subclass of translator.translator must define the following methods:

translate_0_to_1(self, value, dt0, dt1, issue0=None, issue1=None)

Return value, suitably translated from defect tracker 0 to defect tracker 1. If translation is not possible, raise an error.

  1. The value argument is a value in a field in an issue in defect tracker 0.

  2. The dt0 argument is your defect tracker: an instance of your subclass of dt_interface.defect_tracker.

  3. The dt1 argument is Perforce (represented by an instance of a subclass of dt_interface.defect_tracker).

  4. The issue in your defect tracker from which the value comes, or None if the value doesn't come from an issue. An instance of your subclass of dt_interface.defect_tracker_issue.

  5. The job in Perforce to which the value is going, or None if the value isn't going to a job (represented by an instance of a subclass of dt_interface.defect_tracker_issue).

This method takes defect trackers as arguments because it may need to query the defect tracker to carry out the translation.

Example. In the TeamTrack integration, the single select translator needs to read the TS_SELECTIONS table to discover the available selections. To do this it calls the private method read_selections() in dt0.

This method takes issues as arguments because some translators need to know about the whole issue in order to carry out the translation.

Example. In the TeamTrack integration the state translator needs to know the project to which the issue belongs (because different projects may have different states with the same name which correspond to the same Perforce state).

Many translators can ignore the dt0 and dt1 arguments; most can ignore the issue0 and issue1 arguments.

translate_1_to_0(self, value, dt0, dt1, issue0=None, issue1=None)

Return value, suitably translated from defect tracker 1 to defect tracker 0. If translation is not possible, raise an error.

  1. The value argument is a value in a field in an issue in defect tracker 1 (that is, in a job in Perforce).

  2. The dt0 argument is your defect tracker: an instance of your subclass of dt_interface.defect_tracker.

  3. The dt1 argument is Perforce (represnted by an instance of a subclass of dt_interface.defect_tracker).

  4. The issue in your defect tracker 0 to which the value is going, or None if the value isn't going to an issue. An instance of your subclass of dt_interface.defect_tracker_issue.

  5. The job in Perforce from which the value comes, or None if the value doesn't come from a job (represented by an instance of a subclass of dt_interface.defect_tracker_issue).

7.5.1. Date translator

You must define a date translator class, a subclass of translator.translator, to translate dates between your defect tracker and Perforce.

When translating to Perforce:

  1. An empty or null date field must be translated to the empty string.

  2. Any other date must be translated to a string looking like "2000/12/31 23:59:59" (you can do this by calling time.strftime() with "%Y/%m/%d %H:%M:%S" as the first argument).

When translating from Perforce:

  1. The empty string must be translated to an empty or null date field.

  2. A string in the format "2000/12/31 23:59:59" specifies the calendar date. (This form is used by changelists and jobs.)

  3. A string consisting only of digits specifies the number of seconds since 1970-01-01 00:00:00. (This form is used by fixes.)

Example. TeamTrack specifies all dates as seconds since 1970-01-01 00:00:00, so the TeamTrack integration uses time.strftime() to convert from TeamTrack to Perforce, and either time.mktime() or simply int() to convert from Perforce to TeamTrack.

7.5.2. State translator

If your defect tracker has a concept of states for issues, then you must define a state translator class, a subclass of translator.translator.

The state field in Perforce should be a "select" field (see section 8.4) so the values for this field should be legal selections in Perforce. This means no whitespace, hashes, double quotes, semicolons or slashes. Since the defect tracker probably allows these character to appear in state names, you must convert them somehow.

We have provided a translator to do this conversion: it is the keyword_translator class in the translator.py module.

You shouldn't just use the keyword translator as your state translator, since all it does is to convert strings. You should develop a translator that checks that applies the keyword translator, checks that the converted state is legal and raises and error if it is not.

7.5.3. Text translator

You must define a text translator class, a subclass of translator.translator, to translate multi-line text fields between your defect tracker and Perforce.

This translator must translate line endings (if needed). Perforce uses newline ("\n") as the line ending; values always end in a newline (unless the field is empty); values never end in more than one newline.

Example. TeamTrack uses a carriage return plus a newline (\r\n) as its line ending, and there need not be a final newline.

7.5.4. User translator

You must define a user translator class, a subclass of translator.user_translator, to translate users between your defect tracker and Perforce.

It is important not to assume that userids are the same in Perforce and the defect tracker, because an organization may have different policies for assigning userids in the two systems, or there may be legacy users from a previous policy. Therefore, the translator should translate between users based on their e-mail addresses.

When translating from the defect tracker to Perforce:

  1. Map the defect tracker user to a Perforce user with the same e-mail address, if there is one.

  2. Otherwise, map the defect tracker user to the Perforce user with the same userid, if there is one.

  3. Otherwise, return the defect tracker userid unchanged (assuming it is valid syntactically as a Perforce userid; if it isn't, you should apply the keyword translator (see section 7.5.2) to it).

When translating from Perforce to the defect tracker:

  1. Map the Perforce user to a defect tracker user with the same e-mail address, if there is one.

  2. Otherwise, map the Perforce user to the defect tracker user with the same userid, if there is one.

  3. Otherwise, if translating the user in a changelist or fix, map the Perforce user to some dummy defect tracker user (see section 4.8). You can tell that you're translating a changelist or fix rather than an issue because the issue0 argument to the translate_1_to_0() method is None.

  4. Otherwise, you're translating a user field in an issue and you can't find a match either by e-mail address or by name. Raise an error.

Each subclass of translator.user_translator must define the following method:

unmatched_users(self)

This method should examine all the users in the defect tracker and Perforce and return a report on the users in each system that have no corresponding userid in the other.

It must return a tuple of four elements:

  1. A dictionary of users in the defect tracker that have no corresponding userid in Perforce. The keys of the dictionary are strings naming the defect tracker userids; the values of the dictionary are the e-mail addresses of the defect tracker users.

  2. A dictionary of users in Perforce that have no corresponding userid in the defect tracker. The keys of the dictionary are the Perforce userids; the values of the dictionary are the e-mail addresses of the Perforce users.

  3. A comment (a string) about the users in the first dictionary explaining how they will be treated by this user translator.

    Example. The TeamTrack integration says, "These TeamTrack users will appear as themselves in Perforce even though there is no such Perforce user."

  4. A comment (a string) about the users in the second dictionary explaining how they will be treated by this user translator.

    Example. The TeamTrack integration says, "These Perforce users will appear in TeamTrack as the user (None). It will not be possible to assign issues to these users."

This method is called each time the replicator is started. The results are used to compose an e-mail to the P4DTI administrator reporting on unmatched users.

8. Configuration

This section describes how to configure the P4DTI to work with your extension. To understand how the configuration works, see [GDR 2000-09-13, 5].

8.1. The configuration generator

You must write a configuration generator for your defect tracker. This must be a module called config_defect_tracker.py, where defect_tracker is the name you chose for your defect tracker (see section 3), converted to lower case.

It must provide the following function:

configuration(config)

The config argument is a module whose members are the configuration parameters specified by the P4DTI administrator in config.py.

It must check all the user configuration parameters that are specific to your defect tracker.

It must return a tuple of two elements:

  1. A Perforce jobspec (see section 8.4), or None if the Perforce jobspec should not be changed.

  2. A revised configuration module, which should include all the configuration parameters in the config argument, plus the configuration parameters required by your defect tracker module, by the replicator (see section 8.3), and by the Perforce interface (see section 8.2).

8.2. Perforce interface configuration

The revised configuration module must include the following parameter for the Perforce interface. (This is in addition to the parameters p4_client_executable, p4_password, p4_port, and p4_user which came from the user configuration.)

logger

This is a logger object (see section 5.3) to which log messages will be written. It must log to log_file if that is specified, to standard output, and to any appropriate system logging facility. It must respect the log_level.

8.3. Replicator configuration

The revised configuration module must include the following parameters for the replicator. (These are in addition to the administrator_address, p4_user, poll_period, replicate_p, replicator_address, rid, and smtp_server parameters which came from the user configuration.)

date_translator

A date translator instance (see section 7.5.1).

field_map

A description of how fields map from the defect tracker to Perforce and back again. It is a list of tuples, one for each field to be replicated. Each tuple has three elements:

  1. The name of the field in the defect tracker.

  2. The name of the field in Perforce.

  3. A translator instance (see section 7.5) that can be used to translate between values in the two fields.

The field map must match the defect tracker database and the Perforce jobspec (see section 8.4).

job_owner_field

The name of the field in the Perforce jobspec which contains the owner of the job.

Example. In the TeamTrack integration, this is "Owner". In the Bugzilla integration, this is "Assigned_To".

job_status_field

The name of the field in the Perforce jobspec which contains the status of the job.

Example. In the TeamTrack integration, this is "State". In the Bugzilla integration, this is "Status".

logger

This must be the same as the logger object for the Perforce interface.

text_translator

A text translator instance (see section 7.5.3).

user_translator

A user translator instance (see section 7.5.4).

8.4. Perforce jobspecs

The configuration generator must build a Perforce jobspec that matches the field_map configuration parameter that it generates.

The jobspec must be a Perforce dictionary suitable for passing to p4 -G jobspec -i. Such a jobspec has keys called "Fieldsn", "Valuesn" and "Presetsn". The numbers in the keys are arbitrary, but they must correspond (so the presets for "Field12" must be in "Presets12").

Perforce requires that five fields be present:

  1. The jobname ("101 Job word 32 required"). This must not be replicated.

  2. The status of the job (field 102). This should be replicated from the defect tracker. It should be a "select" field. Its values should include all the statuses for issues in the defect tracker, converted using the state translator (see section 7.5.2).

  3. The owner of the job (field 103). This should be replicated from the defect tracker. It should be a "word" field.

  4. The date the job was last modified ("104 Date date 20 always"). This must not be replicated.

  5. The title (field 104). This should be replicated from the defect tracker. It should be a "line" or "text" field.

Example. In the TeamTrack integration, Perforce's five required fields appear in the jobspec like this:

'Fields0':  '101 Job word 32 required,
'Fields1':  '102 State select 32 required',
'Presets1': '_new',
'Values1':  '_new/assigned/closed/verified/deferred',
'Fields2':  '103 Owner word 32 required',
'Presets2': '$user',
'Fields3':  '104 Date date 20 always',
'Presets3': '$now,
'Fields4':  '105 Title line 80 required,
'Presets4': '$blank,

The replicator requires that four fields be present [GDR 2000-09-13, 4]. These should appear in the jobspec as follows:

'Fieldsn':    '191 P4DTI-filespecs text 0 optional',
'Fieldsn+1':  '192 P4DTI-rid word 32 required',
'Presetsn+1': 'None',
'Fieldsn+2':  '193 P4DTI-issue-id word 32 required',
'Presetsn+2': 'None',
'Fieldsn+3':  '194 P4DTI-user word 32 always',
'Presetsn+3': '$user',

These fields have high numbers so that they appear at the bottom of the jobspec where people don't have to look at them.

The remainder of the jobspec should be filled in with the fields that the P4DTI administrator has specified for replication in the replicated_fields configuration parameter. Make sure that the values for "select" fields are legal in Perforce (see section 7.5.2).

Finally, the jobspec should have a "Comments" key. The value for this is a string, each line starting with "# ". You should include comments for as many fields as you can, taking information from the defect tracker's database if that's possible.

Example. The comment for the jobspec in the TeamTrack integration may look like this:

# DO NOT CREATE NEW JOBS IN PERFORCE. USE TEAMTRACK.
#
# Job: The job name.
# State: The state of the job in the TeamTrack workflow.
# Owner: The person responsible for taking action.
# Date: The date this job was last modified.
# Title: A short description of the problem from the user's point of view.
# P4DTI-rid: P4DTI replicator identifier. Do not edit!
# P4DTI-issue-id: TeamTrack issue database identifier. Do not edit!
# P4DTI-user: Last user to edit this job. You can't edit this!

Here, the comments for the State, Owner and Title fields have been taken from the field descriptions in the TeamTrack database. The other comments have been supplied by the TeamTrack configuration generator.

For more information about jobspecs, see Chapter 5, "Customizing Perforce: Job Specifications", in the Perforce user's guide [Perforce 2000-12-22b, 5].

8.5. Adapting the configuration module

If your defect tracker module (see section 7) requires the P4DTI administrator to specify any configuration parameters (such as the hostname on which the defect tracker runs, or the user to connect to the database as), then you must adapt the config.py module, as follows.

  1. Add a new # dt_name = "Defect_Tracker" line near the start of section 2, to indicate that your defect tracker integration is available.

  2. Add a new subsection to section 3, starting elif dt_name == "Defect_Tracker":. This should contain default values for the configuration parameters required only by your integration.

  3. Add a history entry to Appendix B explaining what you've done.

8.6. Making your own configurations

Warning: The configuration methods in this section are not supported by Perforce or TeamShare.

This section describes techniques you can use if you want to adapt a supported integration to do something that's not supported. Here are some of the things that are possible by making your own configuration.

  1. Connecting a defect tracker to multiple Perforce servers.
  2. Working with an existing Perforce jobs database and an existing Perforce jobspec (see section 8.6.3).
  3. Fine control over which defect tracker issues are replicated to Perforce.
  4. Fine control over which issue fields are replicated, and their names (see section 8.6.2).
  5. Arbitrary translation of issue data between the defect tracker and Perforce (see section 8.6.3).
  6. Queries based on combined defect tracker and Perforce data.
  7. Working with locally customized versions of Bugzilla.
  8. Redirection or fine control of logging.
  9. Arbitrary mappings between Perforce and defect tracker user names.
  10. Alternative policies for handling conflicts between defect tracker and Perforce changes to issues.

8.6.1. Steps to making your own configuration

Here's are the steps you need to follow to make your own configuration:

  1. Choose a name for your configuration: my_configuration, say.

  2. Edit config.py, adding the line

    configure_name = 'my_configuration'

  3. Make a new module configure_my_configuration.py.

  4. Make your new module into a configuration generator (see section 8.1). See below for some examples.

The best approach to making a configuration generator is to use an existing one and modify its output. That way, you benefit from improvements and corrections to the configuration generator in future releases of the P4DTI.

8.6.2. Example: change the field names in Perforce

Suppose that you are using the TeamTrack integration, but you have tools that work on Perforce jobs that have assumptions about the names of fields in the jobspec. You want the fields in the Perforce jobspec to be called Status, User and Description, not State, Owner and Title. If the Description field is replicated from TeamTrack, you want it to be called Long_Description in Perforce.

In this case, your configuration generator should look like this:

import configure_teamtrack
import re
import string

convert = {
    'State': 'Status',
    'Owner': 'User',
    'Title': 'Description',
    'Description': 'Long_Description',
    }

def configuration(config):
    jobspec, revised_config = configure_teamtrack.configuration(config)
   
    # Convert field names in the jobspec.
    for k,v in jobspec.items():
        match = re.match("^([0-9]+) (.*) ([a-z]+) ([0-9]+) ([a-z]+)$", v)
        if match and convert.has_key(match.groups(2)):
            groups = match.groups()
            groups[1] = convert[groups[1]]
            jobspec[k] = string.join(groups, ' ')
   
    # Convert field names in the field_map.
    for field_map in revised_config.field_map:
        if convert.has_key(field_map[0]):
            field_map[0] = convert[field_map[0]]
   
    return jobspec, revised_config

8.6.3. Example: use existing Perforce jobspec

Suppose that you want to use the TeamTrack integration, but you have many existing tools and documents that refer to your current Perforce jobspec, so don't want the P4DTI to change your Perforce jobspec.

First, you must add the fields required by the P4DTI [GDR 2000-09-13, 4] to your Perforce jobspec by hand. Then your configuration generator should look like this:

import configure_teamtrack
import message
import translator

product = "My configuration"
error = "My configuration error."
catalog = {
    1: (message.ERR, "Unknown TeamTrack state '%s'."),
    2: (message.ERR, "Unknown Perforce status '%s'."),
}
factory = message.catalog_factory(catalog, product)

# A map from state in TeamTrack to status in Perforce.
state_pairs = [
    (1, 'open'), # Assigned
    (2, 'closed'), # Resolved
    (5, 'suspended'), # Deferred
]

class my_state_translator(translator.translator):
    def translate_0_to_1(self, value, dt0, dt1, issue0 = None, issue1 = None):
        assert isinstance(value, types.IntType)
        for (t,p) in state_pairs:
            if value == t:
                return p
        raise error, factory.new(1, value)

    def translate_1_to_0(self, value, dt0, dt1, issue0 = None, issue1 = None):
        assert isinstance(value, types.StringType)
        for (t,p) in state_pairs:
            if value == p:
                return t
        raise error, factory.new(2, value)

def configuration(config):
    _, revised_config = configure_teamtrack.configuration(config)

    # Make a field_map that works with my existing jobspec.
    revised_config.field_map = [
        ('STATE', 'Status', my_state_translator()),
        ('OWNER', 'User', revised_config.user_translator),
        ('TITLE', 'Description', translator.translator()),
        ('DESCRIPTION', 'User_Impact', revised_config.text_translator),
    ]

    # Return None as the jobspec so the Perforce jobspec won't be changed.
    return None, revised_config

Note the use of coding conventions in this example: message catalogs (see section 5.2) and raising exceptions when a value can't be translated (see section 5.4).

Warning. If you leave your Perforce jobspec unchanged, you must check that it is compatible with the P4DTI. The reason for this is that the replicator uses p4 -G job -o jobname to get a job from Perforce; this command applies more stringent checking than p4 job -o jobname. In particular, you must:

  1. Check that the "Presets" for each select field is valid for that field (that is, it appears as one of the "Values" for that field).

    Some organizations set up a jobspec with a field like this:

    Fields: 120 Severity select 20 required
    Values: Severity critical/essential/optional
    Presets: Severity setme

    Their intention is that since "setme" is not a legal value for the Severity field, the person submitting the job must give it a value; they can't just ignore it and leave it with the default value.

    However, this won't work with the P4DTI, because the command p4 -G job -o won't even give you a blank job form; instead it gives you an error message.

9. Testing

See the test/ directory of the integration kit for test procedures, sample data and automated test cases that can be used to test the integration.

You should adapt the existing tests so that they test your integration.

[ This section needs a lot more work. Sources of information: the index to the test directory; the test_p4dti.py test script; other test cases in that directory; our testing procedures; the Python unit testing framework [Purcell 2001-02-12]; the closed P4DTI issues (these are suitable for regression testing). GDR 2001-03-23]

10. Providing a defect tracker interface to Perforce relations

The defect tracker should display, for each issue that is replicated, a description of the Perforce server to which the issue is replicated. Use the configuration parameter p4_server_description which you should have stored in a table in the defect tracker (see section 4.5).

The defect tracker should display on each issue description page a table of fixes for that issue (if there are any). The table should look like the table below.

Change Effect Date User Description
5493 open 2000-12-05 GDR Added replicator method mail_concerning_job() for e-mailing people about a job.
5524 open 2000-12-06 GDR Fixed the replicator's user_email_address method so that it really returns None when there is no such user.
5541 open 2000-12-06 GDR If the owner of a job and the person who last changed it are the same, include them only once in any e-mail sent by the replicator about that job.
5634 (pending) closed 2000-12-07 GDR Merging back to master sources.

Points to note about this table:

  1. Pending changelists are distinguished from submitted changelists. This is important because the effect of a pending changelist does not happen until the changelist is submitted. So in the above table the status of the job is still "open" but it is understood that when changelist 5634 is submitted it will become "closed".

  2. The user and date are for the change (not for the fix). Knowing when the change was made and by whom is much more important than knowing when the change was linked with the job.

  3. The user is the defect tracker user who corresponds to the Perforce user who made the change.

  4. The change number is a link to the URL given by the changelist_url configuration parameter, with the change number inserted. This configuration parameter is defined in the Administrator's Guide as being suitable for passing to sprintf() as the format string: it must have one %d format specified (for which the change number will be substituted) and it may have any number of doubled percent signs %% (which must become single percent signs in the resulting URL) [RB 2000-08-10a, 5.1].

  5. All the fixes for an issue will be being replicating by the same replicator and from the same Perforce server as the issue itself. So when building this table you only need to select records with the same replicator identifier and Perforce server identifier as the issue.

11. Adapting the manuals

When adding material relating to your defect tracker to the manuals, surround each section with the HTML tags <div class="defect_tracker"> and </div>. This makes the material for a particular defect tracker easy to find, extract and check, to meet requirement 32.

You must adapt the Perforce Defect Tracking Integration Administrator's Guide [RB 2000-08-10a] to describe your integration, as described in the list below.

  1. Add a new subsection to section 3, specifying the software and procedural prerequisites for using your defect tracker with the P4DTI.

  2. If your integration requires a new installation procedure, or installs on a new platform, update section 4.

  3. Add your new configuration parameters to section 5.1.

  4. Add a new subsection to section 5, explaining how to configure your defect tracker for the P4DTI.

  5. Add a new item to the list in section 10, explaining how to uninstall your integration and return your defect tracker to its original state.

  6. Add the error messages that your code can produce to section 11.2 (include the product, message identifier and check digit just like the other errors messages in that section).

  7. Add likely error messages from systems with which your code interacts to section 11.3 (for example, errors from the defect tracker, or from your database interface).

  8. Add references to documentation for your defect tracker, and any other supporting materials that you referred to, to appendix A.

  9. Add a history entry to appendix B explaining what you've done.

If you provided an interface from your defect tracker to the Perforce fixes relation (see section 10), then you must adapt the Perforce Defect Tracking Integration User's Guide [RB 2000-08-10b] to describe your integration, as follows:

  1. Add a paragraph to section 10.3 explaining how to access Perforce fixes from the defect tracker.

12. Making your work available to the community

12.1. Reporting defects

Defects in the P4DTI Kit include (but aren't limited to):

  1. An essential piece of information can't be found in this manual or in the design documents it refers to.

  2. Inconsistencies between this manual, the design documents it refers to, and the sources they document.

  3. Defects in the P4DTI sources or in the test cases.

Please report any defects you find to Peforce support, so that they can fixed and the product improved.

Please provide the following information with your defect report:

  1. The release of the P4DTI Kit you are using (look in the readme.txt that came with the P4DTI Kit to identify the release).

  2. The name and release of the defect tracker you are integrating with.

  3. If you're reporting a defect in documentation:

    1. What you're trying to do.

    2. The information you need.

    3. Where you expected to find it.

    4. Where else you looked for it.

  4. If you're reporting a defect in the code:

    1. What you did immediately prior to the defect's occurrence.

    2. What you think should have happened.

    3. What actually happened.

    4. The Perforce release you are using.

    5. Any source code you've added or modified, including your config.py file.

    6. A section of the P4DTI log that includes the error that you're reporting and some context around that error.

    7. Copies of any related e-mail messages generated by the P4DTI.

12.2. Making a contribution

Please send your contributions (fixes, adaptions and extensions) to Perforce support [Support]. Please include the following:

  1. A description of your contribution: what it is designed to achieve; which files you've changed; which files you've added.

  2. The release of the P4DTI Kit you have been developing against (look in the readme.txt that came with the P4DTI Kit to identify the release).

  3. The complete P4DTI Kit, including your modifications and additions. Make a tarball or a ZIP archive of the whole P4DTI Kit directory. (Please do this even if you've only changed a couple of files. This allows us to add your contribution to Perforce and use p4 diff2 to see exactly what changes you've made.)

  4. What you are prepared for us to do with your contribution. Are you willing for us to make it available for distribution from Perforce or Ravenbrook's web site? Are you willing for us to incorporate it into the P4DTI and maintain and support it? Have you made it available under an open source license?

A. References

[GDR 2000-05-03] "Requirements and Use Cases for Perforce/Defect Tracking Integration"; Gareth Rees; Ravenbrook Limited; 2000-05-03.
[GDR 2000-05-24] "Perforce Defect Tracking Integration Project Requirements"; Gareth Rees; Ravenbrook Limited; 2000-05-24.
[GDR 2000-05-30] "Analysis of architectures for defect tracking integration"; Gareth Rees; Ravenbrook Limited; 2000-05-30.
[GDR 2000-08-08] "Python interface to TeamTrack: design"; Gareth Rees; Ravenbrook Limited; 2000-08-08.
[GDR 2000-09-04] "TeamTrack database schema extensions for integration with Perforce"; Gareth Rees; Ravenbrook Limited; 2000-09-04.
[GDR 2000-09-13] "Replicator design"; Gareth Rees; Ravenbrook; 2000-09-13.
[Lutz 1996] "Programming Python"; Mark Lutz; O'Reilly; 1996-10; ISBN 1-56592-197-6.
[NB 2000-11-14a] "Bugzilla database schema"; Nick Barnes; Ravenbrook Limited; 2000-11-14.
[NB 2000-11-14b] "Bugzilla database schema extensions for integration with Perforce"; Nicholas Barnes; Ravenbrook Limited; 2000-11-14.
[NB 2000-11-14c] "Python interface to Bugzilla: design"; Nicholas Barnes; Ravenbrook Limited; 2000-11-14.
[Perforce 2000-12-22a] "Perforce 2000.2 P4 Command Line User's Guide"; Perforce Software; 2000-12-22; <http://www.perforce.com/ perforce/doc.002/ manuals/p4guide/>, <ftp://ftp.perforce.com/ /pub/perforce/r00.2/doc/ manuals/p4guide/p4guide.pdf>.
[Perforce 2000-12-22b] "Perforce 2000.2 System Administrator's Guide"; Perforce Software; 2000-12-22; <http://www.perforce.com/ perforce/doc.002/ manuals/p4sag/>, <ftp://ftp.perforce.com/ /pub/perforce/ r00.2/doc/ manuals/p4sag/p4sag.pdf>.
[Purcell 2001-02-12] "PyUnit — a unit testing framework for Python"; Steve Purcell; 2001-02-12.
[RB 2000-08-10a] "Perforce Defect Tracking Integration Administrator's Guide"; Richard Brooksby; Ravenbrook Limited; 2000-08-10.
[RB 2000-08-10b] "Perforce Defect Tracking Integration User's Guide"; Richard Brooksby; Ravenbrook Limited; 2000-08-10.
[RB 2000-08-10c] "Perforce Defect Tracking Integration Architecture"; Richard Brooksby; Ravenbrook Limited; 2000-08-10.
[RB 2001-03-07] "P4DTI Project Contributions Procedure"; Richard Brooksby; Ravenbrook Limited; 2001-03-07.
[van Rossum 2000-10-16] "Python Tutorial"; Guido van Rossum; 2000-10-16.

B. Document History

2000-10-16 RB Created placeholder after meeting with LMB.
2000-12-10 GDR Drafted sections 3 and 4.
2000-12-11 GDR Drafted sections 2, 5, and 8 and outlined sections 6 and 7.
2000-12-31 GDR The table of fixes in section 8 now distinguishes pending from submitted changes.
2001-01-02 GDR Added section 7.1 (configuration architecture), figure 1, section 7.5 (customized configuration). Moved text from appendix D of the Administrator's Guide to section 7.5.
2001-02-04 GDR Updated definition of defect_tracker.all_issues method.
2001-02-23 GDR Added corresponding_id method; revised definition of readable_name method.
2001-03-02 RB Transferred copyright to Perforce under their license.
2001-03-13 GDR Deleted the recording of conflicts and the need for manual conflict resolution. Conflict resolution is always immediate.
2001-03-20 GDR Added overviews of requirements, architecture and design. Included replicator block diagram. Improved links.
2001-03-21 GDR Wrote specifications of the defect tracker and translator classes. Described messages, catalogs, logging and errors. Moved logging and errors to new section 5 because it's important; renumbered remaining sections; moved configuration adaption to new section 8.6.
2001-03-22 GDR Specified the configuration generator; explained how to make your own configuration (with example code); explained how to adapt the manuals. Gave warning about incompatible jobspecs. Added prerequisites and where to get help. Added section on testing. Added section on code layout. Explained how to report defects and submit contributions.
2001-03-29 RB Changed internal and external cross-references to conform with our other documents. Tidied up references to requirements. Removed claims about assigning copyright. Removed section promising to do particular things with contributions. Updated references to Perforce manuals to (latest) release 2000.2. Validated and fixed some broken links. Sorted references section.

This document is copyright © 2001 Perforce Software, Inc. All rights reserved.

Redistribution and use of this document in any form, with or without modification, is permitted provided that redistributions of this document retain the above copyright notice, this condition and the following disclaimer.

This document is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the copyright holders and contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this document, even if advised of the possibility of such damage.

$Id: //info.ravenbrook.com/project/p4dti/release/1.1.1/ig/index.html#1 $