Perforce Defect Tracking Integration Project
This is the Perforce Defect Tracking Integration version 0.4 Administrator's Guide. It explains how to installation, configure, maintain, and administer the Perforce Defect Tracking Integration (P4DTI).
Warning: The Perforce Defect Tracking Integration version 0.4 is a is beta test software, only intended for use during the beta program of the project. The software will be defective. We recommend that you do not rely on the integration in your organization. The integration may destroy your data. See the project web site <http://www.ravenbrook.com/project/p4dti/> for information about planned releases. We are very interested in your feedback on the beta. Please write to p4dti-beta-feedback@ravenbrook.com.
This is outline documentation, still in development.
This document is intended for the P4DTI administrator at your organization. Ordinary users of the defect tracker or Perforce should not need to read it. See "Training and Documentation" in section 8.
Although this guide will teach you how to administer the integration, it won't teach you the basics of actually using the integration, Perforce, or the defect tracker. You should also read the Perforce Defect Tracking Integration User's Guide in order to understand the integration from a user's perspective.
All of our documentation is available from our web site at http://www.ravenbrook.com/project/p4dti/master/manual/.
This section provides a summary of the process of installing, configuring, and running the integration, and also describes what the integration does and how it does it.
[Section not written yet. RB 2000-11-27]
The integration mainly works by replication. The replicator is a process which copies (replicates) data between a defect tracker and a Perforce server in order to keep each one up to date with changes made in the other. This allows developers to do their routine defect resolution work entirely from their Perforce client, without needing to use the defect tracker's interface. It also allows developers to relate their changes to defect tracking issues.
Figure 1 shows how the replicator communicates with the defect tracking server and the Perforce server.
The replicator maintains a one-to-one relationship between issues in the defect tracker's database, and jobs in the Perforce repository. In other words, each issue has a corresponding job, and vice versa. (Actually, you can configure the replicator to only handle a subset of the issues. See "Configuring the software" in section 5.) The replicator keeps the contents of certain fields in the defect tracker's issues the same as the contents of the corresponding Perforce job, so that editing one edits the other.
The replicator also copies Perforce's links between jobs and changelists to the defect tracker's database, and makes them visible in the defect tracker's user interface. This makes it possible to track, record, and check a whole bunch of things. In particular, it makes it possible to track and record the changes made for each issue, and find out why a change was made in terms of issues.
[We could do with a diagram of the relationship between issues, jobs, and changelists here. RB 2000-11-27]
Most defect trackers have an idea of workflow — a set of rules which control who can do what to which issues. The replicator enforces the defect tracker's workflow. The replicator rejects changes to jobs in Perforce which would be illegal in the defect tracker. It undoes the change and sending an e-mail message to the user.
The replicator polls the defect tracking server and the Perforce server at regular intervals to get a list of recent changes, and attempts to propogate these changes to the other system. If it can't, and can't figure out what to do, it sends an e-mail message to the resolver (see "Configuring the software" in section 5). This could happen if both a defect tracker issue and the corresponding job have been changed at the same time, for example.
We recommend running the replicator on the same machine as the defect tracker's server, partly to keep all defect tracking related administration local to one machine, and partly because Perforce's network protocol is usually better than the defect tracker's.
Figure 1. The replication architecture
This section explains what you will need before you can install the integration in your organization. It is divided into subsections that explain the Perforce prerequisites and the defect tracker prerequisites, respectively. You need to meet both sets of prerequisites before you install the integration.
We recommend that administrators of the integration have at least six months of Perforce administration experience and a basic knowledge of Python. If you are unfamiliar with Python, you may want to read the Python tutorial at < http://www.python.org/doc/current/tut/tut.html> — it's short. You will certainly need a good working knowledge of Perforce's command line interface, and should have read both the Perforce Command Line User's Guide [Perforce 2000-10-09] and the Perforce System Administrator's Guide [Perforce 2000-10-11].
A copy of P4DTI product release version 0.4. Releases are available from the release page at <http://www.ravenbrook.com/project/p4dti/release/>
You will need to be running a Perforce server of version 2000.2 or later (but see below). Server upgrades can be downloaded from the Perforce FTP server at <ftp://ftp.perforce.com/pub/perforce/>. Be sure to read the release notes (available from <http://www.perforce.com/>) before you install, and contact Perforce technical support if you need help.
As of 2000-11-23, Perforce 2000.2 is not available, so a special pre-release Perforce client, P4/NTX86/2000.1/17595 (2000/09/28), must be used by the replicator. This client can be downloaded from <http://www.ravenbrook.com/project/p4dti/import/2000-09-28/p4-client-ntx86/p4gdr.exe>. You will need to specify the location of this client executable when you configure the replicator (for details, see Section 5). [What is the replicator? Presumably it will have been explained before now, but if not, we need to explain it somewhere. LMB 2000-11-25]
You will also need Perforce licenses for every defect tracking user who is going to work in Perforce, plus an extra "daemon license" for the replicator. A daemon license is a license for an automatic process, rather than a person, and Perforce's policy is to provide these free of charge. Contact Perforce technical support to get one.
We recommend that you back up your Perforce repository before you install the P4DTI, to make sure you don't lose any data. See the Perforce System Administrator's Guide [Perforce 2000-10-11, Chapter 2].
You might want to practice installing and configuring the P4DTI using a test Perforce repository before you try it on your real one. A copy of your real repository would be ideal. See the Perforce System Administrator's Guide [Perforce 2000-10-11, Chapter 1].
The address and port number of your Perforce server will need to be entered into the replicator configuration; for details, see Section 5.
You can skip this section if you're not using the P4DTI with TeamTrack.
You will need to be running TeamTrack version 4.0.4 or later (but see below).
As of 2000-10-15, TeamTrack 4.0.4 is not available, so a special pre-release build of TeamTrack, build 4402, must be used. An installer can be downloaded from <http://www.ravenbrook.com/project/p4dti/import/2000-10-13/teamtrack-4402/ttrk4402.exe>.
We recommend that you back up your TeamTrack database before you install the P4DTI, so that you don't lose any data. See the section named "Copying a Database" in "tTrack 4.0 Administrator Manual" [TeamShare 2000-05, page 67]. (You might want to practice installing and configuring the P4DTI using a test TeamTrack database before you try it on your real one. We've included a sample TeamTrack database with the integration; see Section 4.4 for information on how to use it.)
You will need TeamTrack licenses for every Perforce user who is going to work in TeamTrack, plus one extra for the P4DTI itself. To clarify, every Perforce user who will be assigned issues must also have a TeamTrack license, because only licensed TeamTrack users can own issues. [People probably need to go to TeamShare and ask for a special extra licence for the P4DTI. TeamShare need to make arrangements to cope with this. I'm waiting for a decision. RB 2000-11-23]
You will also need Administrator level access to the TeamTrack server machine, and approximately 5Mb of free disk space for the integration, plus space for logs.
The TeamTrack server machine will also need Python 1.5.2 for Windows, and the Windows extensions. These are available from <http://www.ravenbrook.com/project/p4dti/import/1999-04-13/python-1.5.2/py152.exe> and <http://www.ravenbrook.com/project/p4dti/import/2000-05-06/python-win32all-132/win32all-132.exe> respectively.
You can skip this section if you're not using the P4DTI with Bugzilla.
[Section not written yet. RB 2000-11-23]
[This section and the next need a lot of organizing. Much of what's in this section at the moment is really configuration, or else depends on configuration decisions. GDR 2000-10-19]
This section explains how to install the integration software. It is divided up into the following subsections:
Work through the subsections in the order in which they appear in this manual. [Can they stop halfway through and finish installing the integration later? Need to explain what they can and can't do while installing. LMB 2000-11-25]
Before you go any further, make sure you have already met all the prerequisites for both Perforce and the defect tracker. In particular, you must have installed all the prerequisite software. See Section 3 for a complete list of the prerequisites.
Before you install the integration, you need to make the following configuration decisions:
C:\Program Files\P4DTI\. You will use this information when you unpack the integration software in Section 4.2.The integration software is distributed as a self-extracting executable named p4dti-teamtrack-RELEASE.exe (where RELEASE is the release number, such as 0.4.1). Run this executable on the TeamTrack server machine. The installer unpacks the integration into C:\Program Files\P4DTI\ by default, but you can ask the installer to put it somewhere else.
Before you can use the integration, you need to create two new TeamTrack values in the Windows Registry. You can do this as follows:
regedt32.HKEY_LOCAL_MACHINE\Software\TeamShare\TeamTrack. VC Integration, data type REG_SZ, and string contents perforce.VC Update Group, data type REG_SZ, and string contents Everyone.See Figure 1 for an example of what your Registry should now look like.
[Do these users need a warning about how dangerous it is to edit the Registry? They probably know what they're doing, but... LMB 2000-11-9]
Figure 2. TeamTrack Registry keys
We recommend that you use a sample TeamTrack database when testing the integration. A sample database is distributed with the integration. To use this database, follow the instructions below. (See also the section named "Connecting to databases" [TeamShare 2000-05, page 58].)
Microsoft Access Driver (*.mdb) from the list.C:\Program Files\P4DTI\tTrackSample.mdb. Your dialog should now look like the one in Figure 2. P4DTI test from the list (this is the
data source that you just created). Your dialog should now look like the one in Figure 3. Stop and start the TeamTrack server by following these steps:
Figure 3. Path to data source in the TeamTrack Administrator
Figure 4. Selected data source in the TeamTrack Administrator
Next, you need to create a user called [Huh? The "called" is confusing. LMB 2000-11-25] in TeamTrack to represent the replicator. The replicator expects this user to have the user name "P4DTI-RID" (where RID is the replicator identifier). You can choose any user name you like for the replicator, but you must then configure the replicator to use the name you have chosen; for instructions on how to do this, see Section 5.
To create the user, follow these steps:
Finally, you need to update the Perforce jobspec to add the fields required by the integration. The fields you need to add are described in this section. See also Chapter 5, "Customizing Perforce: Job Specifications", in the Perforce user's guide [Perforce 2000-10-11, Chapter 5].
p4 -p 127.0.0.1:1667 jobspec (use the address and port for the Perforce server you're using for the integration). [Need a reference to Perforce UG here. LMB 2000-11-25]Add the following fields to the Fields section of the Perforce jobspec. It's not essential that the field numbers be as shown, but we recommend that you keep them the same if possible.
110 P4DTI-filespecs text 0 default
111 P4DTI-rid word 32 required
112 P4DTI-issue-id word 0 required
113 P4DTI-user word 32 always
114 P4DTI-action select 32 required
Add the following entries at the bottom of the Perforce jobspec:
Preset-P4DTI-rid: None
Preset-P4DTI-issue-id: None
Preset-P4DTI-user: $user
Values-P4DTI-action: wait/keep/discard/replicate
Preset-P4DTI-action: replicate
Change the Values-Status entry so that states can be replicated from the defect tracker. If you are using the TeamTrack sample database and a new Perforce repository, it should look like this:
Values-Status: open/closed/assigned/deferred/verified
For a real installation, you'll need to decide how states in the defect tracker map to and from statuses in Perforce, and configure the replicator accordingly. This is described in detail in Section 5.
This section describes the configuration decisions that each organization has to make and how to configure the integration to implement those decisions. It also contains example configurations and describes common pitfalls.
Configuration is not a simple task. Each organization must make its own configuration decisions based on its process and workflow. Therefore, you should read all of Section 5.1, "Configuration Decisions", before you start to configure the software.
The integration's configuration will also need to be updated when the organization changes in various ways. See "Maintaining the Configuration" [no section yet] for details.
By default, the integration is configured to work with the defect tracker's default database. In the case of TeamTrack, the integration is configured for the sample database, installed by default in C:\Program Files\TeamShare\TeamTrack\database\tTrackSample.mdb.
The decisions that you (or someone else in your organization) need to make are:
[and lots more. (Moved from final list item. LMB 2000-11-25)]
[Section not written yet. RB 2000-09-20]
The integration is configured by editing definitions in Python. The configuration for a release is defined in the file config_DEFECT_TRACKER.py in the installation directory, which is C:\Program Files\P4DTI by default. (For example, the configuration for the TeamTrack release is in the file config_teamtrack.py.) This file builds two objects, the dt object and the r object, which are then used by the Python programs that run the replicator and check the consistency of the database. The dt object represents the interface to the defect tracker, and the r object represents the replicator itself. Here's an example:
dt = dt_teamtrack.dt_teamtrack(rid, sid, teamtrack_config)
r = replicator.replicator(rid, dt, replicator_config)
rid is the replicator identifier, sid is the Perforce server identifier, and teamtrack_config and replicator_config are Python dictionaries mapping name to value for each configuration parameter. The next section lists the configuration parameters.
[Perhaps there should be a "quick start" section here telling you the minimum you need to do to get going. This would say something like, "go into config_teamtrack.py and provide values wherever it says ????". GDR 2000-10-19]
[How does the administrator check the configuration? RB 2000-09-21]
[There should follow sections on how to implement each of the configuration decisions. RB 2000-09-21]
[We must explain how to replicate just the cases belonging to a single project [RB 2000-11-20, section 8 item 9] RB 2000-11-20]
Replicator configuration parameters:
| Name | Meaning | Default |
|---|---|---|
| administrator-address | The e-mail address of the administrator of the integration (a string). The replicator sends reports of conflicts in the data to this address. If this is None, the replicator sends no e-mail. | None |
| p4-client-executable | The path to the Perforce client executable which the replicator uses. The replicator requires this client to be version P4/NTX86/2000.1/17595 (2000/09/28) or later. | "p4" |
| p4-password | The password with which the replicator logs into Perforce. | "" |
| p4-port | The address and port of the Perforce server which the replicator replicates to and from. | "127.0.0.1:1666" |
| p4-user | The userid with which the replicator logs into Perforce. | "P4DTI-RID" where RID is the replicator identifier. |
| poll-period | The number of seconds that the replicator pauses between polling for changes. | 10 |
| replicator-address | The e-mail address which the replicator uses when sending e-mail. | "p4dti" |
| smtp-server | The address of the SMTP server that the replicator uses when sending e-mail. If this is None, then the replicator sends no e-mail. | None |
| logger | A logger object that the replicator will use to write log messages. [Needs a reference to documentation for loggers. GDR 2000-10-19] | logger.file_logger() |
TeamTrack configuration parameters:
| Name | Meaning | Default |
|---|---|---|
| server | The TeamTrack address and port which the replicator will replicate to. | The local host. |
| user | The user name which the replicator uses to log into TeamTrack. | "P4DTI-RID", where RID is the replicator identifier. |
| password | The password which the replicator uses to log into TeamTrack. | "" |
| p4-server-description | A human-readable description of the Perforce server that the replicator replicates to. | "Perforce server" |
| replicated-fields | A list of fields that will be replicated between issues and jobs. Each entry is a 3-tuple (TeamTrack field name, Perforce field name, field type). Field types can be "user", "text" or "state". [Needs more detail. GDR 2000-10-19] | See "dt_teamtrack.py". |
| state-dt-to-p4 | A map from TeamTrack state name to Perforce status name. [Needs more detail. GDR 2000-10-19] | See "dt_teamtrack.py". |
| state-p4-to-dt | A map from Perforce status name to TeamTrack state name. [Needs more detail. GDR 2000-10-19] | See "dt_teamtrack.py". |
[We should include a simple example which would suit organization new to defect tracking, or who have just migrated from Perforce jobs, and a more complex example, more suitable for TeamTrack users with non-trivial workflows. RB 2000-09-21]
[This topic will probably deserve special treatment, but if not, delete this section. RB 2000-09-21]
This section explains how to migrate your defect tracking data to the integrated system.
[Section not written yet. This will probably involve added P4DTI fields to the set of jobs that the organization wants migrated. We should explain how to do that. RB 2000-09-21]
The integration won't replicate new jobs in Perforce. So, for the moment, you have to migrate them using the migrate_teamtrack.py program in the installation directory, which is C:\Program Files\P4DTI by default. [Need more details. How and where do you run it? What arguments does it need? etc. LMB 2000-11-25]
[Section not written yet. This might be trivial, but it might involve adding P4DTI fields to the set of TeamTrack cases that the organization wants migrated. We should explain how to do that. RB 2000-09-21]
[Section not written yet. This might be trivial, but it might involve adding P4DTI fields to the set of Bugzilla bugs that the organization wants migrated. We should explain how to do that. RB 2000-09-21]
This section describes how you can test the integration setup to make sure it's working properly.
Note: The replicator does not currently start automatically. You must start it yourself by following these steps:
C:\Program Files\P4DTI\ by default; see above [See above where? Need to provide a reference. LMB 2000-11-25]).python run_teamtrack.py.[What should you expect to happen? GDR 2000-10-19]
To stop the replicator, press Control-C and wait for the replicator to next poll (10 seconds by default).
[A good approach to take in this section would be to create a test issue and take it through a complete life-cycle (i.e. through the workflow). We can't know what the workflow will be at the organization, but we can use an example. RB 2000-10-07]
Run the consistency checker. To do this, follow these steps:
C:\Program Files\P4DTI\ by default; see above [See above where? Need to provide a reference. LMB 2000-11-25]).python check_teamtrack.py.[What should you expect to happen? GDR 2000-10-19]
Examine the database by hand using a database application — for example, Microsoft Access — to make sure it looks like the Perforce data is in there.
This section explains how to test the integration as if you were a user of Perforce. You should go through a typical sequence of actions that you expect your developers to go through — a dry run, as it were.
[It's probably not necessary to test the integration from every Perforce interface, but we should describe how to do it for each of them in the manual. Advise the administrator to use whichever their developers are most likely to use.]
The main interfaces are:
[Section not written yet. RB 2000-09-20]
This section explains how to test the integration as if you were a user of the DT.
[Probably need sub-sections for each DT.]
[Section not written yet. RB 2000-09-20]
[We're going to provide the UG which will be "how to" documentation for the use cases and requirements. But this section will recommend getting the users together and telling them about the integration and how to use it, in overview. LMB suggests we provide a presentation outline. GDR suggests that we cover some key stuff that users should or shouldn't do (e.g. "don't edit the P4DTI-* fields in jobs"). RB will write this presentation, as it's needed for the alpha programme anyway. We need to cover how to use it, and what to do when it goes wrong, for example, when a conflict happens, or when they come across a conflicting issue. This presentation material will also be useful for the user guide. RB 2000-10-07]
[Section not written yet. RB 2000-09-20]
[When the admin is configuring and testing, he doesn't want to use the real database. We should explain this earlier on. And we'll need to explain how to make the integration work on the real data smoothly. We can provide a plan, and contingency plans for if things go wrong. For example, recommend coming in early in the morning or over a weekend. We can estimate how long it will take.
[Configuration should've taken place on a duplicate of their real data. This might not be feasible if the real data is a 10Gb Perforce repository and a 100Gb corporate Oracle database. We'll need to think about that one.
[They'll need to re-run the testing on the live system as well. (The earlier testing was to make sure the configuration worked, this is to make sure the live system is working.)
[They should configure the replicator to start automatically when the system comes up, and check that it does.
[They should then watch the system working for a while. We should list things to look out for. RB 2000-10-07]
[Section not written yet. RB 2000-09-20]
[Section not written yet. RB 2000-09-20]
First tell your staff that you're uninstalling the integration. Ask them to stop using either Perforce jobs or the defect tracking, whichever you're not planning to use in future. Then simply stop the replicator process. Remove any hooks that start it again, such as Windows services, entries in /etc/rc.d, and so on. That's all you need to do.
The rest of this section deals with ways to remove data created by the integration, if you want to do that. We don't recommend it — we recommend that you keep all of your records.
If you intend to use Perforce jobs in future, you might consider removing the P4DTI fields from the Perforce jobspec, so that they don't clutter up future jobs or confuse users. We suggest you don't re-use the field numbers, though, so that you can easily re-install the integration later.
[We should talk about how to delete the replicator installation itself, most likely by removing the contents of a directory. We should talk about removing just one replicator when there are several running. RB 2000-10-15]
Problem: A replication failed and a conflict was reported. I went into Perforce and set the action to "keep". But the same conflict happened again.
Analysis: When you set the action to "keep", the replicator tries to replicate again. If it failed before because the data does not pass the consistency checks, it will most probably fail again now for the same reason.
Solution: Figure out what went wrong, fix the problem in the defect tracker or in Perforce, and only then set the action to "keep" or "discard".
| 2000-08-10 | RB | Created placeholder. |
| 2000-09-11 | GDR | Added instructions for demonstrating the integration and notes on version 0.2. |
| 2000-09-20 | RB | Replaced demo instructions with full documentation outline from documentation plan. |
| 2000-10-15 | RB | Added installation and uninstallation sections, and other sections discussed in [RB 2000-10-07]. Removed parts specific to Ravenbrook Information System. |
| 2000-10-16 | RB | Merged with master sources and GDR's demonstration instructions for version 0.2. More edits required to make this consistent with the master sources. |
| 2000-10-19 | GDR | Updated to fix defects in release 0.3.1 [GDR 2000-10-17a] and release 0.3.2 [RB 2000-10-18b]. |
| 2000-11-25 | LMB | Removed "system" from title. Made lots of minor formatting and transition edits. Moved Glossary to end of document. Reorganized Section 4. |
| 2000-11-26 | RB | Impoved prerequisites section. Added draft Bugzilla prerequisites. Formatted troubleshooting section. Updated version 0.3 references to version 0.4. |
| 2000-11-27 | RB | Added readership. Removed some false statements. |
This section describes the way in which the integration represents the Perforce data in the defect tracking system's database (DTDB), so that organizations can write queries on the combined Perforce and defect tracking data [requirement 68].
Details of the schema extensions for TeamTrack aren't yet in the manual. Full documentation is available in the design document "TeamTrack database schema extensions for integration with Perforce (version 0.4)" [GDR 2000-09-04].
[Section not written yet. RB 2000-09-20]
| DT | Defect Tracker, Defect Tracking System | Examples are TeamShare's TeamTrack, Soffront TRACK Defects, Bugzilla |
| DTDB | Defect Tracking DataBase, Defect Tracking DataBase system | Most defect trackers store the defect tracking information in a database of some sort. Some have abstract interfaces which allow them to use any ODBC compliant database (for example, TeamTrack can use Oracle as its DTDB). Some are closely coupled to a particular database (Bugzilla uses MySQL as its DTDB). |
| P4 | The Perforce SCM Software | Perforce Software's fast configuration management system software. We do not use the term P4 to refer to Perforce Software, the company. |
| P4DTI | Perforce Defect Tracking Integration | The software which integrates Perforce Software's fast configuration management system (P4) to defect tracking systems (DTs). |
| RID | Replicator ID | [Need a definition here. LMB 2000-11-25]. |
Copyright © 2000 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute copies and derivative works of this document provided that (1) you do not charge a fee for this document or for its distribution, and (2) you retain as they appear all copyright and licence notices and document history entries, and (3) you append descriptions of your modifications to the document history.
$Id: //info.ravenbrook.com/project/p4dti/branch/2000-11-14/bugzilla/manual/sag/index.html#3 $