P4DTI Version 2 Requirements Analysis

Contents

• 1. Introduction
• 2. Executive Summary
• 3. Requirements
• 4. Analysis
  • 4.1. Bugzilla
  • 4.2. Easier installation
  • 4.3. Solaris
  • 4.4. Easier administration
  • 4.5. Share the p4 logger
  • 4.6. Perforce as a defect tracker
  • 4.7. Provide management information
  • 4.8. Configurable jobspec
  • 4.9. Additional Integrations
  • 4.10. Improved Recovery from replication errors
  • 4.11. Meaningful Filespec Replication
  • 4.12. Closer Web Integration
• 5. Recommendations
• A. References
• B. Document History

1. Introduction

This document analyses some requirements for P4DTI version 2, and presents some solutions with estimates of effort required by Ravenbrook.

The intended readership of this document is Perforce and Ravenbrook P4DTI staff.

This document is not confidential.

2. Executive Summary

We have identified some requirements for P4DTI 2. These are listed in Section 3, and analysed in Section 4. However, we are not satisfied that this accurately represents the true client requirements. We recommend a survey of P4DTI stakeholders to assess true requirements.

For the P4DTI 2 project itself, we recommend an incremental delivery plan, starting with the most critical and distinctive requirements. Starting in 2003-03, such a plan would deliver an initial version ("2.0"?) in about 2003-07, and subsequent versions about every two months thereafter, for an effort of around 100 hours per month.

3. Requirements

This section lists the requirements which have been identified. We recommend conducting some sort of survey of users and potential users of the P4DTI, to identify additional requirements. Such a survey was conducted as part of the analysis phase of the original P4DTI project, and provided very valuable results [RB 2000-04-11].

Section 4 of this document analyses these requirements individually.

Gerry Thompson recorded the requirements for P4DTI 2.0, as he perceived them, in an e-mail message, subject "Feature requests for P4DTI 2.0" [Thompson 2002-07-30].

This table summarizes the requirements identified by Gerry.

• 1. Bugzilla integration enhancements
  • 1.1. Bugzilla 2.16
  • 1.2. Bugzilla 2.18
  • 1.3. Custom fields in Bugzilla 2.18, supporting all Perforce field types
    • 1.3.1. Enhance Bugzilla 2.18 to support requirement 1.3.
• 2. Easier installation
• 3. Solaris
  • 3.1. job000209 (No package for Solaris)
  • 3.2. job000320 (P4DTI is not tested on Solaris)
  • 3.3. job000324 (Bugzilla patch doesn't work well on Solaris 8)
• 4. Easier administration and debugging
  • 4.1. job000524
  • 4.2. job000525 (Perforce job008041)
  • 4.3. Explanation of P4DTI database schema
  • 4.4. Diagnose and modify bad data so that P4DTI can continue
  • 4.5. Support tool for quering and fixing MySQL database entities in Bugzilla
• 5. Share p4 logger with other daemons
• 6. Perforce as a defect tracker (Perforce job008768)
  • 6.1. Don't touch jobspec except to add fields
  • 6.2. Share p4 logger with other daemons (same as requirement 5)
• 7. Provide management information
  • 7.1. Using p4report
  • 7.2. Query: How fast is codeline changing?
  • 7.3. Query: How fast are serious bugs getting fixed?
  • 7.4. Query: Will we meet our targets?
• 8. Configurable jobspec (Perforce job0008678)

The following requirements have arisen on the perforce-user mailing list, in support calls, or in e-mail to P4DTI addresses at Ravenbrook:

• 9. Additional defect tracker integrations;
• 10. Improved recovery from replication errors;
• 11. Meaningful filespec replication;
• 12. Closer web integration.

4. Analysis

Each sub-section below analyses a particular requirement sufficiently to sketch one or more solutions and to estimate the effort required to implement a solution. Making any significant change to the P4DTI will require analysing the work to be done, some design work, implementing the change, writing user documentation, and testing. The estimation here has been done by breaking the change down into these basic parts and estimating each part separately. Note that the parts are not sequential phases: they overlap and alternate.

All estimates are expressed as a number of hours of effort.

Testing is generally the largest single component, as it includes designing and implementing appropriate unit and system tests (both automated and manual), running the tests, analysing and responding to the results. Testing may start before coding, and always continues until after documentation is written.

Some of the estimates are quite vague. This indicates that more analysis of that requirement is required before more precise estimates can be made.

A question mark "?" indicates an item which has not been estimated. A hyphen "-" indicats an item which will not be necessary for that requirement (e.g. many simpler requirements do not require a "Design" stage).

4.1. Bugzilla

Bugzilla 2.16 support (requirement 1.1) is already in P4DTI version 1.5, and so is not included in this analysis [Thompson 2002-09-27].

Bugzilla 2.18 support is hard to estimate. Bugzilla 2.18 does not yet exist and will include many changes, some of which are still quite unclear. However, we can esimate the effort required from our previous experience with Bugzilla version changes, and as the 2.18 release approaches we can prepare the integration work by tracking the Bugzilla sources (available by anonymous CVS and in occasional tarball releases).

In our experience, accommodating a single major change in Bugzilla takes 30-60 hours of P4DTI work, including analysis, development, test development, testing, and documentation. Adjusting to all the minor changes in a Bugzilla release takes about another 50 hours. And simply importing a Bugzilla release, preparing and testing an appropriate Bugzilla patch, updating our test suite, and and updating documentation to reflect changed support, takes another 20 hours.

The major changes planned for Bugzilla 2.18 are as follows:

• Custom fields (see separate remarks below);
• Custom workflow;
• Sybase and PostgreSQL compatibility;
• New groups system.

So on the basis of our experience with previous Bugzilla changes, supporting Bugzilla 2.18 will cost about 260 hours.

4.1.1. Bugzilla "custom fields"

There's aready a patch for custom field support (see <http://bugzilla.mozilla.org/show_bug.cgi?id=91037>) so it's unlikely that Ravenbrook would need to develop custom fields from scratch.

Recent traffic (2002-12) on the bugzilla developers mailing list suggests that the design of the custom fields feature in 2.18 will be quite different from that of the currently available patch, and will involve fairly radical schema changes. Even if this is so, a system incorporating the new design is certain to be available (either as a 2.17 tarball or as a patch to 2.16.1) some time before the release of 2.18.

At the very least we could adapt a Bugzilla custom fields patch to support all Perforce field types and roll it into the P4DTI Bugzilla patch.

Estimate (hours of effort):

4.2. Easier installation

The actual "installation" of the P4DTI is very simple. What takes time is assembling and installing the prerequisite packages, and then configuring the P4DTI and debugging the configuration.

So, there are two parts to the solution:

• 1. bundling and automatic installation of as many prerequisites as possible;
• 2. improved configuration and debugging.

We believe there is a significant overlap between this second part and requirement 4, "easier administration and debugging" (see section 4.4 for analysis). In addition to the solution proposed in that section, we propose a configuration wizard: a step-by-step guided configuration procedure using web-based forms, with documentation included at each stage, and cross-referenced to the manual. This will reduce the incidence of people trying to set up the P4DTI without understanding the configuration parameters.

As part of the analysis of this requirement, we would like to talk to customers who have installed the P4DTI about their experiences, so that we can focus our efforts on the most difficult and time-consuming stages.

We already provide links to all the prerequisites and redistribute them. Bundling them with the P4DTI download may help a little. It's questionable whether we want to install them automatically. Any such system would have to check for existing installations (for example, Bugzilla and MySQL might well already be in use). The Python package system may allow us to bundle and install any necessary Python packagesxo. Note that there may be licensing issues with some such packages. There may be additional advantages to bundling them, in particular it could be used to fix the version of these packages which are in use and therefore to reduce support problems.

There are systems for deploying Python applications, including packages and libraries, without requiring a Python installation. The best-known appears to be py2exe. [Heller 2002-05-07]

Estimate (hours of effort):

4.3. Solaris

We will would need to arrange access to a Solaris machine for development and testing. In the past we have arranged access to a machine at Perforce for this purpose. If this is not possible, the cost of a suitable Solaris machine must be added to the estimate below. Solaris/x86 runs on "ordinary PCs", so this is not a large cost.

One of the main difficulties in installing the P4DTI on Solaris is the lack of a suitable "patch" program for patching Bugzilla. We either need to write a Python program which will do the patching, or add GNU patch as a prerequisite (this has been indicated in the Administrator's Guide in P4DTI version 1.5). A Python patcher might be a good idea across all platforms, as it may be a good solution to the problem of intelligently patching templatized versions of Bugzilla (2.16 and above).

The other main part of this work is investigating the Solaris packaging system and developing a package containing the P4DTI. There are good network resources for this [Mark 2001-01-30].

In addition, the test suite will need to be made to work on Solaris, and the test procedure will have to include Solaris in future. This will increase the cost of each new release of the P4DTI.

Estimate (hours of effort):

4.4. Easier administration and debugging

We believe that at present this is the most pressing requirement, and also the best justification for increasing the major version number of the P4DTI, as a solution would radically change the administrator's experience of the P4DTI.

P4DTI configuration, administration, and debugging is currently done through a command line and text file interface. P4DTI administrators are required to edit some Python code. They must do this while working their way through a fairly long manual. There is no immeditate feedback or assistance with configuration. Monitoring P4DTI activity means reading a log file or dealing with e-mail messages from the replicator.

We propose changing the nature of the P4DTI from a command line tool to a web-based tool. After installation, all configuration, control, and monitoring will be done through a CGI front-end, in a similar manner to Bugzilla. In addition, the web interface will be linked to on-line documentation.

As an alternative to a pure CGI interface, we should also consider integrating the P4DTI administrative interface with one of the Perforce graphical user interfaces, such as P4Win or P4Web. Most of the work for this requirement would be independent of the chosen GUI.

The configuration interface would provide immediate validation of configuration items, and could allow access to advanced configuration items without overburdening administrators of simple installations.

Monitoring P4DTI activity interactively would allow problems to be tracked at a fine level while not overwhelming the administrator with detail during normal operation, and could provide statistical overviews of P4DTI activity (and hence of activity in Perforce and in the defect tracker). It could also provide visibility into P4DTI internal data structures, not normally visible to administrators but potentially useful for diagnosing problems.

Alerts and notifications currently sent by email would also be provided and archived in the web interface.

A web-based interface can be linked to and from both Bugzilla and P4Web, to further integrate Perforce and the defect tracker.

An interactive interface may make it possible to display and edit the relationship between defect tracker fields and the Perforce jobspec, giving greater flexibility.

The front-end would also include a tool to force replication of a fix record or changelist that hasn't been replicated, to re-replicate jobs, to check consistency of particular items, and so forth. This would be like the current "refresh" and "check" scripts, but more specific and interactive.

Estimate (hours of effort):

4.5. Share the p4 logger

It's easy to change the P4DTI replicator not to clear the logger, but this will lead to the logger record growing indefinitely. This may require a change to the Perforce server, because _someone_ has to clear the logger at some point, and this means some kind of liaison between the various daemons using it. Some research will need to be done to work out a solution.

Estimate (hours of effort):

4.6. Perforce as a defect tracker

This means developing a "dt_p4.py" module to treat a Perforce server like a defect tracker. The interesting questions here concern choosing jobs to replicate, making different jobspecs interoperate, and finding an acceptable way to store metadata from another Perforce server (e.g. fix records). Assuming this is straightforward, here's the estimate.

4.7. Provide management information

It's not clear that P4Report is the best means to achieve this, as we need to combine information from both the defect tracker database and the Perforce database. Do Perforce want us to use P4Report for some other reason (e.g. marketing, or to help develop P4Report)? We need to investigate what P4Report can do.

We can certainly adapt and generalize our own issue reporting script to work with the P4DTI through the proposed web front-end. The main challenge is generalizing it to cope with different branching strategies, or even ad-hoc branching.

Estimate (hours of effort):

4.8. Configurable jobspec

We could read the jobspec, the DT config, and the P4DTI config, and provide a web-based front-end for connecting the two, effectively doing "advanced configuration".

Gerry stated that no estimate was required for this at present.

4.9. Additional Integrations

We receive e-mail requests for additional P4DTI integrations about once every two months. These arrive by e-mail to Perforce Support, or to the Perforce Users mailing list, or directly to one of the P4DTI e-mail addresses at Ravenbrook. I expect that such requests also arrive at Perforce through other channels, e.g. sales.

Now that the TeamTrack integration is no longer maintained as part of the P4DTI project, we only have a single current integration to maintain. The flexibility of the P4DTI may inadvertently suffer as a result.

I believe that Perforce Support keep a record of the requests they receive, and could provide a list of candidate defect trackers. We should canvass relevant staff at Perforce, and possibly users on the Perforce Users mailing list, for candidates. We performed this exercise before[RB 2000-06-20], but the results are out-of-date.

This estimate is based on our existing estimate of such an extension [GDR 2000-10-24].

4.10. Improved Recovery from Replication Errors

The most persistent and difficult support calls from the P4DTI are those in which a replication is failing repeatedly. The P4DTI does send e-mail messages when a replication fails, and has an exponential back-off system to prevent polling too frequently until the failure is remedied. However, the design seems inadequate to cope with some installations, in which other aspects of the production process for whatever reason cause replication failures to occur quite often.

A particular problem is that the replicator will not update the Perforce logger or the defect tracker update log until it has successfully completed a replication poll. The effect of this is that each poll repeats the same replication work up to the point of failure, and then sends an identical message to the administrator.

A better system would mark certain jobs and issues as problematic and complete all other replication work in the current poll.

Unresolved questions include the mechanism for marking jobs and issues as problematic. A complete failure to replicate an issue usually results from an inability to update either a job or a defect tracker issue record. So presumably the record of problematic issues would have to be maintained on the side somewhere (e.g. in a plain text file on the P4DTI machine).

Estimate (hours of effort):

4.11. Meaningful Filespec Replication

The P4DTI includes replication of a job field called "p4dti-filespecs", containing a number of filespecs. The original intention of this field was to support defect trackers like DevTrack which allow one to directly associate document revisions with issues, and to allow developers to use the defect tracker interface to perform common defect resolution tasks directly.

One might want to use the filespec replication to say "this defect was observed on this branch" or "develop the fix for this defect on this branch".

However, the TeamTrack user interface was never updated to present filespecs to the user, and (more-or-less) nobody requested it as a feature, so we didn't add it to the Bugzilla user interface either. As a result, the P4DTI code for replicating filespecs is under-exercised and under-tested. But the P4DTI design still supports filespec replication, and it could be used by a new integration or by an improvement to the Bugzilla or TeamTrack integrations.

This requirement is to add the filespecs to the Bugzilla user interface, and extend the Bugzilla user interface to perform useful Perforce actions on filespecs (possibly by interoperation with P4Web).

Users have asked Perforce support for meaningful filespec replication.If we don't do this, we should probably remove all traces of filespec replication from the P4DTI.

Estimate (hours of effort):

4.12. Closer Web Integration

P4DTI 2 installations will include a number of web interfaces: Bugzilla, the P4DTI administration interface, and often a web interface to Perforce (such as P4Web). At present the integration between these interfaces is limited to links from Bugzilla to changelist and job records in a Perforce web interface, via the changelist_url and job_url configuration items. This requirements is to improve the integration of these interfaces, to provide a more unified user experience.

This work might include customizing the Bugzilla user interface (via the template system in Bugzilla 2.16 and above), providing more cross-links (e.g. linking to the P4DTI administration interface from the Bugzilla administration pages), and providing more Perforce functionality from the Bugzilla interface (e.g. for the filespec replication described in the previous requirement.

Estimate (hours of effort):

5. Recommendations

We strongly recommend canvassing more requirements, from a broad base of P4DTI stakeholders. This might include a P4DTI user survey, on the Perforce User and P4DTI Discussion mailing lists. It should also include key people at Perforce, Inc, at key customers, and PCPs. The goal of this exercise would be to identify unknown requirements and to assess the relative value of those requirements and of the requirements already known. When speaking to technical staff, there should be some attempt to identify use cases.

We recommend starting development of P4DTI 2 as soon as the requirements are better known. Analysis and development of some clearly known requirements could take place in parallel with continuing requirements analysis. Our initial analysis suggests that the requirements for such a project would definitely include the following:

• Support for Bugzilla 2.18
• Easier installation
• Easier administration
• Improved error recovery

The total effort required to meet these four requirements is estimated at around 1200 hours.

Such a project could start in 2003-03, and should take an incremental approach to delivery. An incremental project plan would be required, based on better understanding of the urgency and risk of different requirements. We expect that an initial version ("2.0"?) could be delivered within three or four months, and subsequent versions could be delivered about once every two months. We expect such a project would expend around 100 hours of effort every month during the development of the first few versions, eventually tailing off into a cheaper maintenance-only phase.

The incremental delivery plan might look something like this:

Maintenance and support of P4DTI 1 would continue until some release of P4DTI 2 is considered a good enough replacement for existing P4DTI 1 users.

A. References

B. Document History

Copyright © 2002 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.