Ravenbrook / Projects / Perforce Defect Tracking Integration / Project Documents
Perforce Defect Tracking Integration Project
This document is an investigation into ways in which Perforce Software can meet its users' demands for defect tracking. It is intended to inform and advise Perforce about how to meet those demands. It also touches on the ways in which the alternatives would fit various business strategies for Perforce, regardless of current user demand.
If Perforce Software decides to follow these recommendations, then this document will provide an important record of the justification for that decision, so that it can be re-evaluated and understood further down the line when all the discussions have been forgotten.
The readership of this document is the senior management of Perforce Software. However, it could be made available to all staff.
This document is not confidential.
There are four possibilities: create a defect tracking product, enhance Perforce to meet more defect tracking requirements, integrate Perforce with third party defect tracking systems, or do nothing.
Creating a defect tracking product is a large project (many man years) consistent with Perforce expanding into the general software engineering tools market. The project would cause changes in the way that Perforce Software is managed.
Enhancing Perforce is unlikely to meet the requirements of the real market for defect tracking -- engineering management -- and could also compromise the conceptual integrity of Perforce.
Integrating with third party systems is a relatively small but substantial project that would satisfy the demands made by most Perforce users, and would not require major expansion in Perforce Software.
Doing nothing would probably not affect Perforce Software's growth rate, and would allow resources to be put into other projects.
Assuming that Perforce Software would like to continue with its current business model and company culture, I recommend integration with third party systems.
I recommend researching what integration would mean (partly by gathering existing integrations done by Perforce users) and enhancing Perforce to add a general and open interface for integration. Then develop integrations with a few popular systems initially.
Note that this interface would also allow the development of a complete defect tracking product either by Perforce or a third party, so it keeps the option open.
Defect Tracking is how engineering management plans and tracks the resolution of defects in software products. It often extended to apply to the planning and tracking of new product features.
The purpose of Defect Tracking is to help engineering management achieve their goal of producing quality products on time and to budget.
The purpose of planning is to co-ordinate work to achieve a desired result, to anticipate costs and other constraints, and to organize resources effectively.
In the case of defect tracking, a tool would help the management control which defects will be resolved and when. It would also help understand which resources are required, and measure the costs involved.
Specifically, a system usually allows changes to be planned which will resolve defects or implement enhancement requests. The changes are usually related to a release, and often scheduled for a particular time frame or milestone. Note that the changes are often not the same entities as the record of the defect or enhancement request: more than one change may be required at different times, or on different branches, to satisfy the customer.
Management requires a clear and easily modified view of defects and their planned resolution.
The purpose of tracking is to provide visibility into actual progress so that management can take action when performance deviates from plans. Tracking is how managers understand whether progress is matching the plan, and therefore whether targets are likely to be hit.
In the case of defect tracking, this is about monitoring the progress of changes, and intervening when necessary to keep things on schedule. Often re-planning is required, requiring the plan be flexible and cheap to modify. In addition, when controlling a large group, the manager has to be able to get a measurement of overall progress, such as the proportion of changes behind schedule or the total estimated effort behind schedule.
Control is the means by which management applies and controls resources in order to implement the plan.
In the case of defect tracking, the main resource is the effort of the engineers. The defect tracker must allow the manager to assign work to engineers, or even delegate areas of work to subordinate managers. The engineers need to know what work has been assigned to them, and feed back information about the progress of that work to the management for tracking purposes. Since each defect or enhancement request goes through many stages on its way to being resolved, there are usually a number of different tasks which might be assigned, with different criteria for completion.
The process is the program for the organization -- the way that the organization does things. Process evolution is how the organization modifies its process over time in order to improve its methods or deal with changing circumstances.
Often, most of the engineering department's process is implemented by the defect tracking system, especially if enhancement requests are processed like defect reports. A well developed process for a medium sized engineering organization would be like this:
Note, only steps 8 and 11 closely involve SCM.
This is only an example. There are many variations in process in use in different organizations, and processes must change for organizations to remain healthy. The defect tracking system is involved in the process above from start to finish.
A defect tracking system must provide the means to change what people are doing in order to keep things going according to plan, and the means to change the way that resources are used in order to improve quality and timeliness and reduce cost. A defect tracking system must be flexible enough to implement differing processes.
The flexibility to implement varying processes is the number one claim made by most of the products in this market.
The defect tracking system must be accessible to a wide variety of people. Management want to measure and control the activities of the engineering organization using the tracking system. Engineers will receive their instructions and have their work tracked through the system. Sales and marketing will want to know what the development status of various defects or features is so that they can give information to customers. Support need to track the status of issues for the customers. This implies that the defect tracking system must have a very accessible interface. (But also a secure one, see 3.1.6.)
Defect tracking systems are often required to integrate with other information systems, such as customer databases, contact information, requirements management packages, or project planning systems, especially when the software engineering organization is part of a larger project.
Defect tracking systems are usually required to implement security measures, such as only allowing access to information for certain groups, and only allowing modification by those authorized to make take the next step in the process. Engineering organizations are often under contractual conditions which require them to keep information about defects and enhancements secret between clients ([PJ 2000], appendix D). And with the system being critical to the management of the engineering organization, managers must be certain that the information in the system cannot be tampered with or sabotaged.
Defect tracking systems are also seen as sources of knowledge about the product being developed. Management seek to extract knowledge from the system for the use of their support organization, which is sometimes outsourced. They also want to measure overall trends or gather information about the product, such as which subsystems are the most defective or costly to maintain.
The customers of a defect tracking system are usually engineering management, not engineers. Engineering management have all the primary requirements: planning, tracking, control, and process. Engineering management will try to choose a system that is acceptable to engineers, and fits with existing practices, so as to minimize the cost of process change, but in the end these four primary requirements will rule.
Perforce Software could create a defect tracking product which integrates well with Perforce SCM.
This would be consistent with a business strategy in which Perforce Software develops a family of tools for engineering organizations. Proceeding further in this direction would be planning tools, requirements management tools, SQA and test tools. In fact, tools to cover all of CMM level 2 [KPA 1.1] and beyond. The CMM gives a clear road map for expansion. The result would be that Perforce Software becomes rather like Rational. I believe that there would be market pressure to continue in this direction if a defect tracking product were successful.
Existing tracking systems are specialized 4GLs: workflow description languages generating database applications in order to meet the flexibility requirement (3.1.4). They are specialized in that they usually provide easy ways for managers to define the kinds of processes needed for defect tracking, often through graphical interfaces.
My initial charter was to think about "a defect tracking system in line with Perforce's way of doing things, and not necessarily fitting into an existing process" (Nigel Chanter, 2000-01-07). Perforce's way of doing things, in its product, is to provide something clean, elegant, efficient, neutral, and with a simple and well defined interface. Perforce SCM is neutral in that it doesn't impose much in the way of process, and its well defined interface means that it can fit easily into many organizations. At the Perforce User Conference of 1999 it was clear to me that Perforce was being used in many different ways.
I do not believe a defect tracking system can be very successful without also being fairly process neutral, in order to meet the process flexibility requirement (3.1.4). (Note that flexibility to define and evolve processes is the biggest claim of many of the existing systems -- see appendix C.) It also seems that, in order to meet this requirement, a product must be like many of the existing products: specialized 4GLs on a fairly general DBMS. The process definition functionality must be accessible to non-engineers.
Perforce defect tracking system therefore probably wouldn't be fundamentally better than existing ones, unlike Perforce SCM. Perforce would have less of a competitive edge, and the current business model (of expansion through word of mouth and simply doing the job very well) would not apply well.
There appears to be very little pressure from Perforce users to create a defect tracking product (appendix D).
Perforce Software could add features to Perforce SCM to meet defect tracking needs, with the aim of enhancing Perforce's appeal, and increasing the rate of growth in the market.
I believe there are two stereotypes for people who have defect tracking needs: engineers and managers. I also believe that most of the feedback that has reached Perforce has come from engineers, who are the main users of Perforce SCM. This doesn't make their requests invalid, but I think it does mean that Perforce isn't seeing the whole picture. I believe the requirements in section 3.1 are those that management care about, and that management will be the primary customers.
General defect tracking requirements are hard to meet (see 4.1). It would compromise Perforce SCM's integrity to try to meet them by extension, and its integrity is important to Perforce's success. The question is therefore how far to extend Perforce SCM to meet defect tracking requirements. Where to draw the line?
To determine what can be done I would recommend a systematic survey of Perforce customers to determine who uses jobs. This would allow Perforce to understand the type of organization that uses jobs. (I expect it's mostly small or immature organizations, not that they are in short supply.) Perforce should then work to understand how to increase the appeal of jobs to those organizations, and a few more.
The Perforce User Conference 1999 held a "birds of a feather" session on jobs [JW 1999]. The notes from this session indicate that quite a few customers are merely using jobs to integrate with third party systems. Other requested enhancements included metrics (3.1.2, knowledge in 3.1.6), workflow (3.1.3, 3.1.4), tracking between codelines, versioning of jobs (security and integrity in 3.1.6), and making tea. However, the session may not have been representative and I'd recommend a systematic survey to determine requirements before doing much work on enhancement.
My intuition tells me that the most important enhancements would be to provide better ways of organizing jobs into groups, in order, for example, to plan which jobs are to be done for a particular future release (planning 3.1.1) and to improve the workflow and reporting (tracking 3.1.2, control 3.1.3). I would shy away from a fully configurable workflow, but provide some basic workflow customization. I also think a really good web-based interface to jobs, especially for management reports, is essential.
The problem with all this is that I don't think management will be convinced by any partial solution to the defect tracking requirements they have. (See [LM 2000] for anecdotal evidence.) I therefore think that enhancing Perforce SCM will have limited effect on sales. Integration with third party solutions is much more convincing.
Perforce Software could integrate Perforce with third party defect tracking systems. Like enhancing Perforce (4.2), this would have the aim of enhancing Perforce's appeal, and increasing the rate of growth in the market.
There are two possibilities here:
The second option is by far the most requested course of action by the current users (appendix D, [JW 1999]). It sidesteps the problem of satisfying all of management's difficult defect tracking requirements (3.1), or even understanding exactly what they are -- a very significant advantage for a company that doesn't use a system itself. It allows engineers to recommend Perforce for SCM without compromising the management's choice of defect tracking (and other) systems. I also believe that organizations will choose an SCM that integrates with their choice of defect tracker, or simply one that leaves their options open.
It also fits well with Perforce's current business model. It doesn't require launching a major new project. It doesn't step outside Perforce's core competencies. It contributes to a vision in which Perforce is the standard SCM and has open interfaces.
Lastly, it leaves Perforce's options open. An open interface to existing defect tracking systems would allow for the development of a new defect tracking product, either by Perforce or a partner or subsidiary (4.1).
Perforce has already been integrated with systems by a number of users (appendix D), some of whom have offered to share their work. Gathering and organizing this material would be a good starting point, and an immediate way of answering defect tracking queries.
In parallel, I recommend a study of a range of existing defect tracking systems in order to understand what it means to integrate them, and what it would take to do so. This will establish the requirements of the integration pieces. Unifying these and extrapolating would provide a design for a general interface to Perforce SCM to support the integration pieces. I'd recommend specifying this interface and opening it up for review with the Perforce user community. Then implement the interface in parallel with one or two representative integrations. Publish these, then develop further integrations as demanded, and so on into ongoing development and maintenance.
Some very rough estimates, in man months: 1.5 to gather and investigate existing implementations, 1.5 to make these available, 3 to investigate defect tracking systems and establish requirements, 3 to design, 3 to implement, 6 to test (including a beta program).
I believe this is Perforce Software's best course of action, based on its business model, capabilities, and plans for growth.
It's important not to forget this option. The demand from users isn't very strong, and the lack of a well developed defect tracking solution doesn't seem to be holding back Perforce's very respectable growth. It's quite possible that Perforce would be better off putting its resources into other projects, depending on the business plan. Since Perforce is in such a good position, it might be a good time to invest in projects which anticipate the problems of growth.
Assuming that Perforce Software would like to continue with its current business model and company culture, I recommend integration with third party systems. See section 2 for a full summary.
| [GDR 2000-05-11] | "Defect Tracking Project Meeting, 2000-05-11"; Gareth Rees; Ravenbrook Limited; 2000-05-11. |
| [JW 1999] | "Jobs BOF (long time in coming)" (e-mail message); Jo Woodison; Perforce Software; 1999-06-30 15:31:07 -0700. |
| [KPA 1.1] | "Key Practices of the Capability Maturity Model, Version 1.1"; Mark C. Paulk, Charles V. Weber, Suzanne M. Garcia, Mary Beth Chrissis, Marilyn Bush; Software Engineering Institute; 1993-02; CMU/SEI-93-TR-025, ESC-TR-93-178; <URL: http://www.sei.cmu.edu/cmm/cmm.html> |
| [LM 2000] | "Perforce Training" (e-mail message); Lisa Maguire (via Leslie Smith); Esual Software; 2000-01-28. "My boss absolutely hated [the defect tracking function]." |
| [PJ 2000] | Interview with Peter Jackson of Symbian (notes); Richard Brooksby; Ravenbrook Limited; 2000-01-27. "...the security model that goes with defect tracking is going to be quite complicated [for us]." |
| 2000-01-24 | RB | Initial outline after interviews with Leslie Smith, Chris Seiwald, and Laura Wingerd. |
| 2000-01-25 | RB | Added material to outline based on survey of Perforce User mailing list. Added material from interviews with Nigel Chanter, Fanny Nudo, Robert Orenstein. |
| 2000-01-26 | RB | Added analysis based on survey of existing defect tracking products, mainly requirements and target market information. |
| 2000-01-27 | RB | Assembled notes into draft document. Further research on existing products. |
| 2000-01-28 | RB | Added section on enhancing Perforce and integration. First complete draft. Reviewed with Nick Barnes. Completed. |
| 2000-05-25 | RB | Marked as "not confidential" as agreed with Perforce [GDR 2000-05-11, 5]. |
| 2000-05-31 | RB | Converted to XHTML. |
| 2000-06-30 | GDR | Made references link directly to target, rather than the references section. |
This is a list of third party defect tracking systems that have been mentioned on the Perforce User mailing list, in rough priority order. The users almost always asked whether Perforce has been successfully integrated with these systems.
A more complete list of existing products can be found in the Problem Management Tools Summary at <URL: http://www.faqs.org/faqs/sw-config-mgmt/prob-mgt-tools/index.html>. A quick survey of some more of these (e.g. ProblemTracker by NetResults) shows similar claims to those below. Others worth looking at are Bugzilla and Scopus.
<URL: http://www.teamshare.com/products/teamtrack.htm>, <URL: http://www.teamshare.com/products/ttrack/facts/index.htm>
Claims: Out of the box. Scalability. Flexibility to implement any process, with definable states, routeing, process enforcement, configurable terminology, definable forms, workflow, etc. Accessible through web interface. User definable reports. High performance. Secure. Interface to any database back end.
Key marketing phrases: "Gain control over projects." "Track and monitor assignments." "Locate and remove bottlenecks in workflow." "Change your system instantly to improve workflow." "Manage risk." "Improve processes."
<URL: http://www.tower.com/specs_rz.htm>
Claims: Power to the users. ASCII-based. Highly extensible UI. Programmable. E-mail integration. Multiple synchronized databases.
<URL: http://www.soffront.com/>
Claims: Easy to use. Fully customizable [i.e. ability to define process]. Client-server. Web based. Bug tracking. Enhancements. Release tracking. Definable business rules [process, again]. Knowledge base management. Graphical reports. E-mail integration. Integrates with VSS, PVCS, MKS. Definable fields and forms. Attachments. Workflow definition.
<URL: http://www.rational.com/products/clear_ddts/index.jtmpl>
Claims: Easy to use querying. Web interface. E-mail integration. SQL interface. More than 40 management reports. Integrates with ClearCase, etc. Provides quality metrics for key decisions. Easily customizable to fit your process. ISO 9000, ISO 90001, DoD 2167, IEEE 1044.
<URL: http://www.merant.com/pvcs/products/tracker/index.asp>
Claims: Manages and communicates business and technical issues. Web interface. Tailor for your team, task, or responsibility. E-mail integration. Chart progress, trends, identify bottlenecks. Metrics package automates report generation. API. ODBC.
No information available on Clarify's web site. They appear to have shifted focus to providing a complete front office solution of some sort. It's not clear that ClearQuality is still available.
No information available. Judging by <URL: http://www.uw.com/>, UnderWare seem to have been acquired by Compuware/NuMega. They don't appear to be marketing Track Record, though there are some support downloads available.
StoneKeep have almost no statement of product benefits on their web site. Their description is almost entirely functional. They are clearly a very technical outfit with very little high-level material. It's hard to assess their product.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1997-October/000349.html>: A user tells Perforce to integrate, don't build you own.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1999-March/002115.html>: Wes Peters integrating Perforce with Keystone.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1997-October/000347.html>, <URL: http://maillist.perforce.com/pipermail/perforce-user/1997-October/000348.html>: Brad Appleton's wishlist. Includes: Ability to create own entities for call tracking, etc. Define own state transitions. Attachments.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1999-October/003163.html>: What I primarily need is good management of a bug through its lifecycle, status reports, and the ability for non-technical people to get reports and enter stuff.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1999-January/001845.html>: Glen Bachman's requirements.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1999-December/003461.html>: Where can a person go to learn about future directions for Perforce?
<URL: http://maillist.perforce.com/pipermail/perforce-user/1998-August/001178.html>: Richard Gieger of Network Appliance discusses what he's done there.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1999-October/003162.html>: Security and integrity: Can't trust developers to update all the information correctly.
<URL: http://maillist.perforce.com/pipermail/perforce-user/1997-June/000084.html>: "The company that writes a decent, small bug database in a portable way (hell, maybe with Java) that's easy to use, easy to script reports/updates, and easy to query, will make a lot of people very happy."
<URL: http://maillist.perforce.com/pipermail/perforce-user/1998-July/001137.html>: A review of Razor.
Copyright © 2000 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.
$Id: //info.ravenbrook.com/project/p4dti/doc/2000-01-24/options/index.html#6 $
Ravenbrook / Projects / Perforce Defect Tracking Integration / Project Documents