USER REQUIREMENTS FOR DEFECT TRACKING INTEGRATION Richard Brooksby, Ravenbrook Limited, 2000-04-11 1. INTRODUCTION This document summarizes and analyses Perforce users' requirements of integration between Perforce and third-party defect tracking systems. This information was gathered as the "customer requirements" step set out in "Steps to Defect Tracking" [RB 2000-02-02]. Requirements in this document were produced in three ways: 1. a survey of a group of Perforce customers which were chosen to represent the Perforce user community in general (appendix C); 2. an analysis by people involved and the Capability Maturity Model; 3. by applying my own experience as a software engineering manager. This document is intended for Perforce Engineering Management, but could be made available to all staff. This document is not confidential. Please note that this is a summary report and does not contain all the detailed information necessary in a full requirements specification (anti-requirements, classification of requirements, technical details, tests to determine whether requirements are met, etc.) However, it does cover the main requirements from which the specification can be refined. I am developing this as I research the technical requirements. 2. EXECUTIVE SUMMARY The main purpose of integrating Perforce SCM with defect tracking systems is to ensure that the defect tracker's view of the quality of the product sources is consistent with the product sources. This allows the project manager to plan the work necessary to deliver the product, and to make delivery decisions. A secondary but important requirement is ease of use for Perforce users. They want to be able to navigate between related information in the defect tracker and Perforce quickly and easily. They also want a single interface for submitting changes to Perforce and making a corresponding state change in the defect tracker. The latter also contributes to consistency. The integration must also provide documentation of the product sources and changes to the sources in terms of customer requirements, and this means providing an easy-to-follow connection between the sources and changes and the defect tracker's records of the ways in which the product failed to meet its requirements. Perforce users are very keen to have an open, documented, and maintained interface for integration. Some said that this was more important than having an integration supplied by Perforce. Integration must provide for longer term analysis of what is going on as the product develops. What is the rate of defect finding and fixing, for example? Which areas of the system are most costly to maintain? These management questions must be answered for strategic planning. Integration should allow questions like these to be asked in terms of, or with reference to, the product sources. The next stage is to gather information about defect tracking systems in order to understand what it technically involved in integrating with them. This will allow Perforce to deduce what is technically required to meet these user requirements. 3. REQUIREMENTS This section lists the main high-level user requirements that I believe apply to the defect tracking integration project, in order of priority. These are the requirements discovered as described in section 4.3 of "Steps to Defect Tracking" [RB 2000-02-02, 4.3]. During the survey (appendix C) customers often made specific suggestions or requests for changes to Perforce. A common request was that the Perforce "submit" action cause a corresponding change in the defect tracking system, or that Perforce have pre and post triggers on all commands. These requirements look beyond these specific requests to explain why these things are important. This is so that Perforce can design a practical, maintainable, and adaptable solution to meet the underlying requirements of the users, rather than simply adding a shopping list of features -- the doom of many a good piece of software. 3.1. Consistency 3.1.1. Definition The defect tracker state must be consistent with the state of the product sources. Note that this doesn't say consistent with the state of the SCM system. The SCM system is just an intermediary between the sources and the defect tracker. What matters is whether and how well the sources meet the customer requirements. This almost certainly means the SCM state will also need to be consistent with the sources and the defect tracker, but it's important not to lose sight of the real customer requirement. 3.1.2. Justification The organization's requirement of the project manager is to produce quality products on time and to budget. To meet this goal the project manager needs accurate information about the ways in which the product doesn't meet requirements (both defects and enhancements). The project manager needs to plan and track changes to the product to make it meet its requirements. Most, if not all, of these changes are to the product sources, which are stored in the configuration management system. The project manager also needs to prevent changes to the product which do not bring it closer to meeting its requirements. The risk is that the information in the defect tracking system becomes inaccurate and does not match the actual state of the product sources. The manager then does not know the actual quality, and cannot plan, track, or produce a good quality product. The primary requirement of integration is to reduce this risk. 3.1.3. Sources Russel Jackson , 2000-03-13 18:21:46: Job numbers the same as IDs in the defect tracking system. Ed Mack , 2000-03-18 00:24:33: Cross-reference defect IDs and changelists. Peter Jackson , 2000-03-24 08:03:13: Attach defect information to jobs so that it is automatically propogated through Perforce. Pam Ailes 2000-04-07 19:11:45: Avoid double-entry of data, to encourage use and avoid errors Jeff Parrish 2000-04-10 16:49:24: Items entered into a change request system would be automatically linked to something representing that bug/change request in Perforce (perhaps a job) and vice versa. These change requests would be visible in both systems. There would be a way of associating perforce changes with the change request, and that information would be visible directly through the change request system. The change request status would be visible via both Perforce and the change request system. Richard Geiger , 2000-03-13 19:16:12: Information about defects and work captured at submit time. Paul Goffin , 2000-03-13 16:18:53: Need to prove to external auditors that defects have been resolved, with traceability. The requests for "submit" to update the defect tracking system are all relevant to this requirement, because they are to do with ensuring that the information in both systems is consistent. If the systems are not connected, and the connection is not enforced, then the information will become inconsistent. 3.2. Ease of Use 3.2.1. Definition The defect tracking integration must make the jobs of the developers and managers easier (i.e. make it easier for them to produce a quality product etc.). 3.2.2. Justification This requirement is strongly related to the consistency requirement (3.1). It's clear from the complaints of the developers in the user survey that they're required to fill in defect tracker forms and manually ensure that they are consistent with SCM information. This is both tedious and error prone for the developers, and is a problem for the manager who has to enforce discipline in order to get consistent information (3.1). So, not only must the defect tracking integration provide consistency, it must make it easy. A system which gives developers electric shocks until they input consistent information into both systems, for example, would meet the consistency requirement but not the ease of use requirement. 3.2.3. Sources Sandy Currier , 2000-03-23 22:12:03: Reduced overheads for developers and managers. Pam Ailes 2000-04-07 19:11:45: Avoid double-entry of data, to encourage use and avoid errors Randall Robinson , 2000-03-15 18:40:45: Changelist description added to database. Submit screen must include database fields and their possible values using a Windows and Unix GUI. Must be customizable change, capable up updating many fields in the database. Alan Humphrey , 2000-02-20: Developer only has to do one thing, not also go to defect tracking system. Stage of development where defect was fixed. Root cause of defect. Stage of development where defect was introduced. 3.3. Documentation 3.3.1. Definition It must be easy to discover why the product sources are the way they are, and why they have changed, in terms of the customer requirements. For example, if a module is written in a strange way to avoid a situation which caused a defect there should be a simple and obvious connection from the module sources to the defect record, including the record of the decisions taken and what was done about the defect. (Recall that a defect is where the product fails to meet a requirement.) 3.3.2. Justification A connection between the sources, changes, and defects allows changes to be understood, reviewed, and tested, to ensure that they meet their requirements. The connection also provides insight into the system and why it is the way it is, so that design decisions can be made. This contributes to the long term quality of the product. A connection prevents situations in which developers dare not change the sources because they don't understand why they're written in a certain way. This is a common cause of stagnation and of creeping complexity in software. Some of this connection might be better provided by an integration of Perforce with a requirements management system. However, these are not commonly used. Most organizations don't have any better capture of their requirements than the defect tracking system's list of things that are wrong or that the customer wants. So, there is a requirement to provide the trail between the sources, changes, defects, customer requests, and requirements, but, there doesn't seem to be any demand for integration with requirements tools. There's a requirement for the defect tracking integration to provide this information as best it can. 3.3.3. Sources Glenn Bachmann , 2000-03-14 12:27:03: Trace development activities back to why they were done in the first place. Anonymous, 2000-03-17 15:31:53: Associate defect ID with changelist and vice versa, to "justify its existence". 3.4. Openness 3.4.1. Definition The interface that allows Perforce to be integrated with defect tracking systems must be: 1. public, 2. documented, and 3. maintained. 3.4.2. Purpose The interface must be open so that Perforce customers can integrate Perforce with any defect tracking system, including home grown systems. The interface must be open to provide review and feedback to improve the interface and thereby Perforce's ability to integrate. 3.4.3. Sources Sandy Currier , 2000-03-23 22:12:03: Enough design documentation to support change and adaptation. Open source it. Glenn Bachmann : Never want opaque integration. Must be able to do our own. Do one tight integration and leave an open interface. Richard Geiger , 2000-03-13 19:16:12: Give me an interface. Paul Goffin , 2000-03-13 16:18:53: COM/CORBA interface. Jeff Parrish 2000-04-10 16:49:24: I would much rather have the proper interface to support an integration, than a complete solution. 3.5. Analysis 3.5.1. Definition The integration must provide the ability to ask questions involving both the defect tracking system and the SCM system. Some examples: Which defects affected this file? Which customers caused this line of code to be written? Which parts of the system are most defective and cost the most to maintain? 3.5.2. Purpose Analysis allows defect prediction and prevention by development staff. In a mature organization the development team will analyse past defects and introduce standards and procedures to prevent similar problems in future. Organizations often need to measure their rate of both finding and fixing defects in order to predict their future quality and resource levels required for development, support, and so on. There is potential for integration with other CASE tools here. 3.5.3. Sources Alan Humphrey , 2000-02-20: More capture of data about defects. More ability to do defect prevention based on data. Sandy Currier , 2000-03-23 22:12:03. To be able to answer the following queries: (1) what 'bugs/issues' have been fixed in this release? (2) what are the differences in defect repairs lists for two arbitrary releases? (3) what bugs/issues are fixed in this codeline? Between 2 codelines? (4) what changes are present in this codeline? Between 2 codelines? (5) which releases have this bug fix? (6) which codelines are missing this bugfix? (7) have we dropped any repairs since the previous release? These queries should not rely on cached information but should dynamically search both the SCM and bug tracking database [i.e. they must be consistent with the sources (3.1)]. 4. CONCLUSION Based on evidence gathered from Perforce users, on an analysis by the roles of people involved, on the Capability Maturity Model, and on my own experience of engineering management, these are the primary requirements for integration between Perforce SCM and third party defect tracking systems. The next stage is to gather information about defect tracking systems in order to understand what it technically involved in integrating with them. This will allow Perforce to deduce what is technically required to meet these user requirements. Use cases involving both Perforce and some defect tracking systems will also be useful for determining detailed user requirements. Proposed solutions should be analysed against all these user requirements, against the technical requirements, against project requirements [RB 2000-02-02 4.6], and against the overall goals of the whole project [RB 2000-02-02, 3] in order to select the best architecture, design, and plan of implementation. A. REFERENCES [GDR 2000-05-11] "Defect Tracking Project Meeting, 2000-05-11"; Gareth Rees; Ravenbrook Limited; 2000-05-11. [KPA 1.1] "Key Practices of the Capability Maturity Model, Version 1.1"; Mark C. Paulk, Charles V. Weber, Suzanne M. Garcia, Mary Beth Chrissis, Marilyn Bush; Software Engineering Institute; 1993-02; CMU/SEI-93-TR-025, ESC-TR-93-178; . [RB 2000-01-24] "Options for Defect Tracking" (report for Perforce Software); Richard Brooksby; Ravenbrook Limited; 2000-01-24. [RB 2000-02-02] "Steps to Defect Tracking"; Richard Brooksby; Ravenbrook Limited; 2000-02-02. B. DOCUMENT HISTORY 2000-04-11 RB Drafted outline based on data gathered by e-mail. 2000-04-17 RB Drafted full text. 2000-04-18 RB Expanded requirements section with more justification and sources. 2000-04-19 RB Rounded off, added executive summary and conclusion. Reviewed with Nick Barnes. 2000-04-20 RB Edited after review by Gareth Rees. 2000-05-25 RB Marked as "not confidential" as agreed with Perforce [GDR 2000-05-11, 5]. 2000-05-30 RB Removed customer list and replaced with comment. Anonymized some contributors at their request. C. CUSTOMER SURVEY C.1. Selected Customers Leslie Smith and I selected a range of Perforce customers in order to get a representative spread of opinions. Here is the list of contacts that were used. [The list of customers has been removed from the published version of this report to protect their confidentiality. RB 2000-05-30] C.2. Questionnaire Once initial contact had been made, I asked the following questions, originally proposed in "Steps to Defect Tracking" [RB 2000-02-02, 4.3]: 1. What does "integrated" mean to you? What do you want from integrated SCM and Defect Tracking? What would it mean for Perforce to be integrated with defect tracking? Why would you want them integrated? What problems would this solve for you? 2. Do you want a complete integration or just an interface to make your own integration easier? Can our integration be flexible enough to meet your needs? What kind of interface might you need? 3. With which systems do you need Perforce integrated? In which environments? C.3. The Messages I exchanged about 100 messages with the Perforce users. I don't intend to reproduce them all here. Please apply to me if you would like copies of the messages. Copyright (C) 2000 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution. $Id: //info.ravenbrook.com/project/p4dti/doc/2000-04-11/user-reqs/index.txt#3 $