Capacity, performance and responsiveness requirements for the P4DTI

1. Introduction

This document analyzes the capacity, performance and responsiveness requirements for the Perforce Defect Tracking Integration (P4DTI). The intention is to produce numbers that can be used to analyze the results of performance and capacity problems, so as to detect and resolve problems like job000277 (capacity) and job000279 (performance) before they come up again.

The intended readership is project developers.

This document is not confidential.

2. General notes

The goal of the P4DTI is to handle the work of software developers. It must be able to handle human activity. There is no requirement to handle the large volumes of data that might be generated by automatic processes.

Here are some data values which are needed for this analysis:

Users on a Perforce server. The survey [NB 2000-12-21] suggests 10s of users is typical. We know of sites with 100s of licenses. 1000 might be good maximum value.

Number of defects being tracker. The survey [NB 2000-12-21] suggests 1000s of defects is typical. We alpha tested [GDR 2000-11-01] at a site with 10,000s of defects. We know of an on-line defect tracking system [Mozilla] which had 88,680 defects on 2001-07-01. 1,000,000 might be a good maximum value.

Rate of defect activity (number of changes to defects per user per day). Analysis of Ravenbrook's Perforce server suggests average rates in the the 1s and maximum rates in the 10s. 100 might be a good maximum value.

3. Capacity

Capacity is a measure of the volume of data that a system can handle (without reference to the rate at which it handles the data).

We know of many Perforce customers with 1,000s of defects. It is therefore critical that the P4DTI can handle 10,000 defects. We know of some Perforce customers with 10,000s of defects. It is therefore essential that the P4DTI can handle 100,000 defects. We know of organizations with 100,000s of defects. So we should aim at handing 1,000,000 defects. In an ideal world the P4DTI would not be a limitation on the integrated: we are be able to handle as many defects as Perforce and the defect tracker can.

4. Performance

Performance is a measure of the rate at which a system can handle data.

We have two performance requirements: performance of normal interactive operation (that is, during replication); and performance of batch operations (migration, refreshing and consistency checking).

We know of many Perforce customers with 10s of users and we think an average rate of changes of 1s per day may be typical. It is therefore critical that the P4DTI can handle 1,000 events per day. We know of some Perforce customers with 100s of users. It is therefore essential that the P4DTI can handle 10,000 events per day. The rate of activity may be as high as 100 events per user per day. We should aim at handling 100,000 events per day. In an ideal world the P4DTI would be able to handle as much activity as Perforce and the defect tracker can cope with.

It is best for administrators if batch operations can complete overnight (say in 12 hours). However, it is probably acceptable if they can complete over the weekend (say in 60 hours). Combining the capacity requirement with these figures gives the following:

It is critical that the P4DTI handle 10,000 defects in 60 hours (166 per hour).

It is essential that the P4DTI handle 100,000 defects in 60 hours (1666 per hour).

We should plan to handle 1,000,000 defects in 12 hours (83333 per hour or 23 defects per second).

5. Responsiveness

Responsiveness is a measure of how quickly results of computation are made available to users.

The P4DTI has no user interface, but it is still important that it make changes in a timely manner, to avoid a high rate of conflicts. The architecture analysis [GDR 2000-05-30, 2.1.1] suggests that a responsive time of the order of 1 second may be needed to avoid conflicts. Our experience of testing the P4DTI suggests that the analysis is unrealistic: people don't make changes according to the random model used in the analysis.

I believe that a latency of 5 minutes would be acceptable, but we should plan for 1 minute.

A. References

B. Document History

Copyright © 2001 Ravenbrook Limited. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.