Received: from [172.16.1.179] (root@raven.ravenbrook.com [193.82.131.18]) by raven.ravenbrook.com (8.9.3/8.9.3) with ESMTP id BAA02891 for ; Wed, 1 Nov 2000 01:06:02 GMT Mime-Version: 1.0 X-Sender: gdr@pop3 Message-Id: Date: Tue, 31 Oct 2000 17:04:05 -0800 To: p4dti-staff@ravenbrook.com From: Gareth Rees Subject: Alpha test report for Mahi Networks, 2000-10-31 Content-Type: text/plain; charset="us-ascii" ; format="flowed" 1. The replicator.p4_run command needs to take better care of shell characters, quoting etc. Blithely doing "p4 job %s" or "p4 user %s" may not work (what if the job name has shell metacharacters in it?). 2. The P4DTI-user should be created using p4 user and some sensible fields like e-mail address, name and so on. 3. We need some way for administrators to force re-replication from the DT when the configuration changes. For example, RB re-organized the "title" and "description" fields, and needed to re-replicate. GDR tried setting the "last change" back to zero, so that the replicator looked at all changes in the table, but this might not be enough. We had to manually remove the jobnames when we changed their configuration. Perhaps some way to scrub the databases on both sides and start again as if from scratch. 4. cramstad's job was still owned by him when he closed it, even though the owner in TeamTrack was changed to dpalmer by the workflow. We guess that the change of the user doesn't get added to the changes table. 5. We need to test against Perforce's Developer Studio integration. 6. What happens when one of the servers is down? Is there a way to warn the users that the other system is down? How you tell how up to date the information is? Running the consistency checker each night and e-mailing the results would be a sensible thing to do. Would it be enough? 7. Reports. Which files are modified frequently? Where are the "hot spots" in terms of defects? Which files have changed (in total) in the development for a release? Impact analysis: if a file is changed then SQA may be suspicious of a whole bunch of related issues and related files. Mahi use Mercury Interactive's TestDirector to build test sets and use impact analysis to prioritize the regression testing. What release has a bug been fixed in? Could cross-check the release reports from TeamTrack and Perforce. You can implement one of these in Perforce by first finding all files changed on the main branch between release A and release B branch points, then mapping fixes -i onto each file in the set and unioning the results of the fixes commands. 8. People may want to nuke the Perforce data and start replicating again from scratch, perhaps because they want to radically redesign the way their jobspec looks or the details of replication. 9. TeamTrack says "select one" on the Actions menu, but you don't have to. Everyone goes there and asks what they're supposed to select. 10. How can we help Derek to install and support the integration? Derek needs to become more familiar with the source code and design documents. The AG should have a series of graded examples from simple configurations like selecting issues based on project or state to advanced things like building the fix description from the change comments of changes that are associated with the job. There should be a troubleshooting section: some of this will have to be done as support problems come in; how much can we anticipate? 11. It would be nice to have a better interface to job filters. We should certainly recommend appropriate job filters to developers, for example "user=me & status=foo" 12. Those job names are awkward to type on the command line. 14. Do we need to provide a Date: in the e-mail we send? 16. Retrospectively editing fixes: what scenarios are there? You should be able to make associations only in certain states in the workflow, or when you're the owner. Otherwise you could perhaps fix someone else's bugs? Perhaps there should be a set of permissions in the workflow (something like, if you're not the owner you can't make fixes). Note also that you can't actually go back and make associations because it will affect the status of the job and the transition in TeamTrack will fail. 18. Dave's comments: You should add rigidity to the tools to ensure that developers follow the process. Developers won't read the process document more than once. The integration shouldn't let you (say) check in 60 files and then reject your change and destroy your data. 20. The VC Update Group isn't quite general enough. It would be nice if TeamTrack has privileges based on fixes and changes. 22. Control of Perforce submissions based on the workflow. (Use triggers in Perforce.) For example, on a release branch you want to only allow changes that are associated with high-priority bugs. Need to be turned on and off based on branch or date or whatever. The owner of a branch may change depending on project life-cycle (perhaps the development manager at early stages; the SQA manager at late stages). Though it may be better to have different branches for different stages of the life-cycle.