www.ravenbrook.com

MMQA Study (Test System for MPS)

INTRODUCTION

This document is a study of the MMQA system*. The study aims to:
- help the reader understand the internal parts and operation of the MMQA system;
- help RHSK decide what to do with MMQA (use it as-is, modify it, replace it, ...);
- clearly expose the bits of MPS-knowledge that are held within MMQA;
- be useful reading for anyone designing a future test system.

(*: The MMQA system is a system written in Perl and C that is designed to test the MPS. It was written by Richard Tucker ("rit") in years around 1999.)

Readership: MPS staff. Confidential: no. Status: rough, incomplete. Author: RHSK, Ravenbrook Limited, 2005-10-31.

Background: refer to <http://info.ravenbrook.com/project/mps/doc/2002-06-18/obsolete-mminfo/mminfo/guide/mmqa/index.txt>.

Peer documents:
A quick 'howto' that shows you how to run the MMQA system:
<http://www.ravenbrook.com/project/mps/doc/2005-11-30/mmqa-study/mmqa-howto.txt>

SOURCE DOCUMENTS

.ref.guide: <http://info.ravenbrook.com/project/mps/doc/2002-06-18/obsolete-mminfo/mminfo/guide/mmqa/index.txt>
.ref.automation: <http://info.ravenbrook.com/project/mps/doc/2002-06-18/obsolete-mminfo/mminfo/guide/mmqa/automation/index.txt>
.ref.test-writing: <http://info.ravenbrook.com/project/mps/doc/2002-06-18/obsolete-mminfo/mminfo/guide/test/writing/index.txt>
.ref.mmqa-with-rit: <http://info.ravenbrook.com/project/mps/doc/2005-02-23/mmqa-with-rit/>
.ref.source: <http://info.ravenbrook.com/project/mps/master/test/...@155641>

OVERVIEW

.status: MMQA is old and cobwebby, but it contains much that is useful. It doesn't currently quite run on Windows XP or Mac OS X, but it feels as though it nearly does.

.blackbox: MMQA treats the MPS as a black box. Indeed it works hard to investigate the MPS 'ab initio' (eg. by manually parsing header files!). MMQA expects to be given MPS public headers (with "-i <includepath>") and the compiled MPS library (with "-l <library>"), and go from there.

.phase.clib: "qa clib" -- analyses the MPS, and builds MMQA libraries.
.phase.run: "qa run tttt/XX" -- runs test type=tttt number=XX.

.knowledge: The test system (MMQA) holds much knowledge about its test subject (MPS).

.statement: Each MMQA 'test' is simply a statement about the MPS, which may or may not be true for a given version of the MPS. It is not the case that all MMQA tests are expected to pass. Some have never passed. Some, perhaps, should never pass!

TEST SYSTEM CONSIDERATIONS

.part:The test system should be carefully partitioned into:
.part.harness: subject-agnostic test harness
.part.bridge: (if needed) a small 'plugin' to the test harness, ie. a subject-aware helper for the harness.
.part.subject: holds knowledge about the test subject, and should be as separate from test-harness as possible.
.part.subject.common: some support stuff shared by many tests
.part.subject.tests: all the tests.

(.part.bridge.arbitrary: Note that the distinction between .part.bridge and .part.subject.common is somewhat arbitrary. The idea is that .part.bridge holds 'special' stuff that could not be done by any ordinary test, and necessarily has intimate knowledge of both the test harness and the test subject. In contrast, .part.subject.common connects to .part.harness only through a small and clearly defined interface -- see test/test/testlib/testlib.h.)

.part.why.extract: It is important to be able to study and extract the .knowledge. To make this easy, keep it separate from incidental details of how the test harness operates.

.part.why.adapt: Clear separation makes adaptation or re-writing easier and less likely to accidentally throw away .knowledge, when test requirements and scenarios change.

.role.harness: The harness does these things:
.role.harness.user: provide a nice user interface (help text, catalogue of tests, etc)
.role.harness.input: flexibly take input such as configuration data
.role.harness.check: do generic sanity checking on input
.role.harness.exec: execute the tests
.role.harness.output: do the right thing with output, eg. log it, analyse and summarise it, display it clearly, archive it, email it, etc.
.role.harness.auto: allow tests to be run on an automatic schedule (eg. overnight).
.role.harness.insulate: Insulate tests from each other, and prevent one broken test breaking others. For example, "ulimit -c 0" in test/test/script/platform prevents formation of core files that could fill up the disk.

MMQA OPERATIONS

.op.mmqa.make: MMQA needs to get things compiled. It does this 'by hand' with perl calling the compiler (see test/test/script/compile), as opposed to (say) using make.

---- (INCOMPLETE SECTION) ----

PHASES

.phase: Before the MMQA system compiles and runs a test (usually written in C, and calling the mps.h interface), it goes through several phases of preliminary checks. These preliminary phases are logically in .part.bridge.

.phase.platform:
.phase.platform.detect: test/test/script/options&platform_detect()
.phase.platform.settings: Once PLATFORM is known, test/test/script/platform&platform_settings() sets a whole raft of variables (eg. $cc_command, $cc_opts, etc).

.phase.mps-version: From MPS_LINK_OBJ, MMQA looks at the MPS version string, and automatically determines the MPS_PRODUCT (eg. "dylan", "epcore", etc) (.ref.guide). This is logically in .part.bridge. The mechanism that does this is test/test/script/runtest&mpslibvers(), which uses $stringscommand to fill in %mpslibvers.

.command.clib: The command "qa clib" makes MMQA do these phases:

.phase.determine-interface: MMQA uses "the mps header files and link objects to work out what interfaces are available for the test libraries to use" (.ref.guide#compile.libraries). The mechanism that does this is test/test/script/clib&scrutinise(). The facts recorded about the interface are:
.phase.determine-interface.mps-headers: MPS header files %mps_headers
.phase.determine-interface.mps-symbols: symbols in those header files %mps_symbols
.phase.determine-interface.mps-linkable: symbols defined in the MPS library %mps_linkable
.phase.determine-interface.record: All this is recorded as MMQA_{HEADER|SYMBOL|DEFINED}_xxxx in $obj_dir/mmqasym.h.

.phase.prepare-test-common: MMQA compiles and links .part.subject.common, ie. the .c files in test/test/testlib/ that are listed in test/test/testlib/manifest.

.phase.consistency-check.record: At .command.clib time, MMQA records lots of facts about the presence and modification-dates of MPS headers and libraries, into "$obj_dir/record" (See test/test/script/clib&record_clib()). See .phase.consistency-check.test below.

.command.run: The command "qa run" makes MMQA do these phases:

.phase.consistency-check.test: At .command.run time, the facts recorded by .phase.consistency-check.record get checked by test/test/script/clib&test_clib(), to make sure that no headers or libraries have changed since .phase.prepare-test-common.

.phase.compile:

.phase.run:

.phase.run.input: Some tests require large amounts of input data (eg. a very long sequence of sizes to allocate). Such data can be stored in a "data file" associated with the test. MMQA provides the data to the running test code via standard-input.

.phase.output:

---- (END OF INCOMPLETE SECTION) ----

B. DOCUMENT HISTORY

2005-10-31 RHSK Created.
2005-11-30 RHSK Edits, spruce up a bit, add link to concise howto.

C. COPYRIGHT AND LICENSE

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

3. Redistributions in any form must be accompanied by information on how
to obtain complete source code for this software and any
accompanying software that uses this software. The source code must
either be included in the distribution or be available for no more than
the cost of distribution plus a nominal fee, and must be freely
redistributable under reasonable conditions. For an executable file,
complete source code means the source code for all modules it contains.
It does not include source code for modules or files that typically
accompany the major components of the operating system on which the
executable file runs.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, OR NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

$Id: //info.ravenbrook.com/project/mps/doc/2005-11-30/mmqa-study/index.txt#1 $