.. highlight:: none


.. index::
   pair: configuration; design

.. _design-config:


MPS Configuration
=================

.. mps:prefix:: design.mps.config


Introduction
------------

:mps:tag:`intro` This document describes how the `Memory Pool System
<https://www.ravenbrook.com/project/mps/>`_ source code is configured so
that it can target different architectures, operating systems, build
environments, varieties, and products.

:mps:tag:`readership` Any MPS developer; anyone porting the MPS to a new
platform.



Requirements
------------

:mps:tag:`req.import` The MPS must be simple to include in third-party projects.

:mps:tag:`req.arch` Allow architecture specific configurations of the MPS, so
that we can vary the MPS according to the target architecture.

:mps:tag:`req.os` Allow operating system specific configurations of the MPS,
so that we can vary the MPS according to the target OS.

:mps:tag:`req.builder` Allow build environment specific configurations of the
MPS, so that we can vary the MPS according to the compiler, etc.

:mps:tag:`req.var` Allow configurations with different amounts of
instrumentation (assertions, metering, etc.).

:mps:tag:`req.impact` The configuration system should have a minimal effect on
maintainability of the implementation.

:mps:tag:`req.port` The system should be easy to port across platforms.

:mps:tag:`req.maint` Maintenance of the configuration and build system should
not consume much developer time.


Retired requirements
....................

:mps:tag:`req.prod` Allow product specific configurations of the MPS, so that
we can build variants of the MPS for use in different products.  This
requirement has been retired on 2012-09-03 as part of work on the
variety-reform_ branch.  Client-specific customisation of the MPS will
be handled in source control, while the MPS source remains generic, to
reduce costs and increase reliability.  See [RB_2012-09-13]_.

.. _variety-reform: /project/mps/branch/2012-08-15/variety-reform


Definitions
-----------

:mps:tag:`def.platform` A *platform* is a combination of an architecture
(:mps:ref:`.def.arch`), an operating system (:mps:ref:`.def.os`), and a builder
(:mps:ref:`.def.builder`). The set of supported platforms is maintained in the
`Platforms section of "Building the Memory Pool System"
<platforms_>`_.

.. _platforms: ../../../manual/html/guide/build.html#platforms

:mps:tag:`def.arch` An *architecture* is processor type with associated calling
conventions and other binary interface stuff these days often called the
`ABI <https://en.wikipedia.org/wiki/Application_binary_interface>`_.
Most importantly for the MPS it determines the layout of the register
file, thread context, and thread stack.

:mps:tag:`def.os` An *operating system* is the interface to external resources.
Most importantly for the MPS it determines the low level interface to
virtual memory (if any) and threading.

:mps:tag:`def.builder` A *builder* is the tools (C compiler, etc.) used to make
the target (:mps:ref:`.def.target`).  The MPS minimises use of compiler-specific
extensions, but this is handy for suppressing warnings, inlining hints,
etc.

:mps:tag:`def.var` A *variety* determines things like the amount of debugging,
internal consistency checking, annotation, etc.  In modern IDEs this
called a "build configuration" and the usual default is to have two:
"debug" and "release". The MPS predates this convention, but the concept
is the same.

:mps:tag:`def.prod` A *product* is the intended product into which the MPS will
fit, e.g. ScriptWorks, Dylan, etc.  We no longer maintain this concept
as a dimension of configuration since :mps:ref:`.req.prod` has been retired.

:mps:tag:`def.target` The *target* is the result of the build.

:mps:tag:`def.option` An *option* is a feature of the MPS that is not
selected via the *platform* and *variety*. See :mps:ref:`.opt`.



Overview
--------

:mps:tag:`import.source` The MPS can be simply included in client products as
source code.  Since `version 1.110`_ we made it possible to simply
include the file ``mps.c`` in a client's build process, without
requiring a separate build of the MPS or linking a library.  This is
described `section 2.3.1, "Compiling for production" of the MPS manual
<compiling_>`_.

.. _`version 1.110`: https://www.ravenbrook.com/project/mps/version/1.110/
.. _compiling: ../../../manual/html/guide/build.html#compiling-for-production

:mps:tag:`no-gen` No generated code or external tools are required.  On most
platforms the only tool is the C compiler.  On 64-bit Windows we require
the assembler since Microsoft withdrew in-line assembler from their C
compiler.

:mps:tag:`no-spaghetti` Several of the MPS team have worked on some extremely
messy code bases which used a great number of ``#ifdef`` statements.
These quickly became very expensive to maintain and develop.  The
general rule in the MPS is "no ``#ifdefs``".  Instead, platform-specific
code is kept in separate source files and selected by carefully controlled
``#ifdefs``, such as in mps.c_.

.. _mps.c: ../../../code/mps.c

:mps:tag:`min-dep` Dependency on a particular configuration should be
minimized and localized when developing code.  This is enshrined in the
general rules for implementation [ref?] that are enforced by MPS
development procedures including code review and inspection.


The build system
----------------

Abstract build function
.......................

:mps:tag:`build.fun` The MPS implementation assumes only a simple "build
function" that takes a set of sources, possibly in several languages,
compiles them with a set of predefined preprocessor symbols, and links
the result with a set of libraries to form the target::

    target := build(<defs>, <srcs>, <libs>)

:mps:tag:`build.sep` Separate compilation and linkage can be seen as a
memoization of this function, and is not strictly necessary for the
build. Indeed, since `version 1.110` we found that modern compilers
are quite happy to compile the whole MPS in one go :mps:ref:`.import.source`.

:mps:tag:`build.cc` A consequence of this approach is that it should always
be possible to build a complete target with a single UNIX command line
calling the compiler driver (usually "cc" or "gcc"), for example::

    cc -o main -DCONFIG_VAR_COOL foo.c bar.c baz.s -lz

:mps:tag:`build.defs` The "defs" are the set of preprocessor macros which are to be
predefined when compiling the module sources::

    CONFIG_VAR_<variety-code>

:mps:tag:`var.codes` The variety codes are as follows:

:mps:tag:`var.hot` :c:macro:`HOT`

    Intended for release in products.  Optimised, reduced internal
    checking, especially on the critical path [RB_2012-09-07]_.

:mps:tag:`var.cool` :c:macro:`COOL`

    Intended for use during development.  Moderately thorough internal
    consistency checking.  Reduced optimisation to allow for
    single-stepping.

:mps:tag:`var.rash` :c:macro:`RASH`

    No internal checking at all.  Slight performance improvement over
    :mps:ref:`.var.hot` at the cost of early detection of memory management
    bugs.  We do not advise use of this variety, as memory management
    bugs tend to be extremely expensive to deal with.

:mps:tag:`default.hot` If no :c:macro:`CONFIG_VAR` is present, :c:macro:`HOT` is assumed in
config.h_.

:mps:tag:`build.srcs` The "srcs" are the set of sources that must be
compiled in order to build the target. The set of sources may vary
depending on the configuration. For example, different sets of sources
may be required to build different architectures.

.. note::

    This is a dependency between the makefile (or whatever) and the
    module configuration in config.h_.

:mps:tag:`build.libs` The "libs" are the set of libraries to which the
compiled sources must be linked in order to build the target. For
example, when building a test program, it might include the ANSI C
library and an operating system interface library.


File Structure
..............

:mps:tag:`file.dir` The MPS source code is arranged in a single directory
called "code" containing all the sources for the whole family of
targets.

:mps:tag:`file.base` The names of sources must be unique in the first eight
characters in order to conform to FAT filesystem naming restrictions.
(Do not scoff -- this has been an important requirement as recently as
2012!)

:mps:tag:`file.ext` The extension may be up to three characters and directly
indicates the source language.

:mps:tag:`file.platform` Platform-specific files include the platform code
in their name.  See :mps:ref:`.mod.impls`.


Modules and naming
..................

:mps:tag:`mod.unique` Each module has an identifier which is unique within the MPS.

:mps:tag:`mod.impls` Each module has one or more implementations which may be
in any language supported by the relevant build environment.

:mps:tag:`mod.primary` The primary implementation of a module is written in
target-independent ANSI C in a source file with the same name as the
module.

:mps:tag:`mod.an` Where there are platform-specific implementations and an
inferior portable ANSI C fallback implementation, "an" is used in
place of the platform code.

:mps:tag:`mod.secondary` The names of other implementations should begin
with the same prefix (the module id or a shortened version of it) and
be suffixed with on or more target parameter codes (defined below). In
particular, the names of assembly language sources must include the
target parameter code for the relevant architecture.

:mps:tag:`mod.example` For example, the stack scanner is defined in ss.h_
(which is platform-independent). It has some platform-independent C in
ss.c_ and, for example, ssw3i6mv.c_ is specific to Windows on the x64
architecture built with Microsoft Visual C.

.. _ss.c: ../../../code/ss.c
.. _ss.h: ../../../code/ss.h
.. _ssw3i6mv.c: ../../../code/ssw3i6mv.c


Build system rationale
......................

:mps:tag:`build.rat` This simple design makes it possible to build the MPS
using many different tools.  Microsoft Visual C and other graphical
development tools do not support much in the way of generated sources,
staged building, or other such stuff.  The Visual C and Xcode "project"
files correspond closely to a closure of the build function
(:mps:ref:`.build.fun`).  The simplicity of the build function has also made it
easy to set up builds using NMAKE (DOS), MPW (Macintosh), and to get the
MPS up and running on other platforms such as FreeBSD and Linux in very
little time.  The cost of maintaining the build systems on these various
platforms is also reduced to a minimum, allowing the MPS developers to
concentrate on primary development.  The source code is kept simple and
straightforward.  When looking at MPS sources you can tell exactly what
is going to be generated with very little context.  The sources are not
munged beyond the standard ANSI C preprocessor.

:mps:tag:`build.port` The portability requirement (:mps:ref:`.req.port`) implies that
the build system must use only standard tools that will be available on
all conceivable target platforms.  Experience of development
environments on the Macintosh (Metrowerks Codewarrior) and Windows NT
(Visual C++) indicates that we cannot assume much sophistication in the
use of file structure by development environments.  The best that we can
hope for is the ability to combine a fixed list of source files,
libraries, and predefined preprocessor symbols into a single target.

:mps:tag:`build.maint` The maintainability requirement (:mps:ref:`.req.maint`) implies
that we don't spend time trying to develop a set of tools to support
anything more complicated than the simple build function described
above.  The effort in constructing and maintaining a portable system of
this kind is considerable. Such efforts failed in the Electronic
Publishing division of Harlequin.


Warnings and errors
...................

:mps:tag:`warning.free` A consequence of :mps:ref:`.import.source` is that the MPS
needs to compile in the context of the client's build system, with
*whatever compilation and warning options* the client has enabled in
that system, and this might include options causing warnings to be
treated as errors. Accordingly, the MPS should compile without
warnings when enabling the compiler options most likely to be employed
by clients.

:mps:tag:`warning.impl` In order to ensure that the MPS meets the
requirement in :mps:ref:`.warning.free`, during development and testing of the
MPS we compile with a large selection of warning options for each
supported compiler, and with warnings treated as errors so that
developers do not get into the habit of ignoring warning messages.
These are enabled in the compiler makefile fragments for each
compiler, for example ll.gmk_ for Clang/LLVM.

.. _ll.gmk: ../../../code/ll.gmk

:mps:tag:`warning.benefit` The implementation in :mps:ref:`.warning.impl` also helps
us keep the code free of subtle compiler issues that break memory
managers, and free of constructs which might be accidentally
mis-interpreted by other developers.

:mps:tag:`warning.silence` When code needs to be modified, for example by
adding a cast, to silence a warning that has been analyzed and turned
out to be harmless, it is best practice to introduce a macro that
expresses the intention, and cross-reference this paragraph from the
macro's comment. If the macro is general-purpose then misc.h_ is a
good place to put it.

.. _misc.h: ../../../code/misc.h


Implementation
--------------

:mps:tag:`impl` The two implementation files config.h_ and mpstd.h_ can be
seen as preprocessor programs which "accept" build parameters and "emit"
configuration parameters (:mps:ref:`.fig.impl`).  The build parameters are
defined either by the builder (in the case of target detection) or by
the build function (in the case of selecting the variety).

:mps:tag:`fig.impl`

=========================== ============== ===========================================
Build parameters            Source file    Configuration parameters
=========================== ============== ===========================================
:c:macro:`CONFIG_VAR_HOT`   ⟶ ``config.h`` ⟶ :c:macro:`MPS_ASSERT_STRING`, etc.
``_WIN32``                  ⟶ ``mpstd.h``  ⟶ :c:macro:`MPS_OS_W3`, etc.
=========================== ============== ===========================================

:mps:tag:`impl.dep` No source code, other than the directives in config.h_
and mpstd.h_, should depend on any build parameters.  That is,
identifiers beginning "CONFIG\_" should only appear in impl.h.config.
Code may depend on configuration parameters in certain, limited ways, as
defined below (:mps:ref:`.conf`).

.. _config.h: ../../../code/config.h
.. _mpstd.h: ../../../code/mpstd.h


Target platform detection
.........................

:mps:tag:`pf` The target platform is "detected" by the preprocessor directives in
mpstd.h_.

:mps:tag:`pf.form` This file consists of sets of directives of the form::

    #elif <conjunction of builder predefinitions>
    #define MPS_PF_<platform code>
    #define MPS_PF_STRING "<platform code>"
    #define MPS_OS_<operating system code>
    #define MPS_ARCH_<architecture code>
    #define MPS_BUILD_<builder code>
    #define MPS_T_WORD     <word type>
    #define MPS_T_ULONGEST <longest unsigned integer type>
    #define MPS_WORD_WIDTH <word width in bits>
    #define MPS_WORD_SHIFT <log to the base 2 of word width>
    #define MPS_PF_ALIGN   <minimum alignment>

:mps:tag:`pf.detect` The conjunction of builder predefinitions is a constant
expression which detects the target platform.  It is a logical AND of
expressions which look for preprocessor symbols defined by the build
environment to indicate the target.  These must be accompanied by a
reference to the build tool documentation from which the symbols came.
For example::

    /* "Predefined Macros" from "Visual Studio 2010" on MSDN
     * <http://msdn.microsoft.com/en-us/library/b0084kay(v=vs.100).aspx>. */

    #elif defined(_MSC_VER) && defined(_WIN32) && defined(_M_IX86)

:mps:tag:`pf.codes` The declarations of the platform, operating system,
architecture, and builder codes define preprocessor macros corresponding
to the target detected (:mps:ref:`.pf.detect`).  For example::

    #define MPS_PF_W3I3MV
    #define MPS_OS_W3
    #define MPS_ARCH_I3
    #define MPS_BUILD_MV

:mps:tag:`pf.word` The declaration of :c:macro:`MPS_T_WORD` defines the unsigned
integral type which corresponds, on the detected target, to the
machine word. It is used to defined the MPS Word type
(design.mps.type.word_). For example::

    #define MPS_T_WORD      unsigned long

.. _design.mps.type.word: type.html#design.mps.type.word

We avoid using ``typedef`` here because mpstd.h_ could potentially
be included in assembly language source code.

:mps:tag:`pf.word-width` The declaration of :c:macro:`MPS_WORD_WIDTH` defines the
number of bits in the type defined by :c:macro:`MPS_T_WORD` (:mps:ref:`.pf.word`) on the
target. For example::

    #define MPS_WORD_WIDTH  32

:mps:tag:`pf.word-shift` The declaration of :c:macro:`MPS_WORD_SHIFT` defines the log
to the base 2 of :c:macro:`MPS_WORD_WIDTH`.  For example::

    #define MPS_WORD_SHIFT  5

:mps:tag:`pf.pf-align` The declaration of :c:macro:`MPS_PF_ALIGN` defines the minimum
alignment which must be used for a memory block to permit any normal
processor memory access.  In other words, it is the maximum alignment
required by the processor for normal memory access.  For example::

    #define MPS_PF_ALIGN    4

:mps:tag:`pf.ulongest` The declaration of :c:macro:`MPS_T_ULONGEST` defines the
longest available unsigned integer type on the platform.  This is
usually just ``unsigned long`` but under Microsoft C on 64-bit Windows
``unsigned long`` is just 32-bits (curse them!)  For example::

    #define MPS_T_ULONGEST      unsigned __int64

:mps:tag:`pf.pf-string` The declaration of :c:macro:`MPS_PF_STRING` defines a
string that is used to identify the target platform in version.c_. For
example::

    #define MPS_PF_STRING   "w3i6mv"

.. _version.c: ../../../code/version.c


Target varieties
................

:mps:tag:`var` The target variety is handled by preprocessor directives in
impl.h.config.

:mps:tag:`var.form` The file contains sets of directives of the form::

    #if defined(CONFIG_VAR_COOL)
    #define CONFIG_ASSERT
    #define CONFIG_ASSERT_ALL
    #define CONFIG_STATS

:mps:tag:`var.detect` The configured variety is one of the variety
preprocessor definitions passed to the build function
(:mps:ref:`.build.defs`), for example, :c:macro:`CONFIG_VAR_COOL`. These are
decoupled in order to keep the number of supported varieties small,
controlling each feature (for example, assertions) by a single
preprocessor definition, and maintaining flexibility about which
features are enabled in each variety.

:mps:tag:`var.symbols` The directives should define whatever symbols are
necessary to control features. These symbols parameterize other parts
of the code, such as the declaration of assertions, etc. The symbols
should all begin with the prefix :c:macro:`CONFIG_`.


Source code configuration
-------------------------

:mps:tag:`conf` This section describes how the configuration may affect the
source code of the MPS.

:mps:tag:`conf.limit` The form of dependency allowed is carefully limited to
ensure that code remains maintainable and portable (:mps:ref:`.req.impact`).

:mps:tag:`conf.min` The dependency of code on configuration parameters should
be kept to a minimum in order to keep the system maintainable
(:mps:ref:`.req.impact`).


Configuration Parameters
........................

:mps:tag:`conf.params` The compilation of a module is parameterized by::

    MPS_ARCH_<arch-code>
    MPS_OS_<os-code>
    MPS_BUILD_<builder-code>
    MPS_PF_<platform-code>


Abstract and Concrete Module Interfaces
.......................................

:mps:tag:`abs.caller` Basic principle: the caller musn't be affected by
configuration of a module. This reduces complexity and dependency of
configuration.  All callers use the same abstract interface.  Caller
code does not change.

:mps:tag:`abs.interface` Abstract interface includes:

- method definitions (logical function prototypes which may be macro methods)
- names of types
- names of constants
- names of structures and fields which form part of the interface, and
  possibly their types, depending on the protocol defined
- the protocols

:mps:tag:`abs.rule` The abstract interface to a module may not be altered by a
configuration parameter.  However, the concrete interface may vary.

For example, this isn't allowed, because there is a change in the interface::

    #if defined(PROT_FOO)
    void ProtSpong(Foo foo, Bar bar);
    #else
    int ProtSpong(Bar bar, Foo foo);
    #endif

This example shows how::

    #ifdef PROTECTION
    void ProtSync(Space space);
    /* more decls. */
    #else /* PROTECTION not */
    #define ProtSync(space) NOOP
    /* more decls. */
    #endif /* PROTECTION */

or::

    #if defined(PROT_FOO)
    typedef struct ProtStruct {
      int foo;
    } ProtStruct;
    #define ProtSpong(prot)  X((prot)->foo)
    #elif defined(PROT_BAR)
    typedef struct ProtStruct {
      float bar;
    } ProtStruct;
    #define ProtSpong(prot)  Y((prot)->bar)
    #else
    #error "No PROT_* configured."
    #endif

Configuration parameters may not be used to vary implementations in C files.
For example, this sort of thing::

    int map(void *base, size_t size)
    {
    #if defined(MPS_OS_W3)
      VirtualAlloc(foo, bar, base, size);
    #elif defined(MPS_OS_SU)
      mmap(base, size, frob);
    #else
    #error "No implementation of map."
    #endif
    }

This violates :mps:ref:`.no-spaghetti`.


Configuration options
---------------------

:mps:tag:`opt` Options select features of the MPS that are not selected by the *platform* and the *variety*.

:mps:tag:`opt.support` The features selected by options are not supported or
documented in the public interface. This is to keep the complexity of
the MPS manageable: at present the number of supported configuration
is *platforms* × *varieties* (at time of writing, 9 × 3 = 27). Each
supported option would double (or worse) the number of supported
configurations.

:mps:tag:`opt.ansi` :c:macro:`CONFIG_PF_ANSI` tells ``mps.c`` to exclude the
sources for the auto-detected platform, and use the generic ("ANSI")
platform instead.

:mps:tag:`opt.thread` :c:macro:`CONFIG_THREAD_SINGLE` causes the MPS to be built
for single-threaded execution only, where locks are not needed and so
the generic ("ANSI") lock module ``lockan.c`` can be used instead of
the platform-specific lock module.

:mps:tag:`opt.poll` :c:macro:`CONFIG_POLL_NONE` causes the MPS to be built without
support for polling. This means that garbage collections will only
happen if requested explicitly via :c:func:`mps_arena_collect()` or
:c:func:`mps_arena_step()`, but it also means that protection is not needed,
and so shield operations can be replaced with no-ops in ``mpm.h``.

:mps:tag:`opt.signal.suspend` :c:macro:`CONFIG_PTHREADEXT_SIGSUSPEND` names the
signal used to suspend a thread, on platforms using the POSIX thread
extensions module. See design.pthreadext.impl.signals_.

.. _design.pthreadext.impl.signals: pthreadext#impl.signals

:mps:tag:`opt.signal.resume` :c:macro:`CONFIG_PTHREADEXT_SIGRESUME` names the
signal used to resume a thread, on platforms using the POSIX thread
extensions module. See design.pthreadext.impl.signals_.


To document
-----------
- What about constants in config.h?
- Update files to refer to this design document.
- Explain the role of ``mps.c``
- Reference to ``build.txt``
- Procedures for adding an architecture, etc.
- Reduce duplication in this document (especially after
  `Configuration Parameters`_ which looks like it's been pasted in from
  elsewhere.)


References
----------

.. [RB_2012-09-07] Richard Brooksby. Ravenbrook Limited. 2012-09-07. "`The critical path through the MPS <https://www.ravenbrook.com/project/mps/master/design/critical-path>`__".

.. [RB_2012-09-13] Richard Brooksby. Ravenbrook Limited. 2013-09-13. "`The Configura CET custom mainline <https://info.ravenbrook.com/mail/2012/09/13/16-43-35/0/>`__".