23. Porting the MPS

This chapter lists the steps involved in porting the MPS to a new operating system, processor architecture, or compiler. It assumes that you are familiar with Building the Memory Pool System and the Platforms chapter.

23.1. Platform code

Pick two-character codes for the new platform’s operating system, processor architecture, and compiler toolchain, as described under Platforms, and concatenate them to get a six-character platform code “osarct”.

23.2. Functional modules

The MPS requires platform-specific implementations of the functional modules in the list below. You’ll probably find that it’s unnecessary to port them all: unless the new platform is very exotic, some of the existing implementations ought to be usable. In most cases there is a generic (“ANSI”) implementation of the module, that uses only the features of the Standard C Library. These generic implementations are partially functional or non-functional, but can be used as a starting point for a new port if none of the existing implementations is usable.

  1. The clock module provides fast high-resolution clocks for use by the telemetry system.

    See Fast high-resolution clock for the design, and clock.h for the interface. The interface consists only of type declarations and macro definitions, so there is no implementation.

    The header falls back to the clock functions from the plinth if there is no platform-specific interface. See mps_clock() and mps_clocks_per_sec().

  2. The lock module provides binary locks that ensure that only a single thread may be running with a lock held, and recursive locks, where the same thread may safely take the lock again without deadlocking.

    See Lock module for the design, and lock.h for the interface. There are implementations for Linux in lockli.c, POSIX in lockix.c, and Windows in lockw3.c.

    There is a generic implementation in lockan.c, which cannot actually take any locks and so only works for a single thread.

  3. The memory protection module applies protection to areas of memory (2), ensuring that attempts to read or write from those areas cause protection faults, and implements the means for the MPS to catch and handle these faults.

    See Memory protection for the design, and prot.h for the interface. There are implementations for POSIX in protix.c plus protsgix.c, Linux in protli.c, Windows in protw3.c, and OS X using Mach in protxc.c.

    There is a generic implementation in protan.c, which can’t provide memory protection, so it forces memory to be scanned until that there is no further need to protect it. This means it can’t support incremental collection, and has no control over pause times.

  4. The protection mutator context module figures out what the mutator was doing when it caused a protection fault, so that access to a protected region of memory can be handled, or when a thread was suspended, so that its registers and control stack can be scanned.

    See Protection mutator context for the design, and prot.h for the interface. There are implementations on Unix, Windows, and OS X for IA-32 and x86-64.

    There is a generic implementation in prmcan.c, which can’t provide these features, and so only supports a single thread.

  5. The stack probe module checks that there is enough space on the control stack for the MPS to complete any operation that it might start. The purpose is to provoke a stack overflow exception, if necessary, before taking the arena lock.

    See Stack probe for the design, and sp.h for the interface. There are implementations on Windows on IA-32 in spi3w3.c and x86-64 in spi6w3.c.

    There is a generic implementation in span.c, which can’t provide this feature, and so is only suitable for use with a client program that does not handle stack overflow faults, or does not call into the MPS from the handler.

  6. The stack and register scanning module scans the registers and control stack of a thread.

    See Stack and register scanning for the design, and ss.h for the interface. There are implementations for POSIX on IA-32 in ssixi3.c and x86-64 in ssixi6.c, and for Windows with Microsoft Visual C/C++ on IA-32 in ssw3i3mv.c and x86-64 in ssw3i6mv.c.

    There is a generic implementation in ssan.c, which calls setjmp() to spill the registers and scans the whole jump buffer, thus overscanning compared to a platform-specific implementation.

  7. The thread manager module suspends and resumes threads, so that the MPS can gain exclusive access to memory (2), and so that it can scan the registers and control stack of suspended threads.

    See Thread manager for the design, and th.h for the interface. There are implementations for POSIX in thix.c plus pthrdext.c, OS X using Mach in thxc.c, Windows in thw3.c.

    There is a generic implementation in than.c, which necessarily only supports a single thread.

  8. The virtual mapping module reserves address space from the operating system (and returns it), and maps address space to main memory (and unmaps it).

    See Virtual mapping for the design, and vm.h for the interface. There are implementations for POSIX in vmix.c, and Windows in vmw3.c. There is a generic implementation in vman.c, which fakes virtual memory by calling malloc().

23.3. Platform detection

The new platform must be detected in mpstd.h and preprocessor constants like MPS_WORD_WIDTH defined. See MPS Configuration for the design of this header, and Platform interface for the list of preprocessor constants that may need to be defined. For example:

/* "Predefined Macros" from "Visual Studio 2010" on MSDN
 * <http://msdn.microsoft.com/en-us/library/b0084kay(v=vs.100).aspx>.
 * Note that Win32 includes 64-bit Windows!
 * We use the same alignment as MS malloc: 16, which is used for XMM
 * operations.
 * See MSDN -> x64 Software Conventions -> Overview of x64 Calling Conventions
 * <http://msdn.microsoft.com/en-us/library/ms235286>
 */

#elif defined(_MSC_VER) && defined(_WIN32) && defined(_WIN64) && defined(_M_X64) && !defined(__POCC__)
#if defined(CONFIG_PF_STRING) && ! defined(CONFIG_PF_W3I6MV)
#error "specified CONFIG_PF_... inconsistent with detected w3i6mv"
#endif
#define MPS_PF_W3I6MV
#define MPS_PF_STRING   "w3i6mv"
#define MPS_OS_W3
#define MPS_ARCH_I6
#define MPS_BUILD_MV
#define MPS_T_WORD      unsigned __int64
#define MPS_T_ULONGEST  unsigned __int64
#define MPS_WORD_WIDTH  64
#define MPS_WORD_SHIFT  6
#define MPS_PF_ALIGN    16

The comment should justify the platform test (with reference to documentation or to the output of a command like gcc -E -dM), and explain any unusual definitions. For example, here we need to explain the choice of 16 bytes for MPS_PF_ALIGN, since normally a 64-bit platform requires 8-byte alignment.

23.4. Platform configuration

The new platform may be configured, if necessary, in config.h. See MPS Configuration for the design of this header. Avoid platform-specific configuration if possible, to reduce the risk of errors being introduced on one platform and not detected when other platforms are tested.

23.5. Module selection

In mps.c, add a section for the new platform. This must test the platform constant MPS_PF_OSARCT that is now defined in mpstd.h, and then include all the module sources for the platform. For example:

/* Linux on 64-bit Intel with GCC or Clang */

#elif defined(MPS_PF_LII6GC) || defined(MPS_PF_LII6LL)

#include "lockli.c"     /* Linux locks */
#include "thix.c"       /* Posix threading */
#include "pthrdext.c"   /* Posix thread extensions */
#include "vmix.c"       /* Posix virtual memory */
#include "protix.c"     /* Posix protection */
#include "protli.c"     /* Linux protection */
#include "proti6.c"     /* 64-bit Intel mutator context */
#include "prmci6li.c"   /* 64-bit Intel for Linux mutator context */
#include "span.c"       /* generic stack probe */
#include "ssixi6.c"     /* Posix on 64-bit Intel stack scan */

23.6. Makefile

Add a makefile even if you expect to use an integrated development environment (IDE) like Visual Studio or Xcode. Makefiles make it easier to carry out continuous integration and delivery, and are less likely to stop working because of incompatibilities between IDE versions.

On Unix platforms, the makefile must be named osarct.gmk, and must define PFM to be the platform code, MPMPF to be the list of platform modules (the same files included by mps.c), and LIBS to be the linker options for any libraries required by the test cases. Then it must include the compiler-specific makefile and comm.gmk. For example, lii6ll.gmk looks like this:

PFM = lii6ll

MPMPF = \
    lockli.c \
    prmci6li.c \
    proti6.c \
    protix.c \
    protli.c \
    pthrdext.c \
    span.c \
    ssixi6.c \
    thix.c \
    vmix.c

LIBS = -lm -lpthread

include ll.gmk
include comm.gmk

If the platform needs specific compilation options, then define PFMDEFS accordingly, but avoid this if at all possible. We recommend in Building the Memory Pool System that users compile the MPS using a simple command like cc -c mps.c, and we suggest that they can improve performance by compiling the MPS and their object format in the same compilation unit. These steps would be more complicated if the MPS required particular compilation options.

On Windows, the makefile must be named osarct.nmk, and must define PFM to be the platform code, and MPMPF to be the list of platform modules (the same files included by mps.c) in square brackets. Then it must include commpre.nmk, the compiler-specific makefile and commpost.nmk. For example, w3i6mv.nmk looks like this:

PFM = w3i6mv

MPMPF = \
    [lockw3] \
    [mpsiw3] \
    [prmci6w3] \
    [proti6] \
    [protw3] \
    [spw3i6] \
    [ssw3i6mv] \
    [thw3] \
    [thw3i6] \
    [vmw3]

!INCLUDE commpre.nmk
!INCLUDE mv.nmk
!INCLUDE commpost.nmk

23.7. Porting strategy

Start the port by selecting existing implementations of the functional modules, using the generic implementations where nothing else will do. Then check that the “smoke tests” pass, by running:

make -f osarct.gmk testrun    # Unix
nmake /f osarct.nmk testrun   # Windows

Most or all of the test cases should pass at this point. If you’re using the generic threading implementation, then the multi-threaded test cases are expected to fail. If you’re using the generic lock implementation, then the lock utilization test case lockut is expected to fail. If you’re using the generic memory protection implementation, all the tests that rely on incremental collection are expected to fail. See tool/testcases.txt for a database of test cases and the configurations in which they are expected to pass.

Now that there is a working system to build on, porting the necessary modules to the new platform can be done incrementally. It’s a good idea to measure the performance as you go along (for example, using the gcbench benchmark) to check that the new memory protection module is effective.

23.8. Update the documentation

These sections of the manual should be updated to mention the new platform:

In addition, if aspects of the port were especially tricky, then consider writing a design document (see Design) justifying the implementation.

23.9. Contribute

Consider contributing the new platform to the MPS. See Contributing to the MPS.