Release notes

Release 1.117.0

New features

1. On FreeBSD, Linux and macOS, the MPS is now able to run in the child process after fork(). See Fork safety.

2. The MPS now supports Windows Vista or later; it no longer supports Windows XP. (Microsoft’s own support for Windows XP expired in April 2014.) This is so that we can use InitOnceExecuteOnce() to ensure thread-safe initialization.

Interface changes

1. The pool class MV (Manual Variable) is now deprecated.

Other changes

1. References from the MPS’s own stack frames no longer pin objects allocated by the client program in moving pools, which prevented them from moving. See job003525.

2. Creation of arenas is now thread-safe on Windows. See job004056.

3. AWL (Automatic Weak Linked) and LO (Leaf Object) pools now detect (and assert on) invalid exact references. See job004070.

4. The MPS now compiles without warnings on GCC version 7 with -Wextra. See job004076.

5. Deprecated function mps_arena_roots_walk() no longer causes mps_arena_formatted_objects_walk() to miss some objects. See job004090.

Release 1.116.0

New features

1. The MPS now measures the mortality of a generation each time it is collected, and maintains a moving average. This means that it is no longer important to provide an accurate estimate of the mortality when creating a generation chain by calling mps_chain_create().

2. The MPS no longer supports Linux 2.4 and 2.5. (These versions used LinuxThreads instead of POSIX threads; all major distributions have long since ceased to support these versions and so it is no longer convenient to test against them.) See Supported target platforms.

3. New function mps_arena_postmortem() assists with postmortem debugging.

4. New function mps_arena_busy() assists debugging of re-entry errors in dynamic function table callbacks on Windows on x86-64.

Interface changes

1. The pool class SNC (Stack No Checking) is no longer deprecated.

2. Allocation frames are no longer deprecated. See Allocation frames.

3. On Linux and FreeBSD, it is now possible to configure the signals used to suspend and resume threads. See Signal and exception handling issues.

Other changes

1. It is now possible to register a thread with the MPS multiple times on OS X, thus supporting the use case where a program that does not use the MPS is calling into MPS-using code from multiple threads. (This was already supported on other platforms.) See job003559.

2. The function mps_arena_formatted_objects_walk() walks the formatted objects in all pools. Previously this was not implemented for AMS (Automatic Mark and Sweep) pools. See job003738.

3. Objects in SNC (Stack No Checking) pools are no longer scanned after their allocation frame is popped, and so do not keep objects in automatically managed pools alive. See job003883.

4. When the MPS collects a set of generations, it condemns only the blocks in those generations. Previously, it also condemned blocks that happened to share a region of memory with blocks currently or formerly allocated in those generations. See job004000.

5. Memory in allocation points no longer contributes to the decision to start a garbage collection, avoiding wasted work repeatedly collecting generations with very small capacities. See job004007.

6. The MPS no longer considers collecting the world again, without allowing the client program to run first. See job004011.

7. Roots created by mps_root_create_thread_scanned() no longer cause an assertion failure. See job004036.

8. The MPS test suite now compiles and passes with GCC 6.1. See job004037.

9. The MPS no longer passes an uninitialized variable to thread_swap_exception_ports() on OS X. See job004040.

Release 1.115.0

New features

1. The MPS now provides control over the maximum time that operations within an arena may pause the client program for. This can be specified by the new function mps_arena_pause_time_set() or by passing the new keyword argument MPS_KEY_PAUSE_TIME to mps_arena_create_k(). The current value can be retrieved by the new function mps_arena_pause_time().

   The maximum pause time defaults to 0.1 seconds. For the old behaviour (whereby the MPS always returned to the client program as soon as possible), set it to zero.

2. New supported platforms fri3ll (FreeBSD, IA-32, Clang/LLVM) and fri6ll (FreeBSD, x86-64, Clang/LLVM).

3. When creating an AMC (Automatic Mostly-Copying) pool, mps_pool_create_k() accepts the new keyword argument MPS_KEY_EXTEND_BY, specifying the minimum size of the memory segments that the pool requests from the arena.

4. The function mps_arena_create_k() accepts two new keyword arguments. MPS_KEY_COMMIT_LIMIT sets the commit limit for the arena, and MPS_KEY_SPARE_COMMIT_LIMIT sets the spare commit limit for the arena.

5. New area scanning functions mps_scan_area(), mps_scan_area_masked(), mps_scan_area_tagged(), mps_scan_area_tagged_or_zero() for use when scanning, especially when scanning threads and tagged references.

6. New thread root functions mps_root_create_thread(), mps_root_create_thread_tagged(), and mps_root_create_thread_scanned() allow flexible scanning of thread stacks and registers in any format, with convenient implementations provided for tagged references.

7. New function mps_root_create_table_tagged() for tables of roots containing tagged references.

8. New area root functions mps_root_create_area() and mps_root_create_area_tagged() for areas of memory that can be scanned by area scanning functions.

Interface changes

1. The pool class MV (Manual Variable) is no longer deprecated.

2. The type of pool classes is now mps_pool_class_t. The old name mps_class_t is still available via a typedef, but is deprecated.

3. The functions mps_mv_free_size(), mps_mv_size(), mps_mvff_free_size(), mps_mvff_size(), mps_mvt_free_size() and mps_mvt_size() are now deprecated in favour of the generic functions mps_pool_free_size() and mps_pool_total_size().

4. The function mps_root_create_reg() is deprecated in favour of mps_root_create_thread_tagged().

5. The function mps_root_create_table_masked() is deprecated in favour of mps_root_create_table_tagged().

6. The SNC (Stack No Checking) pool class now implements mps_pool_total_size() and mps_pool_free_size().

7. The (undocumented) reservoir functions mps_ap_fill_with_reservoir_permit(), mps_reservoir_available(), mps_reservoir_limit(), mps_reservoir_limit_set(), and mps_reserve_with_reservoir_permit(), together with the has_reservoir_permit arguments to mps_sac_alloc() and MPS_SAC_ALLOC_FAST() are now deprecated.

Other changes

1. mps_arena_committed() now returns a meaningful value (the amount of memory marked as in use in the page tables) for client arenas. See job001887.

2. AMC (Automatic Mostly-Copying) pools now assert that exact references into the pool are aligned to the pool’s alignment. See job002175.

3. Internal calculation of the address space available to the MPS no longer takes time proportional to the number of times the arena has been extended, speeding up allocation when memory is tight. See job003814.

4. Setting MPS_KEY_SPARE for a MVFF (Manual Variable First Fit) pool now works. See job003870.

5. In the hot (production) variety, mps_pool_free_size() now returns the correct result for AWL (Automatic Weak Linked) and LO (Leaf Object) pools. See job003884.

6. When the arena is out of memory and cannot be extended without hitting the commit limit, the MPS now returns MPS_RES_COMMIT_LIMIT rather than substituting MPS_RES_RESOURCE. See job003899.

7. Unfinalizable objects can no longer be registered for finalization. Previously the objects would be registered but never finalized. See job003865.

8. mps_arena_has_addr() now returns the correct result for objects allocated from the MFS (Manual Fixed Small), MV (Manual Variable), and MVFF (Manual Variable First Fit) pools. See job003866.

9. The MPS can now make use of spare committed memory even if it is mapped at an unhelpful address, by unmapping it and remapping at a better address. See job003898.

10. mps_arena_step() now always considers starting a new garbage collection if the remaining idle time is long enough to complete it. (Previously, if there was already a collection in progress when mps_arena_step() was called, it would finish the collection but not consider starting a new one.) See job003934.

11. The MPS no longer carries out garbage collections when there is no collection work to be done. See job003938.

12. The MPS is less aggressive in its use of hardware memory protection to maintain write barrier to speed up future collections. This is particularly important for OS X, where memory protection operations are very expensive. See job003371 and job003975.

13. The MPS coalesces memory protection, reducing the number of system calls. This markedly improves real run time on operating systems where memory protection operations are very expensive, such as OS X, but also has a significant effect on Linux. See job003371 and job003975.

Release 1.114.0

New features

1. Ambiguous interior pointers now keep objects in AMC (Automatic Mostly-Copying) and AMCZ (Automatic Mostly-Copying Zero-rank) pools alive.

   This means that if the compiler optimizes away a pointer to the base of an object, leaving an interior pointer as the only reference keeping the object alive, this does not cause the object to be incorrectly collected. Or, if you are writing your own compiler, you can now perform such an optimization safely.

   If you require the old behaviour (in which ambiguous interior pointers were ignored) then you can set the MPS_KEY_INTERIOR keyword argument to FALSE when calling mps_pool_create_k().

2. The logic for deciding which generations should be collected has changed. Now, a chain may be scheduled for collection if the new size of any of its generations exceeds its capacity, and when a chain is collected, all generations are collected up to, and including, the highest generation whose new size exceeds its capacity. This ensures that all generations are collected reliably on chains where there is no allocation into the nursery generation. See Scheduling of collections.

   (Previously, only the nursery generation in each chain was considered, and a chain was collected up to, but not including, the lowest generation whose new size was within its capacity.)

   As a result of this change, we recommend that you retune your generation sizes. (This is not necessary, but may improve performance.)

3. New pool introspection functions mps_pool_free_size() and mps_pool_total_size().

Interface changes

1. The granularity with which the arena manages memory can now be specified using the MPS_KEY_ARENA_GRAIN_SIZE keyword argument to mps_arena_create_k(). See mps_arena_class_cl() and mps_arena_class_vm().

2. There is now a default value (currently 256 megabytes) for the MPS_KEY_ARENA_SIZE keyword argument to mps_arena_create_k() when creating a virtual memory arena. See mps_arena_class_vm().

3. The keyword argument MPS_KEY_AMS_SUPPORT_AMBIGUOUS now defaults to TRUE in order to better support the general case: the value FALSE is appropriate only when you know that all references are exact. See AMS (Automatic Mark and Sweep).

4. There is now a default value for the MPS_KEY_AWL_FIND_DEPENDENT keyword argument to mps_pool_create_k() when creating an AWL (Automatic Weak Linked) pool. The default value is a function that always returns NULL (meaning that there is no dependent object).

5. It is now possible to configure the alignment of objects allocated in an MV (Manual Variable) pool, by passing the MPS_KEY_ALIGN keyword argument to mps_pool_create_k().

6. The MVFF (Manual Variable First Fit) pool class takes a new keyword argument MPS_KEY_SPARE. This specifies the maximum proportion of memory that the pool will keep spare for future allocations.

7. The alignment requirements for MVFF (Manual Variable First Fit) and MVT (Manual Variable Temporal) pools have been relaxed on the platforms w3i3mv and w3i6mv. On all platforms it is now possible to specify alignments down to sizeof(void*) as the alignment for pools of these classes.

8. The sizes of the templates in a mps_pool_debug_option_s structure no longer have to be related to the alignment of the pools that they are used with. This makes it easier to reuse these structures.

Other changes

1. The AMS (Automatic Mark and Sweep) pool class no longer triggers the assertion !AMS_IS_INVALID_COLOUR(seg, i) under rare circumstances (namely, detaching an allocation point from a grey segment when MPS_KEY_AMS_SUPPORT_AMBIGUOUS is FALSE). See job001549.

2. mps_arena_roots_walk() no longer triggers an assertion failure when run twice in succession. See job003496.

3. The alignment of AWL (Automatic Weak Linked) pools is now configurable via the object format, as documented, and is no longer always MPS_PF_ALIGN. See job003745.

4. The debugging version of the MVFF (Manual Variable First Fit) pool class, mps_class_mvff_debug(), no longer triggers an assertion failure if you allocate a large object. See job003751.

5. mpseventtxt now successfully processes a telemetry log containing multiple labels associated with the same address. See job003756.

6. AMS (Automatic Mark and Sweep), AWL (Automatic Weak Linked) and LO (Leaf Object) pools get reliably collected, even in the case where the pool is the only pool on its generation chain and is allocating into some generation other than the nursery. See job003771.

7. Allocation into AWL (Automatic Weak Linked) pools again reliably provokes garbage collections of the generation that the pool belongs to. (In version 1.113, the generation would only be collected if a pool of some other class allocated into it.) See job003772.

8. All unreachable objects in LO (Leaf Object) pools are finalized. (Previously, objects on a segment attached to an allocation point were not finalized until the allocation point was full.) See job003773.

9. The MVT (Manual Variable Temporal) and MVFF (Manual Variable First Fit) pool classes are now around 25% faster (in our benchmarks) than they were in version 1.113.

10. The default assertion handler in the default plinth now flushes the telemetry stream before aborting. See mps_lib_assert_fail().

11. Garbage collection performance is substantially improved in the situation where the arena has been extended many times. Critical operations now take time logarithmic in the number of times the arena has been extended (rather than linear, as in version 1.113 and earlier). See job003554.

Release 1.113.0

New features

1. In previous releases there was an implicit connection between blocks allocated by AWL (Automatic Weak Linked) and LO (Leaf Object) pools, and blocks allocated by other automatically managed pool classes.

   In particular, blocks allocated by AWL and LO pools were garbage collected together with blocks allocated by AMS (Automatic Mark and Sweep) pools, and blocks allocated by AMC (Automatic Mostly-Copying) pools in generation 1 of their chains.

   This is no longer the case: to arrange for blocks to be collected together you need to ensure that they are allocated in the same generation chain, using the MPS_KEY_CHAIN and MPS_KEY_GEN keyword arguments to mps_pool_create_k().

   So if you have code like this:

   res = mps_pool_create(&my_amc, arena, mps_class_amc(), my_chain);
   res = mps_pool_create(&my_awl, arena, mps_class_awl());
   

   and you want to retain the connection between these pools, then you must ensure that they use the same generation chain:

   MPS_ARGS_BEGIN(args) {
     MPS_ARGS_ADD(args, MPS_KEY_CHAIN, my_chain);
     res = mps_pool_create_k(&my_amc, arena, mps_class_amc(), args);
   } MPS_ARGS_END(args);
   
   MPS_ARGS_BEGIN(args) {
     MPS_ARGS_ADD(args, MPS_KEY_CHAIN, my_chain);
     MPS_ARGS_ADD(args, MPS_KEY_GEN, 1);
     res = mps_pool_create_k(&my_awl, arena, mps_class_awl(), args);
   } MPS_ARGS_END(args);
   

Interface changes

1. When creating a list of keyword arguments, there is no longer any need to call MPS_ARGS_DONE(). See Keyword arguments.

2. When creating an automatically managed pool using mps_pool_create_k(), it is no longer necessary to pass in a generation chain. The arena has a default generation chain and this is used by all automatically managed pools where no generation chain was specified.

3. It is now possible to specify a generation chain for AWL (Automatic Weak Linked) and LO (Leaf Object) pool classes, by using the optional MPS_KEY_CHAIN keyword argument to mps_pool_create_k().

4. It is now possible to specify which generation the AMS (Automatic Mark and Sweep), AWL (Automatic Weak Linked), and LO (Leaf Object) pool classes allocate new objects into, using the optional MPS_KEY_GEN keyword argument to mps_pool_create_k().

Other changes

1. The MPS now retains some unused memory instead of returning it to the operating system. This reduces unnecessary overhead due to system calls, thrashing the operating system’s page table, and zeroing memory when re-allocated. See job003700.

Release 1.112.0

New features

1. New supported platform lii6ll (Linux, x86-64, Clang/LLVM).

2. On Windows, you can now request that the MPS allocate address space from the top down, allowing a 32-bit executable linked with /LARGEADDRESSAWARE to use the top half of the address space. Use the keyword argument MPS_KEY_VMW3_TOP_DOWN when creating an arena of class mps_arena_class_vm().

3. On OS X, multi-threaded programs are now supported. See Threads.

4. On OS X, you can now debug the MPS using lldb.

Interface changes

1. In the hot (production) variety, the default assertion handler now prints messages to standard error but does not terminate the program. Even though assertions indicate serious problems in the program, an end-user does not always want an application to terminate when there is a chance to shut down safely and save work, or even to limp along indefinitely. See Assertion handling.

2. The behaviour when an assertion is triggered is now configurable in the default plinth by installing an assertion handler. See mps_lib_assert_fail_install().

3. Functions that take a variable number of arguments (mps_arena_create(), mps_pool_create(), mps_ap_create()) and their va_list alternatives (mps_arena_create_v() etc.) are now deprecated in favour of functions that use a keyword argument interface (mps_arena_create_k(), mps_pool_create_k(), mps_ap_create_k()).

   Similarly, the object format variant structures (mps_fmt_A_s etc.) and the functions that take them as arguments (mps_fmt_create_A() etc.) are now deprecated in favour of mps_fmt_create_k().

   The new interfaces provide better reporting of errors, default values for arguments, and forward compatibility. See Keyword arguments.

   The old interfaces continue to be supported for now, but new features will become available through the keyword interface only.

4. MFS (Manual Fixed Small) pools no longer refuse to manage blocks that are smaller than the platform alignment. They now round up smaller sizes internally if necessary.

5. MVT (Manual Variable Temporal) pools now allow the client to specify the alignment of blocks. Use the keyword argument MPS_KEY_ALIGN when creating a pool of class mps_class_mvt().

6. On OS X, signals are no longer used for handling memory protection exceptions. This means that programs are free to handle SIGBUS, but must not install a thread-local Mach exception handler for EXC_BAD_ACCESS exceptions. See Signal and exception handling issues.

7. On OS X, when debugging with gdb, you no longer need to turn on dont-handle-bad-access or to request special handling of SIGBUS.

Other changes

1. On Windows, an execute exception no longer triggers an assertion. See job003301.

2. Rehashing of large address-based hash tables no longer provokes a nursery collection that immediately renders the hash table stale again. See job003435.

3. An MVT (Manual Variable Temporal) pool no longer triggers an assertion failure when it runs out of space on its reserved block queue. See job003486.

4. The -i and -o options no longer cause mpseventsql to crash. See job003507.

5. On Windows, telemetry files now have correct clock values. Previously the top 32 bits were incorrectly output as zero. See job003519.

6. On 64-bit Windows, it’s no longer possible to get a stack overflow exception while the MPS is holding the arena lock. See job003640.

Release 1.111.0

New features

1. Reporting features have been removed from the mpseventcnv utility. Instead, the telemetry system comes with two new utility programs to assist with reporting and analysis: mpseventtxt converts an event stream into human-readable form, and mpseventsql loads an event stream into a SQLite database for further analysis. See Telemetry.

2. The new pool class MFS (Manual Fixed Small) provides manually managed allocation of fixed-size objects.

3. The new pool class MVT (Manual Variable Temporal) provides manually managed allocation of variable-size objects using a temporal fit allocation policy (that is, objects that are allocated togther are expected to be freed together).

Interface changes

1. It is no longer necessary for client programs to use mps_tramp() to ensure that exceptions due to barrier hits are caught. This function is now deprecated.

2. You can set the environment variable MPS_TELEMETRY_CONTROL to all to make the telemetry system output all events. See Telemetry.

3. New functions mps_telemetry_get(), mps_telemetry_set() and mps_telemetry_reset() provide a more convenient interface to telemetry control than mps_telemetry_control(), which is now deprecated. See Telemetry.

4. The pool classes MV (Manual Variable) and SNC (Stack No Checking) are now deprecated.

5. Allocation frames are now deprecated. See Allocation frames.

6. Additionally, the functions mps_arena_expose(), mps_arena_unsafe_expose_remember_protection(), mps_arena_unsafe_restore_protection(), mps_arena_roots_walk(), and mps_fix() are now deprecated.

Other changes

1. mps_arena_step() no longer unclamps the arena as a side effect. If the arena is clamped or parked before calling mps_arena_step(), it is clamped afterwards. See job003320.

2. The ambiguous stack scanner, mps_stack_scan_ambig(), no longer asserts on Linux when there are multiple threads. See job003412.

3. It is no longer possible for the “ramp” allocation pattern, mps_alloc_pattern_ramp(), to get stuck. Now mps_ap_alloc_pattern_end() reliably clears this pattern. See job003454.

4. The build system now correctly detects the FreeBSD operating system running on the x86-64 architecture, for FreeBSD version 9.1 or later. See job003473.

Release 1.110.0

New features

1. New supported platforms:

   • fri6gc (FreeBSD, x86-64, GCC)

   • lii6gc (Linux, x86-64, GCC)

   • w3i6mv (Windows, x86-64, Microsoft Visual C)

   • xci3ll (OS X, IA-32, Clang/LLVM)

   • xci6gc (OS X, x86-64, GCC)

   • xci6ll (OS X, x86-64, Clang/LLVM)

2. Support removed for platforms:

   • iam4cc (Irix 6, MIPS R4000, MIPSpro C)

   • lii3eg (Linux, IA-32, EGCS)

   • lippgc (Linux, PowerPC, GCC)

   • o1alcc (OSF/1, Alpha, Digital C)

   • o1algc (OSF/1, Alpha, GCC)

   • s7ppmw (System 7, PowerPC, MetroWerks C)

   • sos8gc (Solaris, SPARC 8, GCC)

   • sos9sc (Solaris, SPARC 9, SunPro C)

   • sus8gc (SunOS, SPARC 8, GCC)

   • xcppgc (OS X, PowerPC, GCC)

3. On Unix platforms, the MPS can now be built and installed by running ./configure && make install. See Building the Memory Pool System.

4. The MPS can be compiled in a single step via the new source file mps.c. This also allows you to compile the MPS in the same compilation unit as your object format, allowing the compiler to perform global optimizations between the two. See Building the Memory Pool System.

5. The set of build varieties has been reduced to three: the cool variety for development and debugging, the hot variety for production, and the rash variety for people who like to live dangerously. See Varieties.

6. The environment variable MPS_TELEMETRY_CONTROL can now be set to a space-separated list of event kinds. See Telemetry.

7. Telemetry output is now emitted to the file named by the environment variable MPS_TELEMETRY_FILENAME, if it is set. See Telemetry.

Interface changes

1. Deprecated constants MPS_MESSAGE_TYPE_FINALIZATION, MPS_MESSAGE_TYPE_GC and MPS_MESSAGE_TYPE_GC_START have been removed. Use mps_message_type_finalization(), mps_message_type_gc() and mps_message_type_gc_start() instead.

2. Deprecated constants MPS_RANK_AMBIG, MPS_RANK_EXACT and MPS_RANK_WEAK have been removed. Use mps_rank_ambig(), mps_rank_exact() and mps_rank_weak() instead.

3. Deprecated functions with names starting mps_space_ have been removed. Use the functions with names starting mps_arena_ instead.