| Title | MPS virtual memory arena class never returns virtual memory |
| Status | closed |
| Priority | essential |
| Assigned user | Richard Kistruck |
| Organization | Ravenbrook |
| Description | MPS virtual memory arena class never returns virtual memory If a client makes lots of objects, then they die and are collected, MPS returns the unused committed memory (the RAM and backing store (swapfile) that was used to actually store the data), but does not return vmem (virtual memory, that is: address space). If virtual memory is a constrained resource -- increasingly the case for programs that must fit within a 4GB address-space -- then MPS's failure to return it may cause problems for non-MPS parts of the client program. |
| Analysis | RHSK 2010-03-23 First improvement is "simple-chunk-return": at traceReclaim, look at all chunks and destroy those which are empty. Trace achieves this by calling a new arena-class method "compact". Additional work (see mps/brach/2010-03-11/vmem): ChunkCache was broken with regard to destroying chunks. Work on zcoll, and rnd()/rnd_state(), to test & demonstrate chunk-return. |
| How found | unknown |
| Evidence | customer |
| Observed in | 1.109.0 |
| Created by | Richard Kistruck |
| Created on | 2010-03-23 05:04:40 |
| Last modified by | Richard Kistruck |
| Last modified on | 2010-03-24 11:21:36 |
| History | 2010-03-23 RHSK Created. |
| Change | Effect | Date | User | Description |
|---|---|---|---|---|
| 170102 | closed | 2010-03-23 08:31:39 | Richard Kistruck | MPS master (integ from br/vmem): simple-chunk-return: New method arena->class->compact: ArenaCompact, ArenaTrivCompact, VMCompact [arena.c, arenavm.c, mpm.h, mpmst.h, mpmtypes.h] trace.c, arenavm.c: - call ArenaCompact() at traceReclaim; - also record trace->preTraceArenaReserved, to show pre- and peak-vmem during a collection arenavm.c: - on VMCompact(), destroy any empty chunks (except the primary); - emit VMCompact diag for all client-requested collections, plus any others where chunks were gained or lost during the collection; - VMCompact diag shows vmem: pre-collection, peak, amount returned, and final - VMCompact diag also shows trace stats: cond, live / % / stuck(pip), notCond - M_whole, M_frac: print counts of bytes as Megabytes and thousandths (also, in VMArenaFinish, null out arena->primary in VMArenaFinish, so it is not left dangling). mpm.c: new "$3" format for 0-padding 3-char-wide field, for thousandths of a MB tract.c -- fix ChunkCache defects: - previously, if cache is empty (chunkCache->chunk == NULL) then other fields are *undefined*; but code looks at them anyway (!) without first checking chunkCache->chunk; - change it (.chunk.empty.fields) so that, if cache is empty, other fields have defined values: cache-using code may look at them, and they are chosen so that no cache hit will occur. --> this fixes crashing defect shown by changelist 170072 - AVERT(ChunkCache) in the many places it should be checked; - use AVERT_CRITICAL in ChunkEncache, because it is called by ChunkOfAddr; - drop never-read chunkCache->pageTableBase and pageTableLimit fields: they were used for ChunkOfSeg(), back when each SegStruct was actually a PageStruct is some chunk's PageStructTable; see VMArenaChunkOfSeg() in //info.ravenbrook.com/project/mps/branch/2001-08-13/trunk/src/arenavm.c#46- there's no need for arena to initialise the chunk cache; this allows en/decache functions to be local to tract.c (ie. declared static) zcoll.c: How to get rid of all the objects, so full collect really collects all automatic objects: - Rootdrop() helps, but we can still retain a 1.2MB object; - stackwipe() does not help much -- these unwanted ambig refs are being left on the stack by MPS code that runs between mps_arena_collect and the flip! - therefore StackScan(0/1) to destroy stack+reg root before full collect: it's the only way to be sure. Reproducibility: - give Make() a random? switch, acted on by df() = diversity function, to allow bypass of rnd(); - ZRndStateSet, to set the seed for rnd() Unpark: - release after mps_arena_collect (job002206) Output: - print_M: Megabytes more useful than Mebibytes - move printf announcing Destroying arena etc to just before, not just after, we do it. testlib.c/h -- Reproducibility: - fix rnd_state so a rnd_state getter is possible; - testlib.h += rnd_state_t, rnd_state(), rnd_state_set(), rnd_state_set_v2() vmix.c: add VM_ix_Create_ok/VM_ix_Destroy diags (note: not in vmw3.c yet) trace.c: no newline on "traceFindGrey" diag please locus.c: no newline on "condemn gens" diag please diag.c: just VMCompact diag readme.txt -- update for release. |
| 170101 | closed | 2010-03-23 08:11:20 | Richard Kistruck | MPS br/vmem: simple-chunk-return: arenavm.c VMCompact diag: also emit where chunks were gained but not lost during collection -- that is: collections where there was a vmem increase, and ArenaCompact was *not* able to return any chunks. |
| 170095 | closed | 2010-03-23 01:49:17 | Richard Kistruck | MPS br/vmem: simple-chunk-return: arenavm.c -- move chunk-return into new function "VMCompact". (also, in VMArenaFinish, null out arena->primary in VMArenaFinish, so it is not left dangling). arena.c, arenavm.c, mpm.h, mpmst.h, mpmtypes.h: arena->class->compact: ArenaCompact, ArenaTrivCompact, VMCompact trace.c -- traceReclaim calls ArenaCompact! |
| 170082 | closed | 2010-03-22 13:24:49 | Richard Kistruck | MPS br/vmem: simple-chunk-return: arenavm.c -- on VMFree(), destroy any empty chunks (except the primary). (VMFree is not the ideal place to do it, but works for proof of concept). tract.c -- fix ChunkCache defects: - previously, if cache is empty (chunkCache->chunk == NULL) then other fields are *undefined*; but code looks at them anyway (!) without first checking chunkCache->chunk; - change it (.chunk.empty.fields) so that, if cache is empty, other fields have defined values: cache-using code may look at them, and they are chosen so that no cache hit will occur. --> this fixes crashing defect shown by changelist 170072 - AVERT(ChunkCache) in the many places it should be checked; - use AVERT_CRITICAL in ChunkEncache, because it is called by ChunkOfAddr; Also: plan.txt -- notes on use of primary chunk. config.h -- DIAG also in ci, please [Do not integ to master] diag.c -- skip ChainCondemnAuto diag thanks zcoll.c -- move printf announcing Destroying arena etc to just before, not just after, we do it. |