| Title | Regular testing omits the MMQA test suite |
| Status | open |
| Priority | optional |
| Assigned user | Gareth Rees |
| Organization | Ravenbrook |
| Description | Continuous integration does not compile, run, and test the MMQA test suite example. This means that we risk breaking these tests and not finding out. |
| Analysis | 1. Make it easier to run the MMQA test suite by adding a make target. 2. Add the MMQA to the Travis continuous integration? The problem here is that the MMQA takes a very long time to run (on my laptop, 83 minutes) and this means not only that we are at risk of hitting the hard Travis time limit (120 minutes) but that we don't get timely results. It would be better to schedule the MMQA tests to run less often, for example once a day, for example using nightli.es [2]. 3. Install Jenkins locally and schedule regular MMQA builds there? 4. Speed up MMQA. Much of the runtime is overkill (do we really learn any more after 100,000 iterations than we did after 1,000?). Here are all the test cases that take more than 10 seconds, in reverse order by how long they take: CASE TIME SOLUTION ----------------------------------------------------------------------- 206 1207 Reduce iterations; systematize configurations 171 1134 Reduce iterations so there aren't so many chunks 227 504 Reduce iterations 215 326 Reduce iterations 140 206 Exponentially reduce commit limit; stop when too small 125 176 Reduce arena extensions 78 168 Reduce iterations 55 166 Reduce iterations 164 160 Exponentially reduce commit limit; stop when too small 207 143 Exponentially reduce commit limit; stop when too small 45 122 Reduce iterations 129 107 Reduce iterations 114 81 Reduce iterations 204 71 Reduce iterations 205 70 Reduce iterations 203 70 Reduce iterations 200 63 Reduce iterations 109 53 More address space to avoid running out of zones 49 52 More address space to avoid running out of zones 70 51 Interactive test, no point in running unattended 35 45 Reduce iterations 113 39 Reduce iterations 61 37 Reduce iterations 226 34 Reduce iterations 12 31 Reduce iterations 48 29 Use smaller objects 134 28 Reduce iterations 128 27 Reduce iterations 127 27 Reduce iterations 223 26 Reduce iterations 124 25 Reduce iterations 77 22 Reduce iterations 60 21 Use smaller objects 30 21 Reduce iterations 44 20 Use smaller objects 97 19 Reduce iterations 96 19 Take bigger steps 29 18 Reduce iterations 25 17 Reduce iterations 43 16 Use smaller objects 42 15 Use smaller objects 104 14 Reduce iterations 41 13 Reduce iterations & use smaller objects 36 11 Use smaller objects & turn off debugging |
| How found | inspection |
| Evidence | [1] <https://info.ravenbrook.com/mail/2016/03/03/21-27-46/0/>[2] < https://nightli.es> |
| Created by | Gareth Rees |
| Created on | 2016-03-06 11:24:47 |
| Last modified by | Gareth Rees |
| Last modified on | 2016-03-14 11:52:03 |
| History | 2016-03-06 GDR Created based on [1]. |
| Change | Effect | Date | User | Description |
|---|---|---|---|---|
| 190027 | open | 2016-03-14 11:51:35 | Gareth Rees | Don't run MMQA on every Travis build: it has intermittent failures. |
| 189957 | closed | 2016-03-13 12:57:02 | Gareth Rees | Regular testing of MMQA: "make test-make-build" at top level now runs MMQA, hence so does "make test" on Linux and FreeBSD. The whole test suite, including testci, testmmqa, testansi, and testpollnone, takes 26 minutes on my laptop. MMQA "run" and "runset" commands now exit with non-zero status if any of the tests failed, so that test failures can be identifed by the calling program (for example, make). Add missing license to files that state "See end of file for license." |
| 189948 | open | 2016-03-13 11:42:53 | Gareth Rees | A few more MMQA speedups. |
| 189947 | open | 2016-03-13 11:42:32 | Gareth Rees | Speed up more MMQA test cases. On my laptop, the whole MMQA test suite (or at least the passing subset) now takes 5 minutes in the hot variety and 11 minutes in the cool variety. |
| 189674 | open | 2016-03-06 14:15:37 | Gareth Rees | Add make target for running the MMQA test suite. |