docs/reference/testing.btft

[table class="topnav"]
[tr]
[td class="logotab"]mp[e]
[td][see name="index"]index[e][e]
[td][see name="guide/index"]guide[e][e]
[td][see name="reference/index"]reference[e][e]
[e]
[e]
 
[h1]testing[e]
 
test/run.sh drives every regression suite this tree ships. it runs inside a disposable sandbox (a chroot, or any throwaway environment) with mpx/mp already built -- never against a real, in-use install.
 
[code]
chroot /path/to/sandbox /bin/sh /path/to/mp/test/run.sh [suite ...]
[e]
 
with no arguments it runs every suite, in order, then a closing "mp verify" + "mp dryrun --force all" sweep against the real ports tree -- a final sanity check that nothing above left it in a broken state, run regardless of which suites were named. name one or more suites (their filename under test/suites/, without .sh) to run just those.
 
[h2]suite anatomy[e]
 
every suite is a plain shell file exposing one run_<name>_suite() function. test/lib.sh supplies the shared framework: t_begin/t_pass/t_fail bookkeeping, assert_contains/assert_file_exists/assert_exit0-style assertions, and mkport/reset_state -- the two helpers every single test uses. reset_state wipes the whole scratch ports tree, working dir, and prefix before each test, so every test starts from a genuinely clean slate and suites can run standalone, together, or in any order.
 
[code]
test_something() {
	t_begin "something"
	reset_state
	mkport myport <<'EOF'
pkg_name="myport"
pkg_ver="1.0"
install:
	touch $MP_PREFIX/myport-installed
remove:
	rm -f $MP_PREFIX/myport-installed
EOF
	mp install myport >/tmp/out 2>&1
	rc=$?
	assert_exit0 "$rc" "$CUR_TEST" || return
	assert_file_exists "$MP_PREFIX/myport-installed" "$CUR_TEST" || return
	t_pass
}
[e]
 
[h2]what's covered[e]
 
resolution, config_loading, config_matrix, dependencies, patches, hooks, commands, version, portformat, graphs, backtracking, slots_advanced, sysroots, multirepo, integration, removal, niches, plugins, and fuzz -- 18 suites (plus fuzz) covering dependency resolution and %tag backtracking (including the CPS planner's own extreme-cycle/diamond/mixed-pipeline cases, in backtracking), SLOT/USE composition, the config-layering precedence chain, patches.conf/hooks.conf's IF_DEP/IF_VER/IF_USE/AFTER grammar, hook firing order and multiset correctness, sysroot isolation, multi-repo priority ordering, mp remove's manifest-driven cleanup sweep and its pkg_manifest_cleanup opt-out (removal), the mp lint/devtest/doctor/depcheck/new toolchain (plugins), and a long list of edge cases (circular slot deps, microarch composition, keyword gating, ambiguous bare-name removal, and more) each with its own regression test tracing back to a real bug found and fixed during development.
 
[h2]the fuzz suite[e]
 
test/suites/fuzz.sh is different in kind from the others: a deterministic, seeded random package-graph generator, not a fixed list of hand-written scenarios. each iteration generates 3-7 ports with randomized versions, slots, USE flags, dependencies (including %tag providers and soft deps), hooks, and occasionally a patches.conf, then asserts a long list of exact invariants against mp's actual behavior -- the install closure (simulating resolve_dep's own depth-first visit order), the prune/runtime closure, db shape, marker file existence (including slot-prefixed paths), hook firing as an exact multiset in the right relative order, patch IF_USE gating (checked against literal patched file content), lowest-pref provider choice, and clean failure on a real conflict.
 
[code]
FUZZ_SEED=1337 FUZZ_ITERATIONS=1000 sh test/run.sh fuzz
[e]
 
FUZZ_SEED seeds everything; FUZZ_ITERATIONS sets the batch size (1000 by default). because it's fully deterministic, a failing iteration reproduces exactly by rerunning with the same seed -- and the suite deliberately stops at the first failure rather than continuing, leaving every bit of that iteration's state (scratch ports, config, db, hook log) on disk for inspection instead of burying it under 999 more iterations of unrelated output.
 
[quote class="note"]
note: the fuzz suite's own port/config names are derived from the iteration number alone, not the seed -- running it interactively many times with different seeds against the same long-lived sandbox (rather than a fresh environment per run) can leave stale config residue that causes a spurious failure unrelated to any real bug. a genuinely fresh sandbox, or a full rebuild between unrelated runs, avoids this.
[e]
 
[h2]adding a test[e]
 
pick the closest existing suite (or add a new one, and list it in test/run.sh's ALL_SUITES) and follow the pattern above: reset_state, mkport (or write directly under $SCRATCH for anything mkport's single-heredoc shape can't express -- a second port, a patches.conf, a hooks.conf), run the real mp command, assert on real files/db/output. every test is self-contained and disposable -- there is no fixture data to keep in sync, and no test depends on another test's leftover state.
 
a scratch marker written FROM a phase script (install:/build:/etc., or a $MP_HOOK-triggered custom hook fired from inside one) can only land under $MP_PREFIX or the stage dir -- mpx sandboxes a new-format phase by default (root read-only except those two, see src/sandbox.c), so /tmp genuinely isn't writable from in there. a plain pre_/post_<phase> lifecycle hook (hook_<phase>= in mp.conf, or a HOOKS_DIR/<phase>/* file) is different: it fires unsandboxed, straight from mp's own process, so /tmp works fine for one of those. four existing tests (hooks_conf_pipeline, hooks_config_override, sysroot_hooks_isolation, update_ordering) got this wrong for years before it was ever actually enforced -- whatever sandbox this suite runs under evidently didn't succeed in setting one up, silently falling back unsandboxed and leaving /tmp writable as a side effect, until it was run somewhere the sandbox genuinely works. don't repeat it: a marker two or more DIFFERENT packages' phases all write to also can't live under $MP_PREFIX either way, sandboxed or not -- mp's own manifest conflict-detection (mplib::db::find_file_conflict) rejects a second package touching a path an earlier one's manifest already claims, so a shared cross-package marker belongs in a hook_post_install= instead (see update_ordering's own fix for the pattern).