Principal numerical and empirical claims in the README, mapped to evidence artifacts, reproduction commands, expected tolerances, and limitations. Per-arm statistics in the Empirical Evidence tables are sourced directly from the JSON artifacts listed below; this document maps the summary-level claims, not every individual table cell.
Methodology
Claims are extracted from README.md as of the current commit.
Each claim must have a JSON artifact in notebooks/results/ or a CI command that produces it.
Tolerances account for hardware-specific floating-point variation (see REPRODUCING.md §Expected Numerical Variation).
“Survives global BH” indicates whether the claim’s p-value survives investigation-wide Benjamini-Hochberg correction across the 59 distinct comparisons (of 76 total computed; see STATISTICAL_SUMMARY.md).
Claim 1: “504 tests; 100% line and 100% branch coverage on the topogeoml package with torch installed”
Field
Value
Evidence
pytest --cov=topogeoml --cov-branch (with pip install -e ".[all]") reports 100% line and 100% branch coverage on the importable topogeoml package.
Artifact
CI reports coverage on every push. The dedicated full-deps coverage-gate job (.github/workflows/ci.yml, Python 3.11 / ubuntu) installs .[all] (torch CPU wheels) and runs pytest -m "not gpu" --cov=topogeoml --cov-branch --cov-fail-under=100, failing the build below 100%.
Tolerance
Exact: pytest -m "not gpu" under pip install -e ".[all]" (the full-deps coverage-gate environment) reports 504 passing tests. The suite defines 509 def test_ functions; parametrization plus optional-dependency skips resolve to 504 passing test items in that environment.
Limitation
The default test CI job installs .[dev] (no torch), so topogeoml/nn/ code paths are import-skipped and package coverage is below 100% in that job (reported, not gated there). The separate coverage-gate job installs .[all] and does gate the package at 100% line + 100% branch under --cov-branch. __init__.py files are omitted per pyproject.toml [tool.coverage.run]. The benchmarks/ research harness is not at 100% and is deliberately outside the gated scope (the gate is --cov=topogeoml; see Claim 6).
median_diff: 0.086 +/- 0.005; p_BH: 4.83e-3 +/- factor of 2
Survives global BH
Yes (rank 22/59, threshold 1.86e-2)
Survives Bonferroni
No (threshold 6.58e-4)
Limitation
One dataset (NCI1), one configuration (1-layer, hidden_dim=32, 10 epochs). Does not replicate on MUTAG or PROTEINS at this configuration. Subsequent ablation (H008-c) showed the operative factor is the external residual, not the Hodge Laplacian.
Claim 3: “topology-aware message passing with external residual outperforms MLP by 8-10 pp”
MUTAG: gap +0.098, p = 4.53e-6; PROTEINS: gap +0.088, p = 1.41e-4; NCI1: gap +0.071, p = 1.93e-5
Survives global BH
Yes (all three)
Limitation
These p-values are from the Hodge-vs-class-prior comparison within the H006 resolver, not the Hodge-vs-MLP comparison in the raw JSON. The class prior is the theoretical baseline (majority-class accuracy), not the MLP’s constant-feature accuracy.
Claim 6: Coverage on the library and the benchmark harness
100% line and 100% branch coverage with torch installed (pip install -e ".[all]", --cov-branch). Verified module-by-module: every topogeoml/* file reports 0 missed lines and 0 partial branches. Enforced by the coverage-gate CI job at --cov-fail-under=100.
Benchmark harness (benchmarks)
Not 100%. With [all] (torch + torch-geometric + GUDHI) the combined topogeoml+benchmarks line coverage is ~93%. The gap splits two ways: (a) the larger share is in cross-backend modules (cli.py, axes/speed.py, axes/stability.py, backends/torch_topological.py, benchmarks/runner.py) whose tests skip without the torch-topological backend (the bench extra); (b) a residual ~82 lines are in the hodge analysis modules (hodge/datasets.py, hodge/h006_analysis.py, hodge/h007_analysis.py, hodge/runner.py) that are genuinely partial within tests that do run, and would not be recovered by any backend install. We could not install torch-topological on Python 3.11–3.13 in our environment (its giotto-ph dependency has no compatible wheel), so the exact post-bench figure was not measured here; the (b) lines remain regardless.
Limitation
The historical “100% on topogeoml and benchmarks” claim is stale and is not currently reproducible: the benchmarks/ harness is below 100% even with the full optional stack. The library package itself is at 100% line + 100% branch and is the gated scope. Closing the remaining benchmarks/ gap is tracked as research-harness debt, not a release blocker. The default test CI job installs .[dev] (no torch) and reports coverage without gating it; the separate coverage-gate job installs .[all] and gates the package (only) at 100% line + branch.
Claim 7: “preregistered hypothesis series (H001-H011, 50+ falsifiable sub-predictions)”
git log --format="%H %ai" -- docs/hypotheses/HYPOTHESIS-008-gin-gat-comparison.md | tail -1 — commit timestamp precedes experiment result timestamp. Replace the filename with any hypothesis document to verify.
Limitation
Hypothesis selection was sequential (each informed by the prior). This is acknowledged in STATISTICAL_SUMMARY.md §4 as legitimate sequential testing, not p-hacking.
Claims not yet independently validated
The following claims have not been reproduced outside the original compute environment:
All per-seed accuracies (hardware-dependent floating-point variation expected)
The investigation-wide BH analysis (computed from the archived JSON artifacts; a third party should re-run the analysis script to verify)