Known Limitations¶
This document enumerates the specific, named limitations of the project
as maintained in the latest repository revision. It complements STATUS.md (which gives the
high-level posture) by listing concrete, addressable items.
If you find a limitation that is not listed here, please open an issue
labelled limitation.
L1 — ACMG criterion mapping is not independently validated¶
Module: src/alphafold_sovereign/tools/precision_medicine.py
(_vep_to_acmg, _unassessed_evidence, and related helpers).
Description: The mapping from VEP consequence terms onto candidate
ACMG criteria (PVS1, BP7) is implemented from a reading of Richards et
al. 2015 by the project author. No clinical geneticist has signed off on
it. gnomAD allele frequencies and predictor outputs (AlphaMissense, SIFT,
PolyPhen) are not mapped onto criteria: they are returned as
observations with BA1, BS1, PM2, PP3 and BP4 marked NOT_ASSESSED,
because those criteria depend on disorder-specific frequency thresholds
and predictor-specific calibration that this project does not evaluate.
CADD is not retrieved: the server does not request it from Ensembl VEP,
and cadd_phred is always null with a NOT_RETRIEVED note.
Impact: Computational evidence could be mistaken for established
clinical criteria if source attribution or the unassessed state is lost.
The corrected classify_variant_acmg output no longer issues a
pathogenicity classification without disease-specific verification.
The multi-source clinical report remains a research aid, not a clinical call.
Mitigation: Preserve the separate source data and unreviewed candidate signals. Determine disease mechanism and clinical criteria from primary sources under qualified independent review.
Planned resolution: Roadmap step 2 (traceability matrix) and step 3
(external review). See STATUS.md.
L2 — Druggability tier thresholds are unvalidated heuristics¶
Module: src/alphafold_sovereign/domain/druggability.py
(score_target_druggability), surfaced by
src/alphafold_sovereign/tools/precision_medicine.py
(_druggability_tier).
Description: The mapping from (drug_count, tractability,
LOEUF, pLDDT) to (HOT, WARM, COLD, NOT_DRUGGABLE) uses
integer score cut-offs (≥4, ≥2, ≥1, 0) chosen by the author. The
score itself is the sum of small literature-informed weights. No
calibration against a benchmark of approved-drug vs. failed-drug
targets has been performed. The weights and cut-offs are now named,
documented constants in the pure domain/druggability.py module so
they can be inspected and, in future, re-calibrated without touching
the tool orchestration.
Impact: A target classified as HOT here may not be druggable in practice. A target classified as COLD here may have an approved drug.
Mitigation: Use the tier only as a triage signal alongside expert
review. To make the strength of each call legible, the
scoring_breakdown now also reports data completeness
(signals.available / signals.missing), a confidence grade,
and a borderline flag — so a HOT that rests on two of four signals
and sits one point from WARM is no longer indistinguishable from a
fully-evidenced HOT. These fields describe the evidence base, not the
scientific validity of the cut-offs, which remain uncalibrated.
Planned resolution: Roadmap step 4 (benchmark calibration).
L3 — Upstream API schemas are not pinned¶
Modules: all of src/alphafold_sovereign/clients/.
Description: We call live APIs (Ensembl REST, Open Targets GraphQL, ClinVar VCV JSON, gnomAD GraphQL, AlphaFold DB, MONDO, HPO, DisGeNET, ChEMBL) at their current schema. If an upstream changes shape, serialisation may break.
Impact: Reproducibility is limited. A run today may produce different results from a run a year from now.
Mitigation: Client-level parsing, selected Pydantic input models,
and synthetic source-response regression tests limit some schema risks.
The former real-API replay fixtures are historical Git artifacts, not
active tests or independently validated scientific reference cases.
These measures are not a full schema contract for every upstream and
cannot rule out silent data drift. AlphaFold prediction-field renames are
handled explicitly; see docs/afdb-compatibility.md.
Planned resolution: Roadmap step 5 (schema pinning with refresh policy).
L4 — No production deployment experience¶
Description: This server has never been run for real users in a long-lived process. We do not know the actual memory footprint of the SQLite knowledge graph after a month of use, the actual rate of upstream 429 throttling, or the actual latency of the slowest tool.
Impact: Performance and reliability claims should be assumed to be "works on a laptop" rather than "works at scale".
Mitigation: CircuitBreaker and UpstreamConfig provide
defensive primitives, but they have only been tested in unit tests
with mocked time.
Planned resolution: First real deployment will produce real observations to publish back to this document.
L5 — macOS Python 3.11 test flake¶
Description: On the GitHub Actions macos-latest runner with
Python 3.11, the test suite has intermittently failed even though
identical code passes on every other matrix entry. The failure is
non-deterministic; the root cause has not been isolated.
Impact: CI may be red on a benign push.
Mitigation: The CI test step retries the run once only on the
macos-latest / Python 3.11 matrix entry; every other entry fails fast.
A one-off timing flake therefore clears automatically, while a genuine
consistent failure still reds the build (it is not masked).
Planned resolution: The matrix entry is now stabilised via the scoped retry above. Isolating the specific timing-sensitive test for a deeper fix remains a follow-up if the flake recurs.
L6 — Single-maintainer bus factor¶
Description: This is currently a single-person project. There is no co-maintainer, no governance body, and no funded continuity plan.
Impact: If the maintainer becomes unavailable, the project stalls.
Mitigation: The Apache 2.0 licence ensures any user can fork; the
code is small enough (3,145 statements as of v1.2.0, verified via
pytest --cov) to be picked up by a reasonably motivated successor.
Planned resolution: Recruit co-maintainers (see
GOVERNANCE.md).
L7 — No telemetry on real-world correctness¶
Description: We do not collect any data on tool invocations, arguments, or outputs from users. We therefore cannot say "in N real-world variant reports, the ACMG call agreed with a geneticist M% of the time."
Impact: Quality claims are based on unit tests, not field data.
Mitigation: None for this release. Intentional — we will not add telemetry without explicit opt-in.
Planned resolution: An opt-in, locally-stored "quality journal" that lets users mark each report as "matches expert / disagrees / unsure", aggregated only with explicit upload consent.
L8 — AlphaFold fragment selection and residue numbering¶
Module: src/alphafold_sovereign/clients/alphafold.py and
src/alphafold_sovereign/tools/structure_intelligence.py.
Description: AlphaFold DB may provide multiple predicted fragments for very long sequences. The client selects the exact requested UniProt accession over different isoforms, but where multiple fragments share that accession it currently uses the first returned match. The tool does not offer a fragment selector, a verified mapping from fragment indices to full-length UniProt residues, or an experimental PDB/SIFTS mapping.
Impact: A model's reported sequence and confidence may cover only a fragment, so users must not interpret the fragment's residue positions or sequence length as an unambiguous full-length protein coordinate system.
Mitigation: Resolve a specific model/fragment through the upstream AlphaFold DB record before interpreting residue positions; retain source model identifiers in downstream research records. The server currently does not project text-mined residue annotations onto model coordinates.
Planned resolution: Document and test the precise fragment-selection and residue-coordinate contract against live AFDB examples before adding a fragment-specific tool parameter or any literature-annotation join.
L9 — PDBe/IUCr literature evidence is source-reported, not independently mapped¶
Module: src/alphafold_sovereign/clients/pdbe_literature.py and the optional
generate_variant_clinical_report literature section.
The client calls PDBe's documented UniProt residue API, checks the identity reported with each annotation, and returns the upstream PDB chain/residue, UniProt residue, article identifiers and AI-extracted sentence. It does not recompute SIFTS, verify that caller-provided coordinates match the HGVS variant's transcript/isoform, or retrieve experimental model-quality metrics. Text-mined mentions can describe wild-type function, unrelated substitutions, or false positives. Missing or inaccessible publications may produce no annotations. API errors are marked unavailable, not absent.
Mitigation: Do not score or classify variants from this evidence. Verify exact residue mapping, the original publication, experimental source structures and variant context before scientific interpretation. The default report remains unchanged unless the literature option is explicitly set.
L10 — Positions of locally elevated PAE are not domain boundaries¶
Module: src/alphafold_sovereign/tools/structure_intelligence.py
(_local_high_pae_onsets, surfaced by analyze_structural_confidence).
Description: analyze_structural_confidence reports
local_high_pae.positions: zero-based model indices where the mean PAE of a
20×20 block on the diagonal of the PAE matrix exceeds the mean of the whole
matrix by more than 1.5 standard deviations of that statistic. The block lies
on the diagonal, so the statistic describes uncertainty among sequence
neighbours. Uncertainty between two confidently predicted regions lies off the
diagonal and is not measured. Releases up to 1.5.12 returned the same positions
as domain_boundaries.candidate_positions and derived n_putative_domains
from their number.
In an internal, AI-assisted audit of 18 AlphaFold DB models (November 2026; not an independent validation) the statistic returned seven positions in total, each within 15 residues of a chain terminus. For the eight proteins with two or more UniProtKB Domain features (SRC, ABL1, GRB2, CRK, PTPN11, CALM1, ALB, BRCA1) it marked no border between two annotated domains: seven models returned no position, and ALB returned one, three residues before its first domain. None of the nine models whose mean PAE was 11.9 Å or more returned a position: an index is flagged only where a diagonal block exceeds the whole-matrix mean, and that mean rises with the overall PAE.
Impact: The earlier field names presented these positions as domain boundaries and as a domain count. They are neither. An empty list does not indicate a single-domain protein, and a listed position does not indicate a domain border.
Mitigation: The positions are reported under local_high_pae together with
their definition. The legacy domain_boundaries block is kept so that existing
clients still find its keys, with status: "NOT_ASSESSED", an empty
candidate_positions list and n_putative_domains: null. The last two are the
values the block already took in earlier releases when no PAE matrix was
available.
Use curated domain annotations, such as InterPro or the UniProtKB Domain
features, for domains.
Planned resolution: None scheduled. Segmenting a PAE matrix into domains would be a new method that needs its own validation against curated annotations.
L11 — Core-residue clusters are not validated binding pockets¶
Module: src/alphafold_sovereign/tools/structure_intelligence.py
(_cluster_core_residues, _geometry_score, surfaced by
score_binding_pocket_geometry).
Description: score_binding_pocket_geometry takes the Cα atoms closer to
the model centroid than the 60th percentile of centroid distances, groups them
greedily into clusters of 8 Å radius, and scores each cluster from its residue
count, radius of gyration, mean pLDDT and distance from the model centroid. It
computes no cavity, solvent-accessible surface or ligand contact. Releases up
to 1.5.12 called the clusters putative pockets, called the score a
druggability index, and attached an EXCELLENT, GOOD, MODERATE or POOR
label to it.
In the same internal audit, on 19 AlphaFold DB models, all 189 reported
clusters were labelled EXCELLENT (133) or GOOD (56): every one scored 55
or more, the lower edge of GOOD, on the 0 to 100 scale. The reference
was the PDBe-KB ligand-site residues of each protein, for ligands of at least
10 atoms, contacted in at least 20% of the ligand records. The top-ranked
cluster was enriched for that site in 4 of the 18 proteins that had one (CDK2,
ADA, AKR1B1, HRAS; empirical p < 0.05 against 2,000 random residue sets of the
same size, not corrected for the number of proteins) and shared no residue with
it in 9. The clusters of a protein together covered 53 to 60% of its residues
and, on average, 85% of its reference-site residues, so they do not localise a
site.
Impact: The labels did not discriminate, and a top-ranked cluster is usually not a known ligand site. Reading the output as pocket detection or as a druggability assessment is not supported.
Mitigation: The output names the clusters core_residue_clusters and the
score geometry_score, and carries assessment_status, methodology and
note fields that state what is computed. The earlier keys remain as
deprecated aliases so that existing clients keep working: putative_pockets
lists the same clusters in the earlier entry shape (pocket_id,
druggability_index, druggability_components) with the same numbers, and
n_pockets_found repeats the cluster count. In those entries
druggability_label is NOT_ASSESSED. deprecated_aliases maps each earlier
key to its current name. The numbers themselves are unchanged. To locate
binding sites, use a dedicated pocket-detection method and experimental
evidence.
Planned resolution: None scheduled. A validated pocket detector would be new work. Removing the deprecated aliases is a decision for a future major version.
L12 — The Cα topology vector holds at most 16 statistics and is not a validated similarity measure¶
Module: src/alphafold_sovereign/tools/structure_intelligence.py
(_compute_tda_fingerprint, _fallback_tda_fingerprint,
_fingerprint_distance, surfaced by compute_topology_fingerprint,
compare_proteins_topologically and find_evolutionary_structural_shifts).
Description: With the optional [tda] extra, the 64-slot vector holds
eight statistics of the finite H₀ bar lifetimes and eight of the finite H₁ bar
lifetimes of a Vietoris-Rips complex on the Cα atoms, truncated at 15 Å. The
other 48 slots are always zero: H₂ is not computed, and slots 24 to 63 are
padding. Without GUDHI the vector holds eight distance and contact statistics
and no persistent homology. With GUDHI, a model with more than 500 Cα atoms is
reduced to 500 evenly spaced ones before the complex is built, so its values
describe that subsample. The fallback is not subsampled: it uses every residue
and builds the full pairwise distance matrix, so its memory and time grow with
the square of the model length. The distance the tools report is the L2
distance between length-normalised vectors.
In the same internal audit (GUDHI 3.13) the Betti numbers and bar lengths were correct on analytic point clouds and unchanged by a rigid motion. On 23 AlphaFold DB models every vector had 16 non-zero slots, and at least 99.94% of its squared norm lay in the two total-persistence slots, so the distance follows the difference in one ratio, total H₁ over total H₀ persistence (Spearman 0.94). Same-family pairs were closer than cross-family pairs on average (AUC 0.98 over 22 and 231 pairs), but on that ratio 25 of the 231 cross-family pairs were closer than the median same-family pair, for example carbonic anhydrase 1 and FABP4. All 23 models had Betti numbers β₀ = 1 and β₁ = 0; none was longer than 500 residues. For three further models that are, the β₁ of the subsample differed from that of the full chain: 9 against 7 for EGFR, 11 against 12 for ABL1, and 53 against 63 for BRCA1.
One precondition applies to the GUDHI path. GUDHI does not compute homology in the highest dimension present in the complex, so if the 15-Å complex contains no triangle, β₁ is reported as 0 whatever its true value (a square with 14-Å sides has one loop and is reported with none), and if it contains no edge, β₀ is reported as 0. Any three consecutive residues form a triangle at this scale, so a model of up to 500 residues is not affected. A longer, subsampled model would be affected only if no three of its sampled Cα atoms lay within 15 Å of one another.
Impact: "64 components" overstated the information in the vector. The
distance is not a measure of fold similarity, homology or function, and GUDHI
values for models longer than 500 residues are not comparable with full-chain
values.
The earlier description of compare_proteins_topologically listed possible
uses (drug-repurposing triage, off-target screening, cross-species comparison)
that no evidence supports.
Mitigation: The tool descriptions and outputs state the slot layout
(fingerprint_layout), the subsampling (subsampled, n_residues_total,
n_residues_used) and the method that actually ran, and the list of uses is
removed. Treat the distance as an exploratory descriptor, and confirm any
similarity with a structural alignment.
Planned resolution: None scheduled. Whether the persistence ratio is useful for any task is a research question.
How to add a new entry¶
When you find a new limitation:
- Open a GitHub issue with the
limitationlabel. - Add a new
L<N>section here in the next PR. Include: module, description, impact, mitigation, planned resolution. - If the limitation rises to the level of a safety concern, also add a clear note in the relevant tool's docstring.
Last updated: 2026-11-11.