Missing simulation datasets: 5 confirmed losses on published models, all 2004-2005 — find the extent and the cause

[jcschaff]

Focused follow-up to #1966, which established the infrastructure facts (cfs07 now mounted, 904 of 907 "missing" simulations never had results) and is closed. This issue is only about the datasets that are genuinely gone: how many, and why.

What is known

A simulation with a vc_simulationjob row carrying hasData='Y' is the database asserting that results exist. Where the files are absent from every share, that assertion is wrong and data has been lost.

Sweeping every such simulation belonging to a published BioModel — 1747 across 258 published models, checked against /simdata (cfs09), /simdata_secondary (cfs07) and /share/apps/vcell12/users (cfs12) from inside the prod data pod:

data present	1742 (99.7%)
missing	5

user	simId	year	bioModel	parentSimRef
mayya	10480740	2004	10829774 (STATmodulePaper)	set
mayya	10483149	2004	10829774	set
mayya	10483210	2004	10829774	10415851
mayya	10633806	2004	10829774	10633707
partharoy	12119710	2005	12119723 (7_12_00_model1)	12117313

Zero files of any kind survive for these — not just the .log the data service looks for first.

Three patterns, none yet explained

1. Time. All five are 2004–2005. Published-model simulations with hasData='Y' span 2002–2026, and 84 of them predate 2006 — so the losses are ~6% of the pre-2006 population and 0% of everything after. Whatever happened appears to have stopped.

2. Concentration. Four of five are on one BioModel (10829774), owned by one user. That model has 32 simulations, all claiming data; 28 are intact. So this is partial loss within a model, not a model-level deletion.

3. Lineage. All five have a non-null PARENTSIMREF, against a base rate of 1229 / 261,111 = 0.47% among simulations with hasData='Y'. Under that base rate, 5-for-5 by chance is vanishingly unlikely.

   But the selection is not clean. These came from published models, which are both more crawled (hence noticed) and plausibly more copied (hence more likely to have parents). The correlation may be with publication, not with having a parent. Test this before building on it.

   The parents themselves — 10415851, 10633707, 12117313 — also carry hasData='Y' and are also missing. So whatever removed the child removed the parent; it is not a copy-versus-original distinction.

What this rules out

• Not ResultSetCrawler retention logic. prod: ~3000 data requests/month for simulations that never had results (only 3 real losses), and the secondary share cfs07 is mounted nowhere #1966 disconfirmed this twice: every affected simulation has a direct vc_biomodelsim link (none is parent-ref-only), and getSimulationInfos(user, false) resolves to WHERE ownerRef = <user> with no join to the link tables (DbDriver.java:1711), so a row is retained regardless of how it is referenced.
• Not the missing cfs07 mount. That is now fixed and verified, and these files are absent from cfs07 as well.
• Not wholesale storage loss. User directories are intact — les 63,769 files, JDitlev 50,327, prangamani 41,834 — and 99.7% of published-model results are present.

Proposed work

1. Extend the sweep beyond published models. The 1747 checked here are the models we care most about, but they are 0.7% of the 261,111 simulations with hasData='Y'. A full sweep is a few hours of ls; a stratified sample by year would answer "is this era-bound?" much faster. The 2004–2005 concentration is the thing to confirm or break first — if the rate outside published models is also ~0% post-2005, that dates the event.
2. Test the PARENTSIMREF correlation properly. Compare against simulations matched on publication status and year. If it survives that, it is a lead; if it does not, it was publication all along.
3. Find out what happened in 2004–2005. Storage migrations, a vcell5-era share retirement, or an early cleanup tool are all candidates. vcell5 (2.3P, 1.4P used) is mounted nowhere today and was searched in the original sweep — worth asking whether it ever held this era's data and what moved off it.
4. Decide what to do about the five. They belong to published papers, so the results are likely reproducible by re-running the simulations, and the models themselves are intact. Options: re-run and re-attach, or mark them honestly as unavailable. Doing nothing means a user opening a published model gets an error with no explanation.
5. Consider making the assertion self-checking. hasData='Y' with no files is a database claim contradicted by the filesystem, and nothing notices until a user asks. A periodic reconciliation over published models — 1747 rows, cheap — would turn silent loss into a report.

Not urgent, and worth saying why

Five datasets out of 1747 on published models, none newer than 2005, on a system where the corresponding models are all still present. This is a data-integrity question worth answering, not an outage. The user-facing symptom that made it visible (#1967) is already fixed and ships to prod with 8.0.25.01.