An archive that forgets on purpose
Engineers at a machine ask a question time-series databases do not answer: have we seen this before, and what was it? Our second report argues that a waveform archive should work as an episodic memory, and that such a memory has to forget: one vibration channel at 64 kHz still grows by about 9 GB a day after the best lossless compression we measured. The question is how to forget deliberately, in steps, with a record of each one, and without losing the events that matter.
What we found
- A model of the episode. An episode is an exact stretch of waveform, the operating context it occurred in, and the outcome it later turned out to have. It has to be found years later, under small time shifts.
- A compaction ladder. Five levels: full, not searchable, represented, summarized, gone. Each level states which questions it still answers exactly. A versioned keep-value decides when an episode moves down, and a ledger records every step.
- The building blocks are measured. Lossless storage at 1.5–1.7 bytes per sample on calibrated vibration data, and chunk summaries whose statistics were bitwise identical to decoding in 3,000 of 3,000 checks.
- Exempting known events keeps them. In study F1, frozen before its two confirmation runs, keeping known events exempt left all of them searchable at every reachable budget. Retention by age kept roughly the budget’s share: 23% of seismic events at a 25% budget. On seismic data the keep-value also answered rare queries better than retention by age at 25% and 10% of the archive.
- Part of it ships. Release 0.3.7-alpha.1 contains phase 1: episodes, the levels, moves with a dry run, and one ledger row per move with a checksum of what was removed. Reads of compacted samples fail and name the level and ledger entry. The policy engine used in F1 is in the repository.
Where it falls short
- The keep-value did not win. By the rule fixed in advance, retention by age stays the default, and the keep-value is not built further as a default.
- Screwdriving cycles. Failed cycles are a quarter of that archive, so the exempt events take most of the budget. At 50% the rare-query answers were slightly worse on average, typical answers degraded (median nearest-hit ratio 1.27 against 1.04), and budgets below 30% could not be met without forgetting events.
- Novelty alone failed. It was never better than retention by age and often worse: the near-duplicates it removes are also the nearest neighbors of rare queries.
- A measure changed after seeing data. Recall at ten was replaced by the nearest hit as the primary measure after development runs, before the freeze.
- Narrow evidence. Two datasets, one budget definition, a static novelty. Consolidation into representatives, dynamic novelty and the visibility of slow drift are not tested. Answer quality is a proxy; only pilots can show usefulness to an engineer.
Read it
Version 0.2, a draft position and design report with a first evaluation. Not peer reviewed, not yet released. Text and figures under CC BY 4.0, code under Apache-2.0.