What audio codecs taught a sensor database
Every sample MorletDB stores goes through its chunk codec, Grizzbit. It is lossless: a value read back has the same bits as the value written. That rule never bends, so the only things a codec can improve are how small the data gets and how fast it moves. This post is about the second version that improves both well enough to ship, Grizzbit 2.1. It covers where its ideas came from, what a frozen study measured, and what comes next.
Where it started
Grizzbit 1 rests on one observation: most sensor floats are integers in disguise. A digitizer produces integer counts, and calibration turns them into floats by a scale and an offset. Grizzbit 1 recovers the integers, predicts each one from the previous few with a fixed polynomial predictor, and writes the small prediction errors with Rice codes. Floating-point rounding means a calibrated value does not sit exactly on its grid, so Grizzbit 1 also stores a correction in the last bits of every sample.
That already beat general-purpose compressors and the float codecs of time-series databases on calibrated waveforms. But vibration, motor currents and EEG are waveforms, just like audio, and lossless audio coding has had decades of work. So we asked a plain question: on the same integers, how far behind the best lossless audio coders is Grizzbit?
What the audio coders showed
We benchmarked Grizzbit against two strong lossless audio coders on our development data only: SRLA, and the reference coder of ITU-T T.261. On the same integers, Grizzbit 1 needed 9.75 bits per sample. SRLA needed 8.97 and T.261 8.87. The gap came from two places.
- Prediction. Audio coders fit a linear predictor to each block of the signal itself, with up to 32 coefficients, instead of using a fixed polynomial. On smooth, periodic signals such as motor currents, that predicts far better.
- Entropy coding. T.261 codes the prediction errors adaptively, with arithmetic coding. That was worth a little more on top.
The third idea came from our own data, not from audio. A calibrated value is rounded the same way every time its integer comes up. So the correction depends only on the integer, not on the sample, and a signal that visits the same few hundred values thousands of times was paying for the same correction thousands of times.
What Grizzbit 2.1 does
- Corrections once per value: each grid index's correction is stored once in a small table, not once per sample.
- A predictor fitted per chunk: up to order 32, beside Grizzbit 1's fixed predictors. The encoder picks the order by estimating code lengths, and the decoder uses integer arithmetic only.
- Never worse: every chunk is coded both ways, and the new form (chunk format version 5) is kept only if it is strictly smaller. Everywhere else the bytes are Grizzbit 1's, unchanged.
- The same bytes everywhere: the predictor fit uses floating point, so the encoder avoids fused multiply-add and maths libraries and sums in a fixed order. The same samples give the same bytes on every platform.
- Proved at write time: as with every format, each chunk is decoded again and compared bit for bit with its input before it is stored.
The failure before it
Grizzbit 2.1 is not our first attempt. Grizzbit 2 added Pcodec as a second coding, aimed at floats without a value grid, where MorletDB loses to ClickHouse and Pcodec. Its frozen study, B5, failed. On two of five series it stayed 1.38× and 1.56× larger than ClickHouse's best, where we had required 1.25×. It also made one read measure slower and lost one result against QuestDB. It never shipped. Its format version, 4, is reserved and rejected by every reader.
How we tested 2.1
Grizzbit 2.1 went through its own frozen study, B6.
- The pass criteria were fixed before any benchmark input was read.
- The inputs were 30 public datasets that none of 2.1's development had touched. We excluded every file of the five bearings it was developed on.
- The codec had to pass fuzzing and formal proofs before it was measured.
| Criterion (fixed in advance) | Result |
|---|---|
| Every chunk and read bit for bit exact | held: 36 inputs, 36 database loads, 0 mismatches |
| Six waveforms together at least 5% smaller, 4 of 6 smaller | held: 12.6% smaller, 6 of 6 |
| No chunk and no input larger | held |
| Decode at least 50 and encode at least 10 M samples/s per waveform | held: lowest 53.1 and 10.8 |
| Same bytes on every platform | held: macOS and Linux on ARM, Linux on x86, Raspberry Pi |
| Every B3 result against QuestDB still holds | held: ingestion and reads unchanged |
The gains are largest where MorletDB is meant to be used:
- motor phase currents 18 to 20% smaller;
- machine vibration 16%;
- calibrated bearing vibration 14%;
- sleep EEG 31%.
What it does not do
- Uneven gains: ECG improved by 0.4% and seismic counts by 4%.
- Floats without a value grid are unchanged: 12 of the 18 low-rate decimal series did not change at all, so the gap to ClickHouse and Pcodec there remains.
- Pcodec still ahead in one case: on one waveform, decimal vibration values, Pcodec is still slightly smaller.
- Tight margins: the speed floors held, but only just, and Grizzbit 2.1 encodes at about half to three quarters of Grizzbit 1's speed. Ingestion end to end was about 5% slower, below the threshold we count as a difference.
- Edge hardware: a Raspberry Pi 3B+ is about 12× slower than the benchmark machine. That still encodes a 64 kHz channel about 18× faster than real time and decodes it about 70× faster. These are development figures, not a study.
Where it stands
Grizzbit 2.1 ships opt-in with the next release. One setting turns it on, and the default stays Grizzbit 1:
ALTER DATABASE plant SET morlet.chunk_format = 'grizzbit21';- Only chunks written from then on use version 5. Old chunks keep their format, and a collection may mix both.
- Clients that cannot read version 5 get plain samples through compressed transfer.
- The embedded edition's file format stays Grizzbit 1.
It becomes the default only once it has run in the field and has more headroom on speed.
What comes next
- Grizzbit 2.2: speed, with no format change (#394). Vectorised predictor and table paths, aiming at decoding at least 1.3× faster with exactly the same bytes. Headroom is what 2.1 needs before it can be the default.
- Prediction across channels (#395). The three phases of a motor's current add up to almost zero, and the axes of a vibration sensor move together. Today every channel is coded alone. This is the direction that could bring the next large step, and it starts as a design study, because one channel's read must not depend on decoding all the others.
- Adaptive entropy coding (#396). Adaptive Golomb-Rice or arithmetic coding could close most of the remaining gap to T.261, but only if decoding stays fast, so it waits for 2.2.
- Segments within a chunk (#397) and larger chunks (#398). Both help where a signal changes character, or where a predictor could be amortised over more samples.
- Floats without a grid (#326) and cycle prediction (#375). Separate problems that 2.1 does not touch. The first is the weakness B5 tried and failed to fix. The second is research: a machine that repeats a cycle may be its own best predictor.
Every change to the format gets its own frozen study before it ships, as 2.1 did. Some will fail, as B5 did, and we will say so.