One file, every bit
A vibration recording that explains a machine failure is usually hard to pass on. It sits in a database that only the plant can reach, or it is exported as CSV with rounded values and no sample rate, or it is saved in a vendor format that needs the vendor's software to open. The engineer who should look at it gets a screenshot.
MorletDB now has an answer for that: a recording can be one file. It holds every sample exactly as it was measured, with its sample rate, unit, channel, start time and metadata, in an ordinary SQLite file. The samples are stored in the same chunks as in the MorletDB server, and the format is written down so that anyone can read it without our software.
What we built
The embedded edition is MorletDB without a server: a small library with SQLite inside, usable from Python, from the command line, and from C and every language that can call C.
import morletdb
with morletdb.open("line3.morlet") as db:
db.ingest("pump-7/accel-x", samples, sample_rate_hz=25_600, channel="accel-x", unit="m/s^2")
iv = db.read("pump-7/accel-x", 40_960, 45_056) # the stored samples, bit for bit
st = db.stats("pump-7/accel-x", 0, 1_000_000) # count, min, max, mean, rms- Lossless: every chunk is decoded again after encoding and compared bit for bit before it is written.
- Durable: every append is one SQLite transaction, synced to disk before it returns.
- Strict: appends must continue at the end of a recording, an identical retry changes nothing, and anything else is refused without changing the file.
The same bytes as the server
A portable format is only useful if nothing changes on the way. The file uses the server's chunk encoding, so we can check that directly. We appended the same 1.2 million samples once through the server and once into a file:
| Check | Result |
|---|---|
| Chunks (start, length, encoded bytes, summary bytes) | 293 of 293 identical |
| Interval statistics over 200 random intervals | identical to the server's, bit for bit |
| Reads against the original samples | 20 of 20 identical |
So a recording can move between a test bench, a laptop, an archive and the plant database. import_file and export_file copy recordings chunk for chunk in either direction. After every 256 chunks they compare a hash of the chunk bytes on both sides, and they continue where an earlier transfer stopped. A failure case exported from the server and attached to a ticket is the same data, not a copy that is close enough.
Readable without MorletDB
A file format you can only read with one program is not portable. So we did three things before calling it a format:
- A written specification. File format 1 defines the tables, what each column means and the rules a valid file follows. The chunk format now specifies decoding down to the bit: predictors, Rice codes, the mapping of floats to integers, last-place corrections, the checksum.
- A second reader. morletfile.py reads the format in 254 lines of Python, using only the standard library: no numpy, no MorletDB code.
- A test corpus. A small file with eight cases covers every kind of chunk the format has: ADC counts, decimal steps that are not exact in binary, calibrated values with an offset, noise, incompressible values, negative zero, the amplitude limits, and chunks without a summary. Our continuous integration checks that both readers decode all of it bit for bit, and that the reference reader detects a single flipped bit.
Format 1 comes with a promise: a format-1 file stays readable by every later version. One caveat on the second reader: we wrote it ourselves, and writing it is how we found the details the specification was missing. It shows that the specification is now complete enough to implement. The real test is someone else implementing it from the text alone, and we would welcome that.
Why SQLite
Because it is everywhere. Every language and platform can open an SQLite file, and so can many tools that engineers already have. The recordings, their metadata, timing and chunk layout are plain tables you can query. Only the samples themselves need a decoder, and that decoder is now documented.
File or server?
| File (embedded edition) | Server (PostgreSQL + pg_morlet) | |
|---|---|---|
| Where | inside one program: logger, test bench, device, laptop | shared by a line or a plant |
| Writers | one at a time | many in parallel |
| Access | the file's permissions | roles and row-level security |
| Search by shape, memory levels | not yet | yes |
They complement each other: record next to the machine, collect centrally, export a case as a file.
What this is, and what not
- The edition is experimental. The format is frozen; the library around it will still change.
- No speed claims yet. Reading in the same process as the program avoids a network round trip, and first measurements look promising. They were taken on a busy laptop without a frozen protocol, so we do not publish them. A benchmark on a desktop and a Raspberry Pi comes first.
- Not yet tested against power loss or a full disk. SQLite has a strong record there; our use of it has not been tested yet, and will be before we recommend it for unattended recording.
- Not yet: search by shape in the file, Windows builds, a package on PyPI.
- Comparisons with other formats: we have not measured file size or speed against WAV, TDMS, HDF5 or Parquet, and claim nothing about them.