Shrinking datasets is quite contextual. It depends on how the dataset is going to be used. Its the same concept as optimizing databases, based on the read/write patterns.
PACLEC and Traffic Patterns are the key determining factors.
Hosted on fly.io : https://materials-db.fly.dev/
The materials trajectory database was around 12Gigs and is mostly used for training models like CHGNet.
But I needed it only for responding to query on finding materials and it didn't need all the variations.
The aim was to be able to serve it using my most loved database - sqlite, in under reasonable timeframe: <500ms.
Summary:
The source data and the problem. The project is a Go API serving 145,923 materials from the Materials Project, backed by a single SQLite file. The source data is 11.35GB, while the database it actually runs on is only 266MB. The raw dataset (MPtrj) is so large because it was built as a training corpus for a machine-learning model - each material's relaxation run gets sampled at several points along a trajectory at different pressures and energies, making every point a valid training example. That's why the dataset has 1.58 million rows, not because of sloppy deduplication.
KeySpace Reduction: Most of the times JSON has long key names, this is unnecessary to storage, so converting keys like "lattice_structure" to "ls" was the first step. This was the initial memory optimization that was needed. The versioned response adapter would expand these back
The big win: deduplication (~20x reduction). The first and largest cut came purely from scope - the ingest step keeps only one structure per material, picked by lowest energy, using a ROW_NUMBER() window function. Every other task retains only five scalar properties in a separate table, but loses its full structure blob. This single decision takes the dataset from 11.35GB down to roughly 592MB. This isn't "free cleanup" - those discarded structures had real scientific value for ML training, but the API's job (ground-state lookups) simply didn't need them.
Compression: 592MB -> 266MB. The remaining bulk was almost entirely structure_json blobs - one per material. Two techniques were applied: first, the verbose pymatgen JSON format was stripped down to a compact positional array (lattice matrix + sites as [element, a, b, c, occupancy]), and then zlib compression was applied on top. Benchmarking showed that stripping first and then compressing outperformed either approach alone. Each blob also got a 4-byte versioned header so old and new rows could coexist during migration without requiring a full re-ingest.
Serving it fast. At 266MB, the entire database fits inside a single 1GB mmap region, so reads hit the page cache after the first touch. A channel-based pool of read-only connections runs warmup queries against key indexes on boot, singleflight collapses concurrent identical requests into one SQLite call, and an LRU cache sits in front of lookups since the data is static. The honest caveat: a cold start on a small VM still takes 15–24 seconds for the first request, but everything after that is sub-500ms. Browser cache headers do the rest of the job
Apart from singleflight, nginx could also be used, but this is fast enough.
A detailed write up is provided in my blog: https://ikouchiha47.github.io/2026/08/05/materialsdb-11gb-to-266mb.html