A file format defines how data is structured and stored so that software can read and write it consistently.
Think of it as the container for your data. Just like music comes in MP3, WAV, or FLAC, data files also come in different formats depending on the use case.
Common data file formats:
CSV – Simple text-based, rows separated by commas (easy to use, but
large in size and no compression).
JSON – Flexible, hierarchical (good for APIs but verbose).
Avro – Row-based binary format, compact, supports schema evolution
(common in streaming).
ORC – Columnar format optimized for Hadoop and Hive.
Parquet – Columnar format designed for analytics and compression
efficiency.
Introduction to Parquet
If you work with data, you have probably heard the term Parquet come up in conversations about data lakes, analytics, or cloud storage. But what exactly is it, and why does everyone seem to love it?
The Everyday Analogy: Organizing Your Groceries
Imagine you run a grocery delivery service, and you have to store all the items customers ordered.
You could store them in two ways:
Row-based storage (like CSV):
You put one of each item from an order into a bag: milk, bread, apples, and pasta all together. To find all the bread you have sold, you have to open every bag and dig around to see whether it is inside.
Column-based storage (like Parquet):
Instead of mixing, you group similar items together:
- One box for all the milk
- One box for all the bread
- One box for all the apples
If you need all the bread you have sold, you simply open the bread box, no need to touch anything else.
That is Parquet in action, organized storage that makes retrieval faster and more efficient.
The Technical View
Parquet is an open-source, columnar storage format developed by Twitter and Cloudera, now an Apache project. It is:
Self-describing – Metadata in the file footer stores schema, number
of rows, and statistics.
Highly compressed – Supports Snappy, Gzip, ZSTD, and more.
Structured internally into:
Row Groups → subsets of rows (horizontal partitions).
Column Chunks → data for each column inside a row group (vertical partitions).
Pages → the smallest unit of storage, making reads efficient.
Magic Number (PAR1) – Ensures the file is recognized as Parquet.
This hybrid layout balances efficient columnar reads with manageable write performance.
Now that we understand Parquet, let us see why it stands out compared to other formats.
- CSV – Human-readable but inefficient; no compression or schema
support.
- JSON – Flexible for nested data, but verbose and large in size.
- Avro – Great for row-based streaming but not optimized for analytics.
- ORC – Similar to Parquet, optimized for Hadoop and Hive workloads.
Parquet vs the rest:
- Much smaller file sizes due to efficient compression
- Faster query performance with column pruning and predicate pushdown
- Widely supported across Spark, Hive, Presto, Trino, BigQuery,
Snowflake, AWS Athena, and more
- Ideal for data lakes and analytical workloads
How Data is Written in Parquet
When writing data into a Parquet file, the process follows a layered structure:
Rows are collected into Row Groups
- Instead of writing each record one by one (like CSV), Parquet groups
multiple rows together into a Row Group (horizontal partition).
Within each Row Group, data is split into Column Chunks
Each column’s values are stored together → Column A values go into
Column Chunk A, Column B values into Column Chunk B, and so on.
This preserves the columnar storage advantage.
Column Chunks are broken into Pages
Pages are the smallest unit of storage in Parquet.
Data Pages → contain the actual column data
Dictionary Pages → optionally store dictionaries used for dictionary encoding
Optional Page Indexes can provide additional information for
efficiently locating and skipping pages:
Column Index → stores statistics such as min/max values for pages
Offset Index → stores the locations and sizes of pages
Metadata is written into the File Footer
Contains schema, row count, statistics (min/max per column), and row
group metadata.
Makes Parquet a self-describing format.
Magic Number (PAR1) is added
- At the start and end of the file → ensures the file can be validated
as a Parquet file.
How Data is Read from a Parquet File
File Footer Read First
Parquet does not scan the entire file right away.
It jumps directly to the footer, where all the metadata is stored.
Metadata tells the engine:
Schema (what columns exist)
Number of rows
Statistics for each column in each row group (min/max)
Row group and column chunk offsets (where the data lives in the file)
➡️ This helps the engine decide which parts of the file to actually read.
Predicate Pushdown & Column Pruning
If the query only needs certain columns (SELECT name, age), only
those column chunks are accessed.
If the query has filters (WHERE age > 30), Parquet uses min/max stats
in metadata to skip entire row groups that don’t match.
➡️ This reduces I/O drastically compared to row-based formats.
Reading Row Groups
The engine opens only the relevant row groups.
Inside each row group, it locates the column chunks needed for the
query.
Decompression & Decoding Pages
Column chunks are split into pages.
Each page is decompressed (Snappy, Gzip, ZSTD, etc.) and decoded
(Dictionary, RLE, Delta encoding).
Pages are read into memory one at a time, which is efficient.
Reconstructing Results
Once the engine reads, decompresses, and decodes the relevant column
data, it reconstructs the requested rows and columns.
The engine materializes the final output as a result set for the
query.
Parquet File Size: Small vs Large
The size of each Parquet file matters a lot for performance.
Too Small Files (< 50 MB each)
- Too much metadata and file-open operations.
- Increases overhead for query engines (Spark, Presto, Hive).
- Known as the small files problem.
Too Large Files (> 1 GB each)
- Slows down reads and reduces parallelism.
💡 Ideal File Size:
- 256 MB to 1 GB for most big data frameworks.
- Large enough to compress well, small enough for parallelism.
How Data Types Affect Compression (and File Size)
Parquet’s efficiency is heavily influenced by data type selection:
- Strings vs Integers: Integers compress far better than storing
numbers as strings.
- Decimals vs Floats: Decimals (fixed precision) often compress better.
- Categorical Data: Repeated values (like “USA”) compress extremely
well with dictionary encoding.
- Timestamps: Storing as INT64 (epoch values) is more efficient than strings.
Tip: Choose the most precise and smallest data type possible to save space and improve query speed.
Encoding Strategies used in Parquet
Run Length Encoding (RLE)
How it works: Stores consecutive repeated values as a single value + a count.
Best for: Columns with long runs of the same value (e.g., flags, booleans, country/state codes).
Benefit: Extremely efficient for repetitive data, reduces size drastically.
Example:
Raw → A A A A B B C C C
RLE → (A,4), (B,2), (C,3)
Dictionary Encoding
How it works: Builds a dictionary of unique values in a column and replaces each value with a small index reference.
Best for: Low-cardinality columns (few unique values, but repeated often).
Benefit: Saves space when values repeat often, lookups are fast.
Example:
Raw → USA, USA, CAN, USA, CAN, MEX
Dictionary → {0: USA, 1: CAN, 2: MEX}
Encoded → 0, 0, 1, 0, 1, 2
Delta Encoding
How it works: Stores the difference (delta) between consecutive values instead of storing full values.
Best for: Sequential/numeric data like timestamps, IDs, counters.
Benefit: Great for compressing increasing sequences, works well with integers and dates.
Example:
Raw → 100, 105, 110, 115
Delta → 100, +5, +5, +5
Parquet Optimization & Best Practices
Predicate Pushdown – Enable in your query engine to filter row groups
at scan time.
Compression Codecs
Snappy: Fast, decent ratio
Gzip: Higher compression, slower
ZSTD: Good balance
Avoid Small Files – Merge into larger files to reduce metadata
overhead.
Row Group Size – Balance between compression (bigger row groups) and
parallelism (smaller ones).
Encoding Strategies
Dictionary Encoding for low-cardinality strings
Run-Length Encoding for repeated values
Delta Encoding for incremental numbers like timestamps
Sort Data – Sorting on commonly filtered columns improves compression
and predicate pushdown.
ACID Transactions – Parquet itself does not provide ACID, use table formats like Delta Lake, Iceberg, or Hudi for transactional
guarantees.
Conclusion
Parquet has become a default choice in the modern data stack because it offers the best of both worlds like compact storage and fast analytics.
By understanding how it works internally (row groups, column chunks, pages) and following best practices (ideal file sizes, data types, compression, and encodings), you can save costs, improve query speeds, and build scalable pipelines.
Further Reading
For a deeper dive into Parquet’s internals, file layout, and optimizations, check out “The Overview of Parquet File Format” by Vu Trinh. It’s a clear and insightful read for data engineers looking to go beyond the basics.
In short, Parquet is the “organized pantry” of the data world: efficient, structured, and always ready when you need it.