Apache Parquet Explained: Writing Efficient Parquet Files in Production

Choosing Apache Parquet as your storage format is only the first step. The way Parquet files are written has an equally significant impact on performance.

Many teams adopt Parquet expecting fast queries, only to discover that their jobs remain slow because the data has been partitioned poorly, written into thousands of tiny files, or compressed using an unsuitable codec.

• Query performance
• Storage efficiency
• Parallel processing
• Scalability
• Long-term maintenance

Let us explore the practices that experienced data engineers follow in production environments.

Choosing the right partitioning strategy

Partitioning divides a dataset into separate directories based on one or more column values. Rather than scanning an entire dataset, query engines can directly access only the relevant partitions.

Consider a sales dataset.

sales.parquet

With partitioning by year and month:

sales/

├── year=2024/
│      ├── month=01/
│      ├── month=02/
│      └── month=03/
│
├── year=2025/
│      ├── month=01/
│      ├── month=02/
│      └── month=03/

Now suppose a query requests only January 2025 data.

SELECT *
FROM sales
WHERE year = 2025
AND month = 1;

Instead of scanning every file, Spark reads only:

year=2025/month=01/

Every other partition is skipped. This optimization is called Partition Pruning, and it dramatically reduces the amount of data that needs to be read.

Avoid over-partitioning

Although partitioning improves query performance, more partitions do not always produce better results. Imagine partitioning customer data using customer_id.

customer_id=1

customer_id=2

customer_id=3

...

customer_id=950000

This creates hundreds of thousands of directories, each containing very little data. Problems quickly appear:

• Large metadata overhead
• Slower file discovery
• Increased storage API calls
• Reduced query performance
• Longer job startup times

A good partition column should have moderate cardinality. Examples include year, month, country, region, and business unit. Avoid highly unique values such as user IDs, order IDs, UUIDs, email addresses, and transaction IDs.

Understanding the small files problem

One of the most common issues in modern data lakes is the Small Files Problem. Imagine a streaming application writing a new Parquet file every few seconds.

orders/

00001.parquet

00002.parquet

00003.parquet

...

983742.parquet

Although the total amount of data may be reasonable, query performance suffers because the processing engine must discover every file, open every file, read metadata from every file, and schedule thousands of individual tasks.

Many organizations discover that millions of tiny Parquet files slow queries more than a handful of properly sized files.

Choosing an appropriate file size

Instead of generating thousands of tiny files, production pipelines typically aim for fewer, larger Parquet files.

| File Size       | Recommendation                                                       |
| --------------- | -------------------------------------------------------------------- |
| Less than 32 MB | Too small for most analytical workloads                              |
| 128-512 MB      | A practical target for many Spark workloads                          |
| 512 MB-1 GB     | Suitable for large-scale batch processing, depending on the platform |
| Several GB      | May reduce parallelism and increase recovery costs                   |

Rather than focusing on an exact number, aim for file sizes that balance efficient scanning with enough parallelism for your processing engine. The optimal size can vary depending on storage system, network bandwidth, cluster resources, and workload characteristics.

Repartition vs coalesce in Apache Spark

When writing Parquet files with Spark, controlling the number of output files is extremely important. Spark provides two commonly used transformations.

Using repartition

val optimizedDF = df.repartition(200)

repartition redistributes data across the cluster, resulting in a more balanced workload. It performs a full shuffle, making it useful when increasing or significantly changing the number of partitions.

• Preparing data for large writes
• Rebalancing skewed datasets
• Improving parallelism

Using coalesce

val optimizedDF = df.coalesce(20)

Unlike repartition, coalesce attempts to reduce the number of partitions with minimal data movement. It is commonly used before writing output files, when reducing partition counts, and to avoid unnecessary shuffle operations.

Selecting the right compression codec

Compression has a direct impact on both storage consumption and query performance. Parquet supports several compression codecs, each with different trade-offs.

| Codec  | Compression Ratio | Read Speed | Write Speed | Recommended For               |
| ------ | ----------------- | ---------- | ----------- | ----------------------------- |
| Snappy | Medium            | Excellent  | Excellent   | General-purpose analytics     |
| Gzip   | High              | Moderate   | Slower      | Archival workloads            |
| ZSTD   | High              | Very Good  | Good        | Balanced production workloads |
| LZ4    | Lower             | Excellent  | Excellent   | Low-latency processing        |

In many Spark deployments, Snappy remains the default choice because it offers fast decompression with reasonable storage savings. However, modern workloads increasingly adopt Zstandard where supported, as it often provides a strong balance between compression efficiency and performance.

The best codec depends on your priorities, whether you optimize for storage costs, query speed, or write throughput.

Sorting data before writing

Sorting data before writing Parquet files can further improve query performance. Suppose transaction data is sorted by date before being written.

2025-01-01

2025-01-02

2025-01-03

...

2025-12-31

Now consider this query:

SELECT *
FROM transactions
WHERE transaction_date = '2025-08-10';

Because values are clustered together, metadata such as minimum and maximum values becomes more effective. Query engines can skip larger portions of the dataset using predicate pushdown and statistics, resulting in fewer disk reads.

Sorting is particularly beneficial for columns that are frequently used in filtering operations.

Monitoring data skew

Even with proper partitioning, uneven data distribution can become a bottleneck. Consider sales data where one country contains 90% of all records while every other country contributes only a small fraction.

USA      ███████████████████████

Canada   ██

Germany  █

Japan    █

If the dataset is partitioned only by country, one partition becomes disproportionately large while others remain almost empty.

• Longer-running tasks
• Underutilized cluster resources
• Increased shuffle times
• Poor overall job performance

Regularly monitoring partition sizes and identifying skewed keys is an important part of maintaining healthy production pipelines.

Production best practices

The following recommendations are widely adopted across modern data engineering teams:

• Use partition columns that align with common query filters.
• Avoid creating partitions with extremely high cardinality.
• Periodically compact small files into larger ones.
• Select a compression codec based on workload requirements rather than defaults alone.
• Keep schemas consistent across writes to simplify downstream processing.
• Monitor partition sizes and address data skew early.
• Validate file quality as part of your ingestion pipeline.
• Benchmark performance after making storage or partitioning changes instead of assuming improvements.

Following these practices helps ensure that Parquet datasets remain efficient as they grow from gigabytes to petabytes.

What is next?

We have learned how to produce efficient Parquet files, but understanding how query engines use those files is equally important. In the next section, we will focus on Apache Spark execution and explore vectorized Parquet readers, Catalyst Optimizer, whole-stage code generation, Adaptive Query Execution, filter pushdown, partition pruning, and Spark execution plans.

These concepts explain why Spark can execute complex analytical queries against Parquet datasets with remarkable efficiency.