Hudi MOR Compaction Lag: How to Monitor It Without Touching the Writer

Apache Hudi MOR (Merge-on-Read) tables accumulate delta log files between compaction runs. As the log-to-base-file ratio grows, read amplification increases. Every query must merge more log files before returning results. Here is how to monitor compaction lag without instrumenting your Spark or Flink writers.

How Hudi MOR tables work

Hudi MOR tables store writes in two layers: base files (Parquet/ORC snapshots of the table state) and delta log files (incremental updates since the last compaction). Readers merge base files with all subsequent log files to produce the current view. The compaction process collapses log files back into base files. When compaction runs frequently, readers merge few log files and reads are fast. When compaction lags, readers merge many log files and reads slow down proportionally.

What compaction lag looks like in S3

Hudi stores its timeline in the .hoodie/ prefix alongside each table. Compaction timeline entries (*.compaction.requested, *.compaction.inflight, *.compaction.commit) appear as S3 objects. By monitoring the age of the latest *.compaction.commit entry and counting the number of delta log files per partition, you can measure compaction lag without running a Hudi API call.

• Count .log files per partition prefix in S3 Inventory: the ratio of .log to .parquet files per partition measures compaction debt
• Find the most recent *.compaction.commit timestamp in the .hoodie/ prefix. This is the last successful compaction
• Calculate time since last compaction commit: (now - last_compaction_commit) is the compaction lag
• Alert when compaction lag exceeds your SLO (typical values: 1 hour for OLTP-like Hudi, 6 hours for batch)

Compaction lag thresholds

The severity of compaction lag depends on the write rate. A table receiving 1,000 updates per second accumulates log files faster than a table receiving 100 updates per hour. Meaningful thresholds are relative to write rate: when the log file count per partition exceeds (write_rate_per_hour * lag_hours * avg_log_file_per_write), compaction is behind.

Inline vs async compaction

Hudi supports inline compaction (runs compaction synchronously during write) and async compaction (runs in a separate Spark job). For high-throughput tables, inline compaction adds latency to every write. For those tables, async compaction is preferred but requires monitoring to ensure the async job is actually running.

What reCost tracks for Hudi compaction

• Log-to-base-file ratio per partition per table, updated from S3 Inventory
• Time since last compaction commit, inferred from .hoodie/ timeline file timestamps
• Alert when log-to-base-file ratio exceeds 10:1 in any partition
• Alert when compaction lag exceeds configurable SLO