Article: Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale

Twilio processes over five trillion monthly records through Apache Hudi pipelines fed by Kafka, facing challenges in measuring data freshness accurately. They developed a time-in-queue metric using Kafka checkpoints from Hudi commits to better monitor data latency and enforce freshness SLAs without impacting pipeline performance.