Indexing the Data Lake for Online Point Queries

Spotify faces challenges in providing low-latency access to vast user data stored in exabyte-scale data lakes for online services and AI agents. They use Random Access Parquet (RAP) indexing to enable fast point queries over large Parquet datasets, reducing overhead from traditional SQL engines and avoiding data duplication.