Benchmarking Delta Lake vs. Delta UniForm (Iceberg Metadata) via Shortcuts in Microsoft Fabric

 

Introduction

With open data lakehouse architectures evolving rapidly, Delta Lake and Apache Iceberg compatibility has reached a major milestone with Delta UniForm (Universal Format) in Databricks. Delta UniForm generates Iceberg metadata alongside standard Delta transaction logs without physically duplicating Parquet data. This allows query engines that speak Iceberg to read Delta tables seamlessly.

By leveraging Microsoft Fabric Lakehouse Shortcuts, we can expose external Databricks tables—both standard Delta tables and Delta UniForm tables—directly into Fabric without moving underlying data.

To evaluate execution characteristics and metadata overhead, a benchmark was conducted in Microsoft Fabric running PySpark queries over standard Delta tables vs. Delta UniForm (accessed via Iceberg metadata) using Fabric shortcuts.

Architecture & Benchmark Design

The architecture test compares two representations of the same underlying cost dataset created in Databricks and shortcutted into a Microsoft Fabric Lakehouse:

1. Standard Delta Table (Cost_Delta): Traditional Delta Lake table read via standard Delta log files.
2. Delta UniForm Table (Cost_Iceburg): Delta table created with UniForm enabled in Databricks, generating Iceberg metadata files and queried via Iceberg metadata in Fabric.

Test Environment & Methodology

• Engine: Microsoft Fabric Spark Engine / Lakehouse Shortcuts
• Cache Management: Initial session cache cleared prior to loop execution (spark.catalog.clearCache())
• Iterations: 10 sequential query runs per table format
• Workload: SQL aggregation with filtering (x_SkuRegion = 'UK South'), grouping, and sorting over financial tracking data.
• Table Metadata: 5 GB table with 100M+ records

PySpark Benchmark Implementation

The benchmark loops 10 times over both shortcut endpoints sequentially, collecting execution times and summary statistical metrics.

Benchmark Results

Key Performance & Architectural Insights

1. Equivalent Warm Cache Runtime
When looking at the minimum query execution times (0.469s for Delta vs. 0.457s for UniForm), execution speeds are practically identical. Since both formats ultimately point to the same underlying Parquet data files, execution time once metadata is cached shows zero penalty for using Universal Format.

2. Cold-Start Latency Variance
Standard Delta experienced a substantial cold-start latency spike on initial query execution (3.146s), pushing its average duration to 0.839s with a high standard deviation (0.822s). In Fabric, initial delta log checkpoint parsing and shortcut initialization can introduce one-off overhead.

 
3. Metadata Traversal Consistency in UniForm
Reading Delta UniForm via Iceberg metadata delivered near-flat consistency across all 10 runs. Execution times ranged strictly between 0.457s and 0.503s with a standard deviation of 0.015s. Iceberg’s manifest file hierarchy allowed Fabric's reader engine to resolve metadata without initial log-parsing overhead.

Conclusion

Delta UniForm transforms cross-platform lakehouse integration from a complex ETL headache into a simple metadata configuration. By generating Iceberg metadata directly alongside Delta transaction logs in Databricks, Fabric shortcuts can query external datasets without copying a single byte of underlying Parquet data.

The benchmark results clearly show that while both formats achieve identical throughput once warmed up (~0.46s minimum execution time), leveraging UniForm's Iceberg metadata pathway in Fabric eliminates the initial metadata parsing overhead that caused Delta’s 3.15-second cold-start spike. Delivering near-flat execution times across all runs (std dev of 0.015s) makes UniForm an exceptional choice for workloads where consistent latency is critical.

Data teams no longer face a forced trade-off between standardizing on Delta Lake or rewriting pipelines for Apache Iceberg. Enabling Delta UniForm allows you to keep Databricks as your heavy-lifting write engine while seamlessly serving Fabric consumers with high predictability and zero storage duplication.