Introduction to Delta Lake Tables in Microsoft Fabric

The Foundation of Lakehouse Tables

Tables within a Microsoft Fabric lakehouse are built upon the Linux Foundation’s Delta Lake table format, a staple in the Apache Spark ecosystem. Delta Lake serves as an open-source storage layer for Spark, empowering relational database capabilities for both batch and streaming data. By leveraging Delta Lake, you can establish a robust lakehouse architecture that supports SQL-based data manipulation semantics in Spark, complete with transaction handling and schema enforcement. The outcome is an analytical data store that combines the strengths of a relational database system with the adaptability of data file storage inherent to a data lake.

Working with Delta Lake in Fabric

While direct interaction with Delta Lake APIs isn’t mandatory to utilize tables in a Fabric lakehouse, understanding the Delta Lake metastore architecture and familiarizing yourself with specialized Delta table operations can significantly enhance your ability to construct sophisticated analytics solutions on Microsoft Fabric.

Key Takeaways:

Microsoft Fabric lakehouse tables are powered by the Delta Lake format.
Delta Lake brings relational database features to Spark, including transactions and schema enforcement.
This combination allows for a lakehouse architecture, merging the benefits of data lakes and relational databases.
While not essential for basic table usage, understanding Delta Lake’s inner workings unlocks advanced analytics capabilities.

In the upcoming sections, we’ll dive deeper into the Delta Lake metastore and explore some specialized Delta table operations.

Understand Delta Lake

Relational Capabilities for Data Lakes

Delta Lake is an open-source storage layer that brings the power of relational database semantics to Spark-based data lake processing. In essence, tables within Microsoft Fabric lakehouses are Delta tables, visually denoted by the triangular Delta (▴) icon in the lakehouse user interface.

Structure of Delta Tables

Delta tables function as schema abstractions layered over data files stored in the Delta format. For each table, the lakehouse maintains a folder containing Parquet data files and a _delta_log folder where transaction details are recorded in JSON format.

Key Benefits of Delta Tables

Relational Table Support: Delta tables within Apache Spark enable you to perform familiar CRUD (create, read, update, and delete) operations, just like in traditional relational database systems. You can SELECT, INSERT, UPDATE, and DELETE rows of data with ease.
ACID Transaction Support: Relational databases are designed to handle transactional data modifications that uphold ACID properties: atomicity (transactions complete as a single unit), consistency (transactions maintain database consistency), isolation (concurrent transactions don’t interfere), and durability (committed changes persist). Delta Lake extends this transactional support to Spark through a transaction log and serializable isolation enforcement for concurrent operations.
Data Versioning and Time Travel: The transaction log meticulously tracks changes, allowing you to trace multiple versions of each table row. This capability enables the powerful “time travel” feature, allowing you to retrieve previous row versions within queries.
Batch and Streaming Data Support: While traditional databases typically store static data, Spark natively supports streaming data through its Structured Streaming API. Delta Lake tables seamlessly act as both sinks (destinations) and sources for streaming data.
Standard Formats and Interoperability: Delta tables’ underlying data is stored in the widely used Parquet format, commonly employed in data lake ingestion pipelines. Furthermore, you can query Delta tables using SQL through the Microsoft Fabric lakehouse’s SQL analytics endpoint.

By understanding these key aspects of Delta Lake, you’ll gain a deeper appreciation for the capabilities and advantages of working with tables in a Microsoft Fabric lakehouse. In the next section, we’ll dive into the Delta Lake metastore architecture and explore how it contributes to the lakehouse’s robust functionality.

This blog post is based on information and concepts derived from the Microsoft Learn module titled “Work with Delta Lake tables in Microsoft Fabric.” The original content can be found here:
https://learn.microsoft.com/en-us/training/modules/work-delta-lake-tables-fabric/