Databricks debuts FILE type to query unstructured data

Databricks has released the beta version of FILE, a native column type designed to store unstructured data such as video, audio, images, and documents directly inside database tables. Instead of saving heavy file binaries directly in the table, which degrades query speeds, the FILE type stores lightweight pointers. The system only retrieves and processes the actual binary data when a query explicitly demands it, maintaining high performance even when handling gigabytes of video or audio.

This integration brings unstructured data under the same governance umbrella as structured data. By linking the new column type with Databricks Unity Catalog, organizations can apply fine-grained security policies, attribute-based access control, and row- or column-level permissions. Furthermore, the lifecycle of the unstructured file is tied directly to its table row. If a row is deleted, the corresponding binary file in cloud object storage is automatically erased, simplifying compliance with privacy regulations like GDPR.

The new column type supports standard SQL and Python user-defined functions, allowing developers to build incremental pipelines. Users can ingest data from sources like Google Drive and SharePoint using Lakeflow connectors, or reference files in place without moving them. Through Spark Declarative Pipelines, teams can build automated, incremental multimodal pipelines. Databricks also supports native AI functions, such as AI_PARSE_DOCUMENT and AI_QUERY, to extract structured insights directly from raw files.

To prevent vendor lock-in, Databricks is collaborating with the open-source community to integrate FILE support directly into Parquet, Delta Lake, Iceberg, and Spark. Looking ahead, the company plans to introduce features that allow teams to stream datasets directly into PyTorch, clone datasets without duplicating underlying binaries, and perform feature engineering without rewriting entire tables.