BigQuery adds first-party support for Delta Lake
6 juin 2024 à 04:42
BigQuery, now with first-party support for Delta Lake, grows Delta Lake’s vibrant connector ecosystem and simplifies its integration with Databricks.
Vue normale
À partir d’avant-hierFlux principal
How we improved DatabricksIQ LLM quality for AI-generated table comments
29 avril 2024 à 23:30
We recently made significant improvements to the underlying algorithms supporting AI-generated comments in Unity Catalog and we’re excited to share our results. Through...
Announcing the State Reader API: The New "Statestore" Data Source
28 mars 2024 à 13:06
Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming 's internal state data: the State Reader...
PySpark in 2023: A Year in Review
25 mars 2024 à 15:19
With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use...
GGML GGUF File Format Vulnerabilities
22 mars 2024 à 11:23
The GGUF file format is a binary file format used for storing and loading model weights for the GGML library. The library documentation...
Simplify PySpark testing with DataFrame equality functions
6 mars 2024 à 11:28
The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set...
-
Databricks
- A Deep Dive into the Latest Performance Improvements of Stateful Pipelines in Apache Spark Structured Streaming
A Deep Dive into the Latest Performance Improvements of Stateful Pipelines in Apache Spark Structured Streaming
28 février 2024 à 14:17
This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this...
Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming
28 février 2024 à 08:46
Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the...
Announcing Ray Autoscaling support on Databricks and Apache Spark™
9 janvier 2024 à 13:15
Ray is an open-source unified compute framework that simplifies scaling AI and Python workloads in a distributed environment. Since we introduced support for...
