Skip to content
Live
Trending storyPast story

Spark streaming mode is officially available and supports stateful processing

2 reports1 outlets14 days agoUpdate

The story so far

AI overview

Databricks announced that the Real-Time Mode of Spark Declarative Pipelines is now officially available, stating that this mode achieves millisecond-level end-to-end latency through continuous streaming processing, parallel query phases, and immediate data delivery. Subsequently released Apache Spark 4.3 is available in Databricks Runtime 19, with Real-Time Mode adding stateful processing in this version, supporting stream aggregation, dropDuplicates, and transformWithState. Databricks claims that in three representative stateful benchmark tests, the end-to-end latency reaches P99 below 150 milliseconds.

AI-written from the reports · Updated 13 days ago

Developments

2 developments

  1. 5 Nov, 08:00 · 1 reports
    Introducing Apache Spark® 4.3
    Databricks Blog:Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19
  2. 30 Oct, 06:30 · 1 reports
    General Availability of Real-Time Mode for Spark Declarative
    Databricks Blog:Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

Coverage timeline

Follow the coverage to see every side of the story.

5 Nov
  1. Databricks BlogFeatured
    Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

    Apache Spark 4.3 was released and is available in Databricks Runtime 19, with the Real-Time Mode now adding stateful processing, supporting streaming aggregation, dropDuplicates, and transformWithState.

30 Oct
  1. Databricks Blog
    Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

    Databricks announces the general availability of Spark Declarative Pipelines Real-Time Mode, which achieves millisecond-level end-to-end latency through continuous streaming, parallel query stages, and instant data delivery.