[jira] [Resolved] (SPARK-51727) SPIP: Declarative Pipelines

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-51727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-51727.
-----------------------------------
    Fix Version/s: 4.1.0
         Assignee: Sandy Ryza
       Resolution: Fixed

I marked this as `Resolved` to start a community QA.

For SPARK-54020, the PR looks almost ready to me. I'll merge it when the CIs pass on it.

Thank you for leading and managing this, [~sandyr].

> SPIP: Declarative Pipelines
> ---------------------------
>
>                 Key: SPARK-51727
>                 URL: https://issues.apache.org/jira/browse/SPARK-51727
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core
>    Affects Versions: 4.1.0
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>            Priority: Major
>              Labels: releasenotes
>             Fix For: 4.1.0
>
>
> The SPIP proposes a new abstraction that combines multiple transformations into a single declarative dataflow graph, to simplify the development and management of data pipelines. 
>  
> The approach extends Spark's lazy, declarative execution model beyond single queries, to pipelines that keep multiple datasets up to date. This reduces cognitive overhead and the need for manual orchestration of dependencies.
>  
> Declarative pipelines can include both batch and streaming computations, leveraging Spark Streaming for stream processing and new materialized view syntax for batch processing.
>  
> SPIP doc: [https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org