[jira] [Commented] (SPARK-51727) SPIP: Declarative Pipelines

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-51727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18036737#comment-18036737 ] 

Dongjoon Hyun commented on SPARK-51727:
---------------------------------------

Hi, [~sandyr]. Could you share the status of this umbrella JIRA issue?

> SPIP: Declarative Pipelines
> ---------------------------
>
>                 Key: SPARK-51727
>                 URL: https://issues.apache.org/jira/browse/SPARK-51727
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core
>    Affects Versions: 4.1.0
>            Reporter: Sandy Ryza
>            Priority: Major
>              Labels: releasenotes
>
> The SPIP proposes a new abstraction that combines multiple transformations into a single declarative dataflow graph, to simplify the development and management of data pipelines. 
>  
> The approach extends Spark's lazy, declarative execution model beyond single queries, to pipelines that keep multiple datasets up to date. This reduces cognitive overhead and the need for manual orchestration of dependencies.
>  
> Declarative pipelines can include both batch and streaming computations, leveraging Spark Streaming for stream processing and new materialized view syntax for batch processing.
>  
> SPIP doc: [https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org