You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sandy Ryza <sa...@apache.org> on 2025/04/05 21:30:09 UTC
[DISCUSS] SPIP: Declarative Pipelines
Hi all – starting a discussion thread for a SPIP that I've been working on
with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
<https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
<https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
].
The SPIP proposes extending Spark's lazy, declarative execution model
beyond single queries, to pipelines that keep multiple datasets up to date.
It introduces the ability to compose multiple transformations into a single
declarative dataflow graph.
Declarative pipelines aim to simplify the development and management of
data pipelines, by removing the need for manual orchestration of
dependencies and making it possible to catch many errors before any
execution steps are launched.
Declarative pipelines can include both batch and streaming computations,
leveraging Structured Streaming for stream processing and new materialized
view syntax for batch processing. Tight integration with Spark SQL's
analyzer enables deeper analysis and earlier error detection than is
achievable with more generic frameworks.
Let us know what you think!
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Kent Yao <ya...@apache.org>.
+1
Kent Yao
Sem <ss...@apache.org> 于2025年4月9日周三 14:08写道：
> +1 (non-binding)
>
>
> On April 9, 2025 7:29:40 AM GMT+02:00, Rishab Joshi <ri...@gmail.com>
> wrote:
>
>> +1 Exciting.
>> Rishab Joshi
>>
>> On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng <ru...@apache.org> wrote:
>>
>>> +1
>>>
>>> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee <de...@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang <yu...@apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <
>>>>> kabhwan.opensource@gmail.com> wrote:
>>>>>
>>>>>> +1 looking forward to seeing this make progress!
>>>>>>
>>>>>> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>>>> > +1
>>>>>>> >
>>>>>>> > I am actually pretty excited to have this. Happy to see this being
>>>>>>> proposed.
>>>>>>> >
>>>>>>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>>>>> >
>>>>>>> > > +1. Super excited about this effort!
>>>>>>> > >
>>>>>>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>> > >
>>>>>>> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>>>> management and
>>>>>>> > >> enhances error detection.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >>> Excited to see this heading toward open source — materialized
>>>>>>> views and
>>>>>>> > >>> other features will bring a lot of value.
>>>>>>> > >>> +1 (non-binding)
>>>>>>> > >>>
>>>>>>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>>>>> wrote:
>>>>>>> > >>>
>>>>>>> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>>>>> built on
>>>>>>> > >>>> top of internal APIs for constructing and launching pipeline
>>>>>>> executions.
>>>>>>> > >>>> We'll have the option to expose these in the future.
>>>>>>> > >>>>
>>>>>>> > >>>> It would be worthwhile to understand the use cases in more
>>>>>>> depth before
>>>>>>> > >>>> exposing these, because APIs are one-way doors and can be
>>>>>>> costly to
>>>>>>> > >>>> maintain.
>>>>>>> > >>>>
>>>>>>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>>>> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>>>> > >>>>
>>>>>>> > >>>>> Looks great!
>>>>>>> > >>>>> QQ: will user able to run this pipeline from normal code?
>>>>>>> I.e. can I
>>>>>>> > >>>>> trigger a pipeline from *driver* code based on some
>>>>>>> condition etc. or
>>>>>>> > >>>>> it must be executed via separate shell command ?
>>>>>>> > >>>>> As a background Databricks imposes similar limitation where
>>>>>>> as you
>>>>>>> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>>>>>> some reason
>>>>>>> > >>>>> and forces to use two clusters increasing the cost and
>>>>>>> latency.
>>>>>>> > >>>>>
>>>>>>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>>>> wrote:
>>>>>>> > >>>>>
>>>>>>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>>>>>> been
>>>>>>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>>>> Yang: [JIRA
>>>>>>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>>>> > >>>>>> <
>>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>>>> >
>>>>>>> > >>>>>> ].
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>>>> execution model
>>>>>>> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>>>> datasets up to date.
>>>>>>> > >>>>>> It introduces the ability to compose multiple
>>>>>>> transformations into a single
>>>>>>> > >>>>>> declarative dataflow graph.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Declarative pipelines aim to simplify the development and
>>>>>>> management
>>>>>>> > >>>>>> of data pipelines, by removing the need for manual
>>>>>>> orchestration of
>>>>>>> > >>>>>> dependencies and making it possible to catch many errors
>>>>>>> before any
>>>>>>> > >>>>>> execution steps are launched.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Declarative pipelines can include both batch and streaming
>>>>>>> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>>>> processing and new
>>>>>>> > >>>>>> materialized view syntax for batch processing. Tight
>>>>>>> integration with Spark
>>>>>>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>>>> detection than is
>>>>>>> > >>>>>> achievable with more generic frameworks.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Let us know what you think!
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Sem <ss...@apache.org>.
+1 (non-binding)
On April 9, 2025 7:29:40 AM GMT+02:00, Rishab Joshi <ri...@gmail.com> wrote:
>+1 Exciting.
>Rishab Joshi
>
>On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng <ru...@apache.org> wrote:
>
>> +1
>>
>> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee <de...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang <yu...@apache.org> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com> wrote:
>>>>
>>>>> +1 looking forward to seeing this make progress!
>>>>>
>>>>> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>>> > +1
>>>>>> >
>>>>>> > I am actually pretty excited to have this. Happy to see this being
>>>>>> proposed.
>>>>>> >
>>>>>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>>>> >
>>>>>> > > +1. Super excited about this effort!
>>>>>> > >
>>>>>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>>> management and
>>>>>> > >> enhances error detection.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >>> Excited to see this heading toward open source — materialized
>>>>>> views and
>>>>>> > >>> other features will bring a lot of value.
>>>>>> > >>> +1 (non-binding)
>>>>>> > >>>
>>>>>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>>>> wrote:
>>>>>> > >>>
>>>>>> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>>>> built on
>>>>>> > >>>> top of internal APIs for constructing and launching pipeline
>>>>>> executions.
>>>>>> > >>>> We'll have the option to expose these in the future.
>>>>>> > >>>>
>>>>>> > >>>> It would be worthwhile to understand the use cases in more
>>>>>> depth before
>>>>>> > >>>> exposing these, because APIs are one-way doors and can be
>>>>>> costly to
>>>>>> > >>>> maintain.
>>>>>> > >>>>
>>>>>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>>> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>>> > >>>>
>>>>>> > >>>>> Looks great!
>>>>>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>>>>>> can I
>>>>>> > >>>>> trigger a pipeline from *driver* code based on some condition
>>>>>> etc. or
>>>>>> > >>>>> it must be executed via separate shell command ?
>>>>>> > >>>>> As a background Databricks imposes similar limitation where as
>>>>>> you
>>>>>> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>>>>> some reason
>>>>>> > >>>>> and forces to use two clusters increasing the cost and latency.
>>>>>> > >>>>>
>>>>>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>>> wrote:
>>>>>> > >>>>>
>>>>>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>>>>> been
>>>>>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>>> Yang: [JIRA
>>>>>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>>> > >>>>>> <
>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>>> >
>>>>>> > >>>>>> ].
>>>>>> > >>>>>>
>>>>>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>>> execution model
>>>>>> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>>> datasets up to date.
>>>>>> > >>>>>> It introduces the ability to compose multiple transformations
>>>>>> into a single
>>>>>> > >>>>>> declarative dataflow graph.
>>>>>> > >>>>>>
>>>>>> > >>>>>> Declarative pipelines aim to simplify the development and
>>>>>> management
>>>>>> > >>>>>> of data pipelines, by removing the need for manual
>>>>>> orchestration of
>>>>>> > >>>>>> dependencies and making it possible to catch many errors
>>>>>> before any
>>>>>> > >>>>>> execution steps are launched.
>>>>>> > >>>>>>
>>>>>> > >>>>>> Declarative pipelines can include both batch and streaming
>>>>>> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>>> processing and new
>>>>>> > >>>>>> materialized view syntax for batch processing. Tight
>>>>>> integration with Spark
>>>>>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>>> detection than is
>>>>>> > >>>>>> achievable with more generic frameworks.
>>>>>> > >>>>>>
>>>>>> > >>>>>> Let us know what you think!
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> >
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Rishab Joshi <ri...@gmail.com>.
+1 Exciting.
Rishab Joshi
On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng <ru...@apache.org> wrote:
> +1
>
> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee <de...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang <yu...@apache.org> wrote:
>>
>>> +1
>>>
>>> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com> wrote:
>>>
>>>> +1 looking forward to seeing this make progress!
>>>>
>>>> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>> > +1
>>>>> >
>>>>> > I am actually pretty excited to have this. Happy to see this being
>>>>> proposed.
>>>>> >
>>>>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>>> >
>>>>> > > +1. Super excited about this effort!
>>>>> > >
>>>>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>> > >
>>>>> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>> management and
>>>>> > >> enhances error detection.
>>>>> > >>
>>>>> > >>
>>>>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>>>> wrote:
>>>>> > >>
>>>>> > >>> Excited to see this heading toward open source — materialized
>>>>> views and
>>>>> > >>> other features will bring a lot of value.
>>>>> > >>> +1 (non-binding)
>>>>> > >>>
>>>>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>>> wrote:
>>>>> > >>>
>>>>> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>>> built on
>>>>> > >>>> top of internal APIs for constructing and launching pipeline
>>>>> executions.
>>>>> > >>>> We'll have the option to expose these in the future.
>>>>> > >>>>
>>>>> > >>>> It would be worthwhile to understand the use cases in more
>>>>> depth before
>>>>> > >>>> exposing these, because APIs are one-way doors and can be
>>>>> costly to
>>>>> > >>>> maintain.
>>>>> > >>>>
>>>>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>> > >>>>
>>>>> > >>>>> Looks great!
>>>>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>>>>> can I
>>>>> > >>>>> trigger a pipeline from *driver* code based on some condition
>>>>> etc. or
>>>>> > >>>>> it must be executed via separate shell command ?
>>>>> > >>>>> As a background Databricks imposes similar limitation where as
>>>>> you
>>>>> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>>>> some reason
>>>>> > >>>>> and forces to use two clusters increasing the cost and latency.
>>>>> > >>>>>
>>>>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>> wrote:
>>>>> > >>>>>
>>>>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>>>> been
>>>>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>> Yang: [JIRA
>>>>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>> > >>>>>> <
>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>> >
>>>>> > >>>>>> ].
>>>>> > >>>>>>
>>>>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>> execution model
>>>>> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>> datasets up to date.
>>>>> > >>>>>> It introduces the ability to compose multiple transformations
>>>>> into a single
>>>>> > >>>>>> declarative dataflow graph.
>>>>> > >>>>>>
>>>>> > >>>>>> Declarative pipelines aim to simplify the development and
>>>>> management
>>>>> > >>>>>> of data pipelines, by removing the need for manual
>>>>> orchestration of
>>>>> > >>>>>> dependencies and making it possible to catch many errors
>>>>> before any
>>>>> > >>>>>> execution steps are launched.
>>>>> > >>>>>>
>>>>> > >>>>>> Declarative pipelines can include both batch and streaming
>>>>> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>> processing and new
>>>>> > >>>>>> materialized view syntax for batch processing. Tight
>>>>> integration with Spark
>>>>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>> detection than is
>>>>> > >>>>>> achievable with more generic frameworks.
>>>>> > >>>>>>
>>>>> > >>>>>> Let us know what you think!
>>>>> > >>>>>>
>>>>> > >>>>>>
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Ruifeng Zheng <ru...@apache.org>.
+1
On Wed, Apr 9, 2025 at 12:57 PM Denny Lee <de...@gmail.com> wrote:
> +1 (non-binding)
>
> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang <yu...@apache.org> wrote:
>
>> +1
>>
>> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> +1 looking forward to seeing this make progress!
>>>
>>> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>>
>>>> +1
>>>>
>>>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>> > +1
>>>> >
>>>> > I am actually pretty excited to have this. Happy to see this being
>>>> proposed.
>>>> >
>>>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>> >
>>>> > > +1. Super excited about this effort!
>>>> > >
>>>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>> > >
>>>> > >> +1 I support this SPIP because it simplifies data pipeline
>>>> management and
>>>> > >> enhances error detection.
>>>> > >>
>>>> > >>
>>>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >>> Excited to see this heading toward open source — materialized
>>>> views and
>>>> > >>> other features will bring a lot of value.
>>>> > >>> +1 (non-binding)
>>>> > >>>
>>>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>> wrote:
>>>> > >>>
>>>> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>> built on
>>>> > >>>> top of internal APIs for constructing and launching pipeline
>>>> executions.
>>>> > >>>> We'll have the option to expose these in the future.
>>>> > >>>>
>>>> > >>>> It would be worthwhile to understand the use cases in more depth
>>>> before
>>>> > >>>> exposing these, because APIs are one-way doors and can be costly
>>>> to
>>>> > >>>> maintain.
>>>> > >>>>
>>>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>> > >>>> khalidmammadov9@gmail.com> wrote:
>>>> > >>>>
>>>> > >>>>> Looks great!
>>>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>>>> can I
>>>> > >>>>> trigger a pipeline from *driver* code based on some condition
>>>> etc. or
>>>> > >>>>> it must be executed via separate shell command ?
>>>> > >>>>> As a background Databricks imposes similar limitation where as
>>>> you
>>>> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>>> some reason
>>>> > >>>>> and forces to use two clusters increasing the cost and latency.
>>>> > >>>>>
>>>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>> wrote:
>>>> > >>>>>
>>>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>>>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
>>>> [JIRA
>>>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>> > >>>>>> <
>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>> >
>>>> > >>>>>> ].
>>>> > >>>>>>
>>>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>> execution model
>>>> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>> datasets up to date.
>>>> > >>>>>> It introduces the ability to compose multiple transformations
>>>> into a single
>>>> > >>>>>> declarative dataflow graph.
>>>> > >>>>>>
>>>> > >>>>>> Declarative pipelines aim to simplify the development and
>>>> management
>>>> > >>>>>> of data pipelines, by removing the need for manual
>>>> orchestration of
>>>> > >>>>>> dependencies and making it possible to catch many errors
>>>> before any
>>>> > >>>>>> execution steps are launched.
>>>> > >>>>>>
>>>> > >>>>>> Declarative pipelines can include both batch and streaming
>>>> > >>>>>> computations, leveraging Structured Streaming for stream
>>>> processing and new
>>>> > >>>>>> materialized view syntax for batch processing. Tight
>>>> integration with Spark
>>>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>> detection than is
>>>> > >>>>>> achievable with more generic frameworks.
>>>> > >>>>>>
>>>> > >>>>>> Let us know what you think!
>>>> > >>>>>>
>>>> > >>>>>>
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Denny Lee <de...@gmail.com>.
+1 (non-binding)
On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang <yu...@apache.org> wrote:
> +1
>
> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> +1 looking forward to seeing this make progress!
>>
>> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>
>>> +1
>>>
>>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>> > +1
>>> >
>>> > I am actually pretty excited to have this. Happy to see this being
>>> proposed.
>>> >
>>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>> >
>>> > > +1. Super excited about this effort!
>>> > >
>>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>>> wrote:
>>> > >
>>> > >> +1 I support this SPIP because it simplifies data pipeline
>>> management and
>>> > >> enhances error detection.
>>> > >>
>>> > >>
>>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>> wrote:
>>> > >>
>>> > >>> Excited to see this heading toward open source — materialized
>>> views and
>>> > >>> other features will bring a lot of value.
>>> > >>> +1 (non-binding)
>>> > >>>
>>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>> wrote:
>>> > >>>
>>> > >>>> Hi Khalid – the CLI in the current proposal will need to be built
>>> on
>>> > >>>> top of internal APIs for constructing and launching pipeline
>>> executions.
>>> > >>>> We'll have the option to expose these in the future.
>>> > >>>>
>>> > >>>> It would be worthwhile to understand the use cases in more depth
>>> before
>>> > >>>> exposing these, because APIs are one-way doors and can be costly
>>> to
>>> > >>>> maintain.
>>> > >>>>
>>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>> > >>>> khalidmammadov9@gmail.com> wrote:
>>> > >>>>
>>> > >>>>> Looks great!
>>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>>> can I
>>> > >>>>> trigger a pipeline from *driver* code based on some condition
>>> etc. or
>>> > >>>>> it must be executed via separate shell command ?
>>> > >>>>> As a background Databricks imposes similar limitation where as
>>> you
>>> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>> some reason
>>> > >>>>> and forces to use two clusters increasing the cost and latency.
>>> > >>>>>
>>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>> wrote:
>>> > >>>>>
>>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
>>> [JIRA
>>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>> > >>>>>> <
>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>> >
>>> > >>>>>> ].
>>> > >>>>>>
>>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution
>>> model
>>> > >>>>>> beyond single queries, to pipelines that keep multiple datasets
>>> up to date.
>>> > >>>>>> It introduces the ability to compose multiple transformations
>>> into a single
>>> > >>>>>> declarative dataflow graph.
>>> > >>>>>>
>>> > >>>>>> Declarative pipelines aim to simplify the development and
>>> management
>>> > >>>>>> of data pipelines, by removing the need for manual
>>> orchestration of
>>> > >>>>>> dependencies and making it possible to catch many errors before
>>> any
>>> > >>>>>> execution steps are launched.
>>> > >>>>>>
>>> > >>>>>> Declarative pipelines can include both batch and streaming
>>> > >>>>>> computations, leveraging Structured Streaming for stream
>>> processing and new
>>> > >>>>>> materialized view syntax for batch processing. Tight
>>> integration with Spark
>>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>> detection than is
>>> > >>>>>> achievable with more generic frameworks.
>>> > >>>>>>
>>> > >>>>>> Let us know what you think!
>>> > >>>>>>
>>> > >>>>>>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Yuming Wang <yu...@apache.org>.
+1
On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <ka...@gmail.com>
wrote:
> +1 looking forward to seeing this make progress!
>
> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>
>> +1
>>
>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> > +1
>> >
>> > I am actually pretty excited to have this. Happy to see this being
>> proposed.
>> >
>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>> >
>> > > +1. Super excited about this effort!
>> > >
>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>> wrote:
>> > >
>> > >> +1 I support this SPIP because it simplifies data pipeline
>> management and
>> > >> enhances error detection.
>> > >>
>> > >>
>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>> wrote:
>> > >>
>> > >>> Excited to see this heading toward open source — materialized views
>> and
>> > >>> other features will bring a lot of value.
>> > >>> +1 (non-binding)
>> > >>>
>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>> wrote:
>> > >>>
>> > >>>> Hi Khalid – the CLI in the current proposal will need to be built
>> on
>> > >>>> top of internal APIs for constructing and launching pipeline
>> executions.
>> > >>>> We'll have the option to expose these in the future.
>> > >>>>
>> > >>>> It would be worthwhile to understand the use cases in more depth
>> before
>> > >>>> exposing these, because APIs are one-way doors and can be costly to
>> > >>>> maintain.
>> > >>>>
>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> > >>>> khalidmammadov9@gmail.com> wrote:
>> > >>>>
>> > >>>>> Looks great!
>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>> can I
>> > >>>>> trigger a pipeline from *driver* code based on some condition
>> etc. or
>> > >>>>> it must be executed via separate shell command ?
>> > >>>>> As a background Databricks imposes similar limitation where as you
>> > >>>>> cannot run normal Spark code and DLT on the same cluster for some
>> reason
>> > >>>>> and forces to use two clusters increasing the cost and latency.
>> > >>>>>
>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>> > >>>>>
>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
>> [JIRA
>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> > >>>>>> <
>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>> >
>> > >>>>>> ].
>> > >>>>>>
>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution
>> model
>> > >>>>>> beyond single queries, to pipelines that keep multiple datasets
>> up to date.
>> > >>>>>> It introduces the ability to compose multiple transformations
>> into a single
>> > >>>>>> declarative dataflow graph.
>> > >>>>>>
>> > >>>>>> Declarative pipelines aim to simplify the development and
>> management
>> > >>>>>> of data pipelines, by removing the need for manual
>> orchestration of
>> > >>>>>> dependencies and making it possible to catch many errors before
>> any
>> > >>>>>> execution steps are launched.
>> > >>>>>>
>> > >>>>>> Declarative pipelines can include both batch and streaming
>> > >>>>>> computations, leveraging Structured Streaming for stream
>> processing and new
>> > >>>>>> materialized view syntax for batch processing. Tight integration
>> with Spark
>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>> detection than is
>> > >>>>>> achievable with more generic frameworks.
>> > >>>>>>
>> > >>>>>> Let us know what you think!
>> > >>>>>>
>> > >>>>>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
This sounds quite interesting.
+1 to What Szheon said about excitement around MVs. Happy to collaborate.
On Wed, Apr 9, 2025 at 5:29 PM Ángel Álvarez Pascua <
angel.alvarez.pascua@gmail.com> wrote:
> +1 (non-binding)
>
> El jue, 10 abr 2025, 1:50, Burak Yavuz <br...@gmail.com> escribió:
>
>> +1
>>
>> On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho <sz...@gmail.com> wrote:
>>
>>> +1 really excited to finally see Materialized View finally make its way
>>> to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg)
>>> already supporting it.
>>>
>>> Thanks
>>> Szehon
>>>
>>> On Wed, Apr 9, 2025 at 2:33 AM Martin Grund
>>> <ma...@databricks.com.invalid> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Dr Mich Talebzadeh,
>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> Glad to see Spark SQL extended to streaming use cases.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cheng Pan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>>>>>>
>>>>>>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>>>>>>> integration with Iceberg/Delta to unlock more value.
>>>>>>>>
>>>>>>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>>>>>>> >
>>>>>>>> > +1 looking forward to seeing this make progress!
>>>>>>>> >
>>>>>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> +1
>>>>>>>> >>
>>>>>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>>>>> >> > +1
>>>>>>>> >> >
>>>>>>>> >> > I am actually pretty excited to have this. Happy to see this
>>>>>>>> being proposed.
>>>>>>>> >> >
>>>>>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org>
>>>>>>>> wrote:
>>>>>>>> >> >
>>>>>>>> >> > > +1. Super excited about this effort!
>>>>>>>> >> > >
>>>>>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>>> >> > >
>>>>>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>>>>> management and
>>>>>>>> >> > >> enhances error detection.
>>>>>>>> >> > >>
>>>>>>>> >> > >>
>>>>>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <
>>>>>>>> dkbiswal@gmail.com> wrote:
>>>>>>>> >> > >>
>>>>>>>> >> > >>> Excited to see this heading toward open source —
>>>>>>>> materialized views and
>>>>>>>> >> > >>> other features will bring a lot of value.
>>>>>>>> >> > >>> +1 (non-binding)
>>>>>>>> >> > >>>
>>>>>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <
>>>>>>>> sandy@apache.org> wrote:
>>>>>>>> >> > >>>
>>>>>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to
>>>>>>>> be built on
>>>>>>>> >> > >>>> top of internal APIs for constructing and launching
>>>>>>>> pipeline executions.
>>>>>>>> >> > >>>> We'll have the option to expose these in the future.
>>>>>>>> >> > >>>>
>>>>>>>> >> > >>>> It would be worthwhile to understand the use cases in
>>>>>>>> more depth before
>>>>>>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>>>>>>> costly to
>>>>>>>> >> > >>>> maintain.
>>>>>>>> >> > >>>>
>>>>>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>>>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>>>>> >> > >>>>
>>>>>>>> >> > >>>>> Looks great!
>>>>>>>> >> > >>>>> QQ: will user able to run this pipeline from normal
>>>>>>>> code? I.e. can I
>>>>>>>> >> > >>>>> trigger a pipeline from *driver* code based on some
>>>>>>>> condition etc. or
>>>>>>>> >> > >>>>> it must be executed via separate shell command ?
>>>>>>>> >> > >>>>> As a background Databricks imposes similar limitation
>>>>>>>> where as you
>>>>>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster
>>>>>>>> for some reason
>>>>>>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>>>>>>> latency.
>>>>>>>> >> > >>>>>
>>>>>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <
>>>>>>>> sandy@apache.org> wrote:
>>>>>>>> >> > >>>>>
>>>>>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that
>>>>>>>> I've been
>>>>>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and
>>>>>>>> Jie Yang: [JIRA
>>>>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>]
>>>>>>>> [Doc
>>>>>>>> >> > >>>>>> <
>>>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>>>>> >
>>>>>>>> >> > >>>>>> ].
>>>>>>>> >> > >>>>>>
>>>>>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>>>>> execution model
>>>>>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>>>>> datasets up to date.
>>>>>>>> >> > >>>>>> It introduces the ability to compose multiple
>>>>>>>> transformations into a single
>>>>>>>> >> > >>>>>> declarative dataflow graph.
>>>>>>>> >> > >>>>>>
>>>>>>>> >> > >>>>>> Declarative pipelines aim to simplify the development
>>>>>>>> and management
>>>>>>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>>>>>>> orchestration of
>>>>>>>> >> > >>>>>> dependencies and making it possible to catch many
>>>>>>>> errors before any
>>>>>>>> >> > >>>>>> execution steps are launched.
>>>>>>>> >> > >>>>>>
>>>>>>>> >> > >>>>>> Declarative pipelines can include both batch and
>>>>>>>> streaming
>>>>>>>> >> > >>>>>> computations, leveraging Structured Streaming for
>>>>>>>> stream processing and new
>>>>>>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>>>>>>> integration with Spark
>>>>>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier
>>>>>>>> error detection than is
>>>>>>>> >> > >>>>>> achievable with more generic frameworks.
>>>>>>>> >> > >>>>>>
>>>>>>>> >> > >>>>>> Let us know what you think!
>>>>>>>> >> > >>>>>>
>>>>>>>> >> > >>>>>>
>>>>>>>> >> >
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
+1 (non-binding)
El jue, 10 abr 2025, 1:50, Burak Yavuz <br...@gmail.com> escribió:
> +1
>
> On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho <sz...@gmail.com> wrote:
>
>> +1 really excited to finally see Materialized View finally make its way
>> to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg)
>> already supporting it.
>>
>> Thanks
>> Szehon
>>
>> On Wed, Apr 9, 2025 at 2:33 AM Martin Grund <ma...@databricks.com.invalid>
>> wrote:
>>
>>> +1
>>>
>>> On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> Glad to see Spark SQL extended to streaming use cases.
>>>>>>
>>>>>> Thanks,
>>>>>> Cheng Pan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>>>>>
>>>>>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>>>>>> integration with Iceberg/Delta to unlock more value.
>>>>>>>
>>>>>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>>>>>> >
>>>>>>> > +1 looking forward to seeing this make progress!
>>>>>>> >
>>>>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> +1
>>>>>>> >>
>>>>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>>>> >> > +1
>>>>>>> >> >
>>>>>>> >> > I am actually pretty excited to have this. Happy to see this
>>>>>>> being proposed.
>>>>>>> >> >
>>>>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org>
>>>>>>> wrote:
>>>>>>> >> >
>>>>>>> >> > > +1. Super excited about this effort!
>>>>>>> >> > >
>>>>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>> >> > >
>>>>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>>>> management and
>>>>>>> >> > >> enhances error detection.
>>>>>>> >> > >>
>>>>>>> >> > >>
>>>>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <
>>>>>>> dkbiswal@gmail.com> wrote:
>>>>>>> >> > >>
>>>>>>> >> > >>> Excited to see this heading toward open source —
>>>>>>> materialized views and
>>>>>>> >> > >>> other features will bring a lot of value.
>>>>>>> >> > >>> +1 (non-binding)
>>>>>>> >> > >>>
>>>>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <
>>>>>>> sandy@apache.org> wrote:
>>>>>>> >> > >>>
>>>>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to
>>>>>>> be built on
>>>>>>> >> > >>>> top of internal APIs for constructing and launching
>>>>>>> pipeline executions.
>>>>>>> >> > >>>> We'll have the option to expose these in the future.
>>>>>>> >> > >>>>
>>>>>>> >> > >>>> It would be worthwhile to understand the use cases in more
>>>>>>> depth before
>>>>>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>>>>>> costly to
>>>>>>> >> > >>>> maintain.
>>>>>>> >> > >>>>
>>>>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>>>> >> > >>>>
>>>>>>> >> > >>>>> Looks great!
>>>>>>> >> > >>>>> QQ: will user able to run this pipeline from normal code?
>>>>>>> I.e. can I
>>>>>>> >> > >>>>> trigger a pipeline from *driver* code based on some
>>>>>>> condition etc. or
>>>>>>> >> > >>>>> it must be executed via separate shell command ?
>>>>>>> >> > >>>>> As a background Databricks imposes similar limitation
>>>>>>> where as you
>>>>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster
>>>>>>> for some reason
>>>>>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>>>>>> latency.
>>>>>>> >> > >>>>>
>>>>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>>>> wrote:
>>>>>>> >> > >>>>>
>>>>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that
>>>>>>> I've been
>>>>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>>>> Yang: [JIRA
>>>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>]
>>>>>>> [Doc
>>>>>>> >> > >>>>>> <
>>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>>>> >
>>>>>>> >> > >>>>>> ].
>>>>>>> >> > >>>>>>
>>>>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>>>> execution model
>>>>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>>>> datasets up to date.
>>>>>>> >> > >>>>>> It introduces the ability to compose multiple
>>>>>>> transformations into a single
>>>>>>> >> > >>>>>> declarative dataflow graph.
>>>>>>> >> > >>>>>>
>>>>>>> >> > >>>>>> Declarative pipelines aim to simplify the development
>>>>>>> and management
>>>>>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>>>>>> orchestration of
>>>>>>> >> > >>>>>> dependencies and making it possible to catch many errors
>>>>>>> before any
>>>>>>> >> > >>>>>> execution steps are launched.
>>>>>>> >> > >>>>>>
>>>>>>> >> > >>>>>> Declarative pipelines can include both batch and
>>>>>>> streaming
>>>>>>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>>>> processing and new
>>>>>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>>>>>> integration with Spark
>>>>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>>>> detection than is
>>>>>>> >> > >>>>>> achievable with more generic frameworks.
>>>>>>> >> > >>>>>>
>>>>>>> >> > >>>>>> Let us know what you think!
>>>>>>> >> > >>>>>>
>>>>>>> >> > >>>>>>
>>>>>>> >> >
>>>>>>> >>
>>>>>>> >>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Burak Yavuz <br...@gmail.com>.
+1
On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho <sz...@gmail.com> wrote:
> +1 really excited to finally see Materialized View finally make its way to
> Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg)
> already supporting it.
>
> Thanks
> Szehon
>
> On Wed, Apr 9, 2025 at 2:33 AM Martin Grund <ma...@databricks.com.invalid>
> wrote:
>
>> +1
>>
>> On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> +1
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> Glad to see Spark SQL extended to streaming use cases.
>>>>>
>>>>> Thanks,
>>>>> Cheng Pan
>>>>>
>>>>>
>>>>>
>>>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>>>>
>>>>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>>>>> integration with Iceberg/Delta to unlock more value.
>>>>>>
>>>>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>>>>> >
>>>>>> > +1 looking forward to seeing this make progress!
>>>>>> >
>>>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> +1
>>>>>> >>
>>>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>>> >> > +1
>>>>>> >> >
>>>>>> >> > I am actually pretty excited to have this. Happy to see this
>>>>>> being proposed.
>>>>>> >> >
>>>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org>
>>>>>> wrote:
>>>>>> >> >
>>>>>> >> > > +1. Super excited about this effort!
>>>>>> >> > >
>>>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>> >> > >
>>>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>>> management and
>>>>>> >> > >> enhances error detection.
>>>>>> >> > >>
>>>>>> >> > >>
>>>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <
>>>>>> dkbiswal@gmail.com> wrote:
>>>>>> >> > >>
>>>>>> >> > >>> Excited to see this heading toward open source —
>>>>>> materialized views and
>>>>>> >> > >>> other features will bring a lot of value.
>>>>>> >> > >>> +1 (non-binding)
>>>>>> >> > >>>
>>>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>>>> wrote:
>>>>>> >> > >>>
>>>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>>>> built on
>>>>>> >> > >>>> top of internal APIs for constructing and launching
>>>>>> pipeline executions.
>>>>>> >> > >>>> We'll have the option to expose these in the future.
>>>>>> >> > >>>>
>>>>>> >> > >>>> It would be worthwhile to understand the use cases in more
>>>>>> depth before
>>>>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>>>>> costly to
>>>>>> >> > >>>> maintain.
>>>>>> >> > >>>>
>>>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>>> >> > >>>>
>>>>>> >> > >>>>> Looks great!
>>>>>> >> > >>>>> QQ: will user able to run this pipeline from normal code?
>>>>>> I.e. can I
>>>>>> >> > >>>>> trigger a pipeline from *driver* code based on some
>>>>>> condition etc. or
>>>>>> >> > >>>>> it must be executed via separate shell command ?
>>>>>> >> > >>>>> As a background Databricks imposes similar limitation
>>>>>> where as you
>>>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster
>>>>>> for some reason
>>>>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>>>>> latency.
>>>>>> >> > >>>>>
>>>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>>> wrote:
>>>>>> >> > >>>>>
>>>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that
>>>>>> I've been
>>>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>>> Yang: [JIRA
>>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>>> >> > >>>>>> <
>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>>> >
>>>>>> >> > >>>>>> ].
>>>>>> >> > >>>>>>
>>>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>>> execution model
>>>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>>> datasets up to date.
>>>>>> >> > >>>>>> It introduces the ability to compose multiple
>>>>>> transformations into a single
>>>>>> >> > >>>>>> declarative dataflow graph.
>>>>>> >> > >>>>>>
>>>>>> >> > >>>>>> Declarative pipelines aim to simplify the development and
>>>>>> management
>>>>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>>>>> orchestration of
>>>>>> >> > >>>>>> dependencies and making it possible to catch many errors
>>>>>> before any
>>>>>> >> > >>>>>> execution steps are launched.
>>>>>> >> > >>>>>>
>>>>>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>>>>>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>>> processing and new
>>>>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>>>>> integration with Spark
>>>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>>> detection than is
>>>>>> >> > >>>>>> achievable with more generic frameworks.
>>>>>> >> > >>>>>>
>>>>>> >> > >>>>>> Let us know what you think!
>>>>>> >> > >>>>>>
>>>>>> >> > >>>>>>
>>>>>> >> >
>>>>>> >>
>>>>>> >>
>>>>>> ---------------------------------------------------------------------
>>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Szehon Ho <sz...@gmail.com>.
+1 really excited to finally see Materialized View finally make its way to
Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg)
already supporting it.
Thanks
Szehon
On Wed, Apr 9, 2025 at 2:33 AM Martin Grund <ma...@databricks.com.invalid>
wrote:
> +1
>
> On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> +1
>>
>> Dr Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> Glad to see Spark SQL extended to streaming use cases.
>>>>
>>>> Thanks,
>>>> Cheng Pan
>>>>
>>>>
>>>>
>>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>>>
>>>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>>>> integration with Iceberg/Delta to unlock more value.
>>>>>
>>>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>>>> >
>>>>> > +1 looking forward to seeing this make progress!
>>>>> >
>>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>>> wrote:
>>>>> >>
>>>>> >> +1
>>>>> >>
>>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> > I am actually pretty excited to have this. Happy to see this
>>>>> being proposed.
>>>>> >> >
>>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>>> >> >
>>>>> >> > > +1. Super excited about this effort!
>>>>> >> > >
>>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>>> huaxin.gao11@gmail.com> wrote:
>>>>> >> > >
>>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>>>> management and
>>>>> >> > >> enhances error detection.
>>>>> >> > >>
>>>>> >> > >>
>>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <
>>>>> dkbiswal@gmail.com> wrote:
>>>>> >> > >>
>>>>> >> > >>> Excited to see this heading toward open source — materialized
>>>>> views and
>>>>> >> > >>> other features will bring a lot of value.
>>>>> >> > >>> +1 (non-binding)
>>>>> >> > >>>
>>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>>> wrote:
>>>>> >> > >>>
>>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>>> built on
>>>>> >> > >>>> top of internal APIs for constructing and launching pipeline
>>>>> executions.
>>>>> >> > >>>> We'll have the option to expose these in the future.
>>>>> >> > >>>>
>>>>> >> > >>>> It would be worthwhile to understand the use cases in more
>>>>> depth before
>>>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>>>> costly to
>>>>> >> > >>>> maintain.
>>>>> >> > >>>>
>>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>>>> >> > >>>>
>>>>> >> > >>>>> Looks great!
>>>>> >> > >>>>> QQ: will user able to run this pipeline from normal code?
>>>>> I.e. can I
>>>>> >> > >>>>> trigger a pipeline from *driver* code based on some
>>>>> condition etc. or
>>>>> >> > >>>>> it must be executed via separate shell command ?
>>>>> >> > >>>>> As a background Databricks imposes similar limitation where
>>>>> as you
>>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster
>>>>> for some reason
>>>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>>>> latency.
>>>>> >> > >>>>>
>>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>>> wrote:
>>>>> >> > >>>>>
>>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>>>> been
>>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>>> Yang: [JIRA
>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>> >> > >>>>>> <
>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>>> >
>>>>> >> > >>>>>> ].
>>>>> >> > >>>>>>
>>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>>> execution model
>>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>>> datasets up to date.
>>>>> >> > >>>>>> It introduces the ability to compose multiple
>>>>> transformations into a single
>>>>> >> > >>>>>> declarative dataflow graph.
>>>>> >> > >>>>>>
>>>>> >> > >>>>>> Declarative pipelines aim to simplify the development and
>>>>> management
>>>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>>>> orchestration of
>>>>> >> > >>>>>> dependencies and making it possible to catch many errors
>>>>> before any
>>>>> >> > >>>>>> execution steps are launched.
>>>>> >> > >>>>>>
>>>>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>>>>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>>>>> processing and new
>>>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>>>> integration with Spark
>>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>>> detection than is
>>>>> >> > >>>>>> achievable with more generic frameworks.
>>>>> >> > >>>>>>
>>>>> >> > >>>>>> Let us know what you think!
>>>>> >> > >>>>>>
>>>>> >> > >>>>>>
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> ---------------------------------------------------------------------
>>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Martin Grund <ma...@databricks.com.INVALID>.
+1
On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh <mi...@gmail.com>
wrote:
> +1
>
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
>
>> +1
>>
>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> Glad to see Spark SQL extended to streaming use cases.
>>>
>>> Thanks,
>>> Cheng Pan
>>>
>>>
>>>
>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com>
>>> wrote:
>>>
>>> +1
>>>
>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>>
>>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>>> integration with Iceberg/Delta to unlock more value.
>>>>
>>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>>> >
>>>> > +1 looking forward to seeing this make progress!
>>>> >
>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> +1
>>>> >>
>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>>> >> > +1
>>>> >> >
>>>> >> > I am actually pretty excited to have this. Happy to see this being
>>>> proposed.
>>>> >> >
>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>>> >> >
>>>> >> > > +1. Super excited about this effort!
>>>> >> > >
>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <
>>>> huaxin.gao11@gmail.com> wrote:
>>>> >> > >
>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>>> management and
>>>> >> > >> enhances error detection.
>>>> >> > >>
>>>> >> > >>
>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>>> wrote:
>>>> >> > >>
>>>> >> > >>> Excited to see this heading toward open source — materialized
>>>> views and
>>>> >> > >>> other features will bring a lot of value.
>>>> >> > >>> +1 (non-binding)
>>>> >> > >>>
>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>>> wrote:
>>>> >> > >>>
>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>>> built on
>>>> >> > >>>> top of internal APIs for constructing and launching pipeline
>>>> executions.
>>>> >> > >>>> We'll have the option to expose these in the future.
>>>> >> > >>>>
>>>> >> > >>>> It would be worthwhile to understand the use cases in more
>>>> depth before
>>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>>> costly to
>>>> >> > >>>> maintain.
>>>> >> > >>>>
>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>>> >> > >>>>
>>>> >> > >>>>> Looks great!
>>>> >> > >>>>> QQ: will user able to run this pipeline from normal code?
>>>> I.e. can I
>>>> >> > >>>>> trigger a pipeline from *driver* code based on some
>>>> condition etc. or
>>>> >> > >>>>> it must be executed via separate shell command ?
>>>> >> > >>>>> As a background Databricks imposes similar limitation where
>>>> as you
>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>>> some reason
>>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>>> latency.
>>>> >> > >>>>>
>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>>> wrote:
>>>> >> > >>>>>
>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>>> been
>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>>> Yang: [JIRA
>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>> >> > >>>>>> <
>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>>> >
>>>> >> > >>>>>> ].
>>>> >> > >>>>>>
>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>>> execution model
>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>>> datasets up to date.
>>>> >> > >>>>>> It introduces the ability to compose multiple
>>>> transformations into a single
>>>> >> > >>>>>> declarative dataflow graph.
>>>> >> > >>>>>>
>>>> >> > >>>>>> Declarative pipelines aim to simplify the development and
>>>> management
>>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>>> orchestration of
>>>> >> > >>>>>> dependencies and making it possible to catch many errors
>>>> before any
>>>> >> > >>>>>> execution steps are launched.
>>>> >> > >>>>>>
>>>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>>>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>>>> processing and new
>>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>>> integration with Spark
>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>>> detection than is
>>>> >> > >>>>>> achievable with more generic frameworks.
>>>> >> > >>>>>>
>>>> >> > >>>>>> Let us know what you think!
>>>> >> > >>>>>>
>>>> >> > >>>>>>
>>>> >> >
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Mich Talebzadeh <mi...@gmail.com>.
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 9 Apr 2025 at 08:07, Peter Toth <pe...@gmail.com> wrote:
> +1
>
> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> Glad to see Spark SQL extended to streaming use cases.
>>
>> Thanks,
>> Cheng Pan
>>
>>
>>
>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com> wrote:
>>
>> +1
>>
>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>>
>>> +1 I'm delighted that it will be open-sourced, enabling greater
>>> integration with Iceberg/Delta to unlock more value.
>>>
>>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>>> >
>>> > +1 looking forward to seeing this make progress!
>>> >
>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>> >>
>>> >> +1
>>> >>
>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>>> >> > +1
>>> >> >
>>> >> > I am actually pretty excited to have this. Happy to see this being
>>> proposed.
>>> >> >
>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>>> >> >
>>> >> > > +1. Super excited about this effort!
>>> >> > >
>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>>> wrote:
>>> >> > >
>>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>>> management and
>>> >> > >> enhances error detection.
>>> >> > >>
>>> >> > >>
>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>>> wrote:
>>> >> > >>
>>> >> > >>> Excited to see this heading toward open source — materialized
>>> views and
>>> >> > >>> other features will bring a lot of value.
>>> >> > >>> +1 (non-binding)
>>> >> > >>>
>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>>> built on
>>> >> > >>>> top of internal APIs for constructing and launching pipeline
>>> executions.
>>> >> > >>>> We'll have the option to expose these in the future.
>>> >> > >>>>
>>> >> > >>>> It would be worthwhile to understand the use cases in more
>>> depth before
>>> >> > >>>> exposing these, because APIs are one-way doors and can be
>>> costly to
>>> >> > >>>> maintain.
>>> >> > >>>>
>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>>> >> > >>>>
>>> >> > >>>>> Looks great!
>>> >> > >>>>> QQ: will user able to run this pipeline from normal code?
>>> I.e. can I
>>> >> > >>>>> trigger a pipeline from *driver* code based on some condition
>>> etc. or
>>> >> > >>>>> it must be executed via separate shell command ?
>>> >> > >>>>> As a background Databricks imposes similar limitation where
>>> as you
>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>>> some reason
>>> >> > >>>>> and forces to use two clusters increasing the cost and
>>> latency.
>>> >> > >>>>>
>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>>> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>>> been
>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>>> Yang: [JIRA
>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>> >> > >>>>>> <
>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>>> >
>>> >> > >>>>>> ].
>>> >> > >>>>>>
>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>>> execution model
>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>>> datasets up to date.
>>> >> > >>>>>> It introduces the ability to compose multiple
>>> transformations into a single
>>> >> > >>>>>> declarative dataflow graph.
>>> >> > >>>>>>
>>> >> > >>>>>> Declarative pipelines aim to simplify the development and
>>> management
>>> >> > >>>>>> of data pipelines, by removing the need for manual
>>> orchestration of
>>> >> > >>>>>> dependencies and making it possible to catch many errors
>>> before any
>>> >> > >>>>>> execution steps are launched.
>>> >> > >>>>>>
>>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>>> processing and new
>>> >> > >>>>>> materialized view syntax for batch processing. Tight
>>> integration with Spark
>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>>> detection than is
>>> >> > >>>>>> achievable with more generic frameworks.
>>> >> > >>>>>>
>>> >> > >>>>>> Let us know what you think!
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Peter Toth <pe...@gmail.com>.
+1
On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pa...@gmail.com> wrote:
> +1 (non-binding)
>
> Glad to see Spark SQL extended to streaming use cases.
>
> Thanks,
> Cheng Pan
>
>
>
> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com> wrote:
>
> +1
>
> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
>
>> +1 I'm delighted that it will be open-sourced, enabling greater
>> integration with Iceberg/Delta to unlock more value.
>>
>> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>> >
>> > +1 looking forward to seeing this make progress!
>> >
>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>> >>
>> >> +1
>> >>
>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> >> > +1
>> >> >
>> >> > I am actually pretty excited to have this. Happy to see this being
>> proposed.
>> >> >
>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>> >> >
>> >> > > +1. Super excited about this effort!
>> >> > >
>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
>> wrote:
>> >> > >
>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>> management and
>> >> > >> enhances error detection.
>> >> > >>
>> >> > >>
>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
>> wrote:
>> >> > >>
>> >> > >>> Excited to see this heading toward open source — materialized
>> views and
>> >> > >>> other features will bring a lot of value.
>> >> > >>> +1 (non-binding)
>> >> > >>>
>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>> wrote:
>> >> > >>>
>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
>> built on
>> >> > >>>> top of internal APIs for constructing and launching pipeline
>> executions.
>> >> > >>>> We'll have the option to expose these in the future.
>> >> > >>>>
>> >> > >>>> It would be worthwhile to understand the use cases in more
>> depth before
>> >> > >>>> exposing these, because APIs are one-way doors and can be
>> costly to
>> >> > >>>> maintain.
>> >> > >>>>
>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> >> > >>>> khalidmammadov9@gmail.com> wrote:
>> >> > >>>>
>> >> > >>>>> Looks great!
>> >> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>> can I
>> >> > >>>>> trigger a pipeline from *driver* code based on some condition
>> etc. or
>> >> > >>>>> it must be executed via separate shell command ?
>> >> > >>>>> As a background Databricks imposes similar limitation where as
>> you
>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster for
>> some reason
>> >> > >>>>> and forces to use two clusters increasing the cost and latency.
>> >> > >>>>>
>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
>> wrote:
>> >> > >>>>>
>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've
>> been
>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>> Yang: [JIRA
>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> >> > >>>>>> <
>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>> >
>> >> > >>>>>> ].
>> >> > >>>>>>
>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
>> execution model
>> >> > >>>>>> beyond single queries, to pipelines that keep multiple
>> datasets up to date.
>> >> > >>>>>> It introduces the ability to compose multiple transformations
>> into a single
>> >> > >>>>>> declarative dataflow graph.
>> >> > >>>>>>
>> >> > >>>>>> Declarative pipelines aim to simplify the development and
>> management
>> >> > >>>>>> of data pipelines, by removing the need for manual
>> orchestration of
>> >> > >>>>>> dependencies and making it possible to catch many errors
>> before any
>> >> > >>>>>> execution steps are launched.
>> >> > >>>>>>
>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>> >> > >>>>>> computations, leveraging Structured Streaming for stream
>> processing and new
>> >> > >>>>>> materialized view syntax for batch processing. Tight
>> integration with Spark
>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>> detection than is
>> >> > >>>>>> achievable with more generic frameworks.
>> >> > >>>>>>
>> >> > >>>>>> Let us know what you think!
>> >> > >>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Cheng Pan <pa...@gmail.com>.
+1 (non-binding)
Glad to see Spark SQL extended to streaming use cases.
Thanks,
Cheng Pan
> On Apr 9, 2025, at 14:43, Anton Okolnychyi <ao...@gmail.com> wrote:
>
> +1
>
> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qcsd2011@gmail.com <ma...@gmail.com>> пише:
>> +1 I'm delighted that it will be open-sourced, enabling greater
>> integration with Iceberg/Delta to unlock more value.
>>
>> Jungtaek Lim <kabhwan.opensource@gmail.com <ma...@gmail.com>> 于2025年4月9日周三 10:47写道：
>> >
>> > +1 looking forward to seeing this make progress!
>> >
>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <yangjie01@apache.org <ma...@apache.org>> wrote:
>> >>
>> >> +1
>> >>
>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> >> > +1
>> >> >
>> >> > I am actually pretty excited to have this. Happy to see this being proposed.
>> >> >
>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <sunchao@apache.org <ma...@apache.org>> wrote:
>> >> >
>> >> > > +1. Super excited about this effort!
>> >> > >
>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <huaxin.gao11@gmail.com <ma...@gmail.com>> wrote:
>> >> > >
>> >> > >> +1 I support this SPIP because it simplifies data pipeline management and
>> >> > >> enhances error detection.
>> >> > >>
>> >> > >>
>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dkbiswal@gmail.com <ma...@gmail.com>> wrote:
>> >> > >>
>> >> > >>> Excited to see this heading toward open source — materialized views and
>> >> > >>> other features will bring a lot of value.
>> >> > >>> +1 (non-binding)
>> >> > >>>
>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sandy@apache.org <ma...@apache.org>> wrote:
>> >> > >>>
>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be built on
>> >> > >>>> top of internal APIs for constructing and launching pipeline executions.
>> >> > >>>> We'll have the option to expose these in the future.
>> >> > >>>>
>> >> > >>>> It would be worthwhile to understand the use cases in more depth before
>> >> > >>>> exposing these, because APIs are one-way doors and can be costly to
>> >> > >>>> maintain.
>> >> > >>>>
>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> >> > >>>> khalidmammadov9@gmail.com <ma...@gmail.com>> wrote:
>> >> > >>>>
>> >> > >>>>> Looks great!
>> >> > >>>>> QQ: will user able to run this pipeline from normal code? I.e. can I
>> >> > >>>>> trigger a pipeline from *driver* code based on some condition etc. or
>> >> > >>>>> it must be executed via separate shell command ?
>> >> > >>>>> As a background Databricks imposes similar limitation where as you
>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster for some reason
>> >> > >>>>> and forces to use two clusters increasing the cost and latency.
>> >> > >>>>>
>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sandy@apache.org <ma...@apache.org>> wrote:
>> >> > >>>>>
>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> >> > >>>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>> >> > >>>>>> ].
>> >> > >>>>>>
>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution model
>> >> > >>>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>> >> > >>>>>> It introduces the ability to compose multiple transformations into a single
>> >> > >>>>>> declarative dataflow graph.
>> >> > >>>>>>
>> >> > >>>>>> Declarative pipelines aim to simplify the development and management
>> >> > >>>>>> of data pipelines, by removing the need for manual orchestration of
>> >> > >>>>>> dependencies and making it possible to catch many errors before any
>> >> > >>>>>> execution steps are launched.
>> >> > >>>>>>
>> >> > >>>>>> Declarative pipelines can include both batch and streaming
>> >> > >>>>>> computations, leveraging Structured Streaming for stream processing and new
>> >> > >>>>>> materialized view syntax for batch processing. Tight integration with Spark
>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
>> >> > >>>>>> achievable with more generic frameworks.
>> >> > >>>>>>
>> >> > >>>>>> Let us know what you think!
>> >> > >>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Anton Okolnychyi <ao...@gmail.com>.
+1
вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qc...@gmail.com> пише:
> +1 I'm delighted that it will be open-sourced, enabling greater
> integration with Iceberg/Delta to unlock more value.
>
> Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
> >
> > +1 looking forward to seeing this make progress!
> >
> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
> >>
> >> +1
> >>
> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
> >> > +1
> >> >
> >> > I am actually pretty excited to have this. Happy to see this being
> proposed.
> >> >
> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
> >> >
> >> > > +1. Super excited about this effort!
> >> > >
> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
> wrote:
> >> > >
> >> > >> +1 I support this SPIP because it simplifies data pipeline
> management and
> >> > >> enhances error detection.
> >> > >>
> >> > >>
> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
> wrote:
> >> > >>
> >> > >>> Excited to see this heading toward open source — materialized
> views and
> >> > >>> other features will bring a lot of value.
> >> > >>> +1 (non-binding)
> >> > >>>
> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
> wrote:
> >> > >>>
> >> > >>>> Hi Khalid – the CLI in the current proposal will need to be
> built on
> >> > >>>> top of internal APIs for constructing and launching pipeline
> executions.
> >> > >>>> We'll have the option to expose these in the future.
> >> > >>>>
> >> > >>>> It would be worthwhile to understand the use cases in more depth
> before
> >> > >>>> exposing these, because APIs are one-way doors and can be costly
> to
> >> > >>>> maintain.
> >> > >>>>
> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
> >> > >>>> khalidmammadov9@gmail.com> wrote:
> >> > >>>>
> >> > >>>>> Looks great!
> >> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
> can I
> >> > >>>>> trigger a pipeline from *driver* code based on some condition
> etc. or
> >> > >>>>> it must be executed via separate shell command ?
> >> > >>>>> As a background Databricks imposes similar limitation where as
> you
> >> > >>>>> cannot run normal Spark code and DLT on the same cluster for
> some reason
> >> > >>>>> and forces to use two clusters increasing the cost and latency.
> >> > >>>>>
> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org>
> wrote:
> >> > >>>>>
> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
> [JIRA
> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
> >> > >>>>>> <
> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
> >
> >> > >>>>>> ].
> >> > >>>>>>
> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative
> execution model
> >> > >>>>>> beyond single queries, to pipelines that keep multiple
> datasets up to date.
> >> > >>>>>> It introduces the ability to compose multiple transformations
> into a single
> >> > >>>>>> declarative dataflow graph.
> >> > >>>>>>
> >> > >>>>>> Declarative pipelines aim to simplify the development and
> management
> >> > >>>>>> of data pipelines, by removing the need for manual
> orchestration of
> >> > >>>>>> dependencies and making it possible to catch many errors
> before any
> >> > >>>>>> execution steps are launched.
> >> > >>>>>>
> >> > >>>>>> Declarative pipelines can include both batch and streaming
> >> > >>>>>> computations, leveraging Structured Streaming for stream
> processing and new
> >> > >>>>>> materialized view syntax for batch processing. Tight
> integration with Spark
> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
> detection than is
> >> > >>>>>> achievable with more generic frameworks.
> >> > >>>>>>
> >> > >>>>>> Let us know what you think!
> >> > >>>>>>
> >> > >>>>>>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Jacky Lee <qc...@gmail.com>.
+1 I'm delighted that it will be open-sourced, enabling greater
integration with Iceberg/Delta to unlock more value.
Jungtaek Lim <ka...@gmail.com> 于2025年4月9日周三 10:47写道：
>
> +1 looking forward to seeing this make progress!
>
> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
>>
>> +1
>>
>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> > +1
>> >
>> > I am actually pretty excited to have this. Happy to see this being proposed.
>> >
>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>> >
>> > > +1. Super excited about this effort!
>> > >
>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com> wrote:
>> > >
>> > >> +1 I support this SPIP because it simplifies data pipeline management and
>> > >> enhances error detection.
>> > >>
>> > >>
>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com> wrote:
>> > >>
>> > >>> Excited to see this heading toward open source — materialized views and
>> > >>> other features will bring a lot of value.
>> > >>> +1 (non-binding)
>> > >>>
>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
>> > >>>
>> > >>>> Hi Khalid – the CLI in the current proposal will need to be built on
>> > >>>> top of internal APIs for constructing and launching pipeline executions.
>> > >>>> We'll have the option to expose these in the future.
>> > >>>>
>> > >>>> It would be worthwhile to understand the use cases in more depth before
>> > >>>> exposing these, because APIs are one-way doors and can be costly to
>> > >>>> maintain.
>> > >>>>
>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> > >>>> khalidmammadov9@gmail.com> wrote:
>> > >>>>
>> > >>>>> Looks great!
>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e. can I
>> > >>>>> trigger a pipeline from *driver* code based on some condition etc. or
>> > >>>>> it must be executed via separate shell command ?
>> > >>>>> As a background Databricks imposes similar limitation where as you
>> > >>>>> cannot run normal Spark code and DLT on the same cluster for some reason
>> > >>>>> and forces to use two clusters increasing the cost and latency.
>> > >>>>>
>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>> > >>>>>
>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> > >>>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>> > >>>>>> ].
>> > >>>>>>
>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution model
>> > >>>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>> > >>>>>> It introduces the ability to compose multiple transformations into a single
>> > >>>>>> declarative dataflow graph.
>> > >>>>>>
>> > >>>>>> Declarative pipelines aim to simplify the development and management
>> > >>>>>> of data pipelines, by removing the need for manual orchestration of
>> > >>>>>> dependencies and making it possible to catch many errors before any
>> > >>>>>> execution steps are launched.
>> > >>>>>>
>> > >>>>>> Declarative pipelines can include both batch and streaming
>> > >>>>>> computations, leveraging Structured Streaming for stream processing and new
>> > >>>>>> materialized view syntax for batch processing. Tight integration with Spark
>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
>> > >>>>>> achievable with more generic frameworks.
>> > >>>>>>
>> > >>>>>> Let us know what you think!
>> > >>>>>>
>> > >>>>>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Jungtaek Lim <ka...@gmail.com>.
+1 looking forward to seeing this make progress!
On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <ya...@apache.org> wrote:
> +1
>
> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
> > +1
> >
> > I am actually pretty excited to have this. Happy to see this being
> proposed.
> >
> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
> >
> > > +1. Super excited about this effort!
> > >
> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com>
> wrote:
> > >
> > >> +1 I support this SPIP because it simplifies data pipeline management
> and
> > >> enhances error detection.
> > >>
> > >>
> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com>
> wrote:
> > >>
> > >>> Excited to see this heading toward open source — materialized views
> and
> > >>> other features will bring a lot of value.
> > >>> +1 (non-binding)
> > >>>
> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
> > >>>
> > >>>> Hi Khalid – the CLI in the current proposal will need to be built on
> > >>>> top of internal APIs for constructing and launching pipeline
> executions.
> > >>>> We'll have the option to expose these in the future.
> > >>>>
> > >>>> It would be worthwhile to understand the use cases in more depth
> before
> > >>>> exposing these, because APIs are one-way doors and can be costly to
> > >>>> maintain.
> > >>>>
> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
> > >>>> khalidmammadov9@gmail.com> wrote:
> > >>>>
> > >>>>> Looks great!
> > >>>>> QQ: will user able to run this pipeline from normal code? I.e. can
> I
> > >>>>> trigger a pipeline from *driver* code based on some condition etc.
> or
> > >>>>> it must be executed via separate shell command ?
> > >>>>> As a background Databricks imposes similar limitation where as you
> > >>>>> cannot run normal Spark code and DLT on the same cluster for some
> reason
> > >>>>> and forces to use two clusters increasing the cost and latency.
> > >>>>>
> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
> > >>>>>
> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
> [JIRA
> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
> > >>>>>> <
> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
> >
> > >>>>>> ].
> > >>>>>>
> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution
> model
> > >>>>>> beyond single queries, to pipelines that keep multiple datasets
> up to date.
> > >>>>>> It introduces the ability to compose multiple transformations
> into a single
> > >>>>>> declarative dataflow graph.
> > >>>>>>
> > >>>>>> Declarative pipelines aim to simplify the development and
> management
> > >>>>>> of data pipelines, by removing the need for manual orchestration
> of
> > >>>>>> dependencies and making it possible to catch many errors before
> any
> > >>>>>> execution steps are launched.
> > >>>>>>
> > >>>>>> Declarative pipelines can include both batch and streaming
> > >>>>>> computations, leveraging Structured Streaming for stream
> processing and new
> > >>>>>> materialized view syntax for batch processing. Tight integration
> with Spark
> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
> detection than is
> > >>>>>> achievable with more generic frameworks.
> > >>>>>>
> > >>>>>> Let us know what you think!
> > >>>>>>
> > >>>>>>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Yang Jie <ya...@apache.org>.
+1
On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
> +1
>
> I am actually pretty excited to have this. Happy to see this being proposed.
>
> On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
>
> > +1. Super excited about this effort!
> >
> > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com> wrote:
> >
> >> +1 I support this SPIP because it simplifies data pipeline management and
> >> enhances error detection.
> >>
> >>
> >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com> wrote:
> >>
> >>> Excited to see this heading toward open source — materialized views and
> >>> other features will bring a lot of value.
> >>> +1 (non-binding)
> >>>
> >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
> >>>
> >>>> Hi Khalid – the CLI in the current proposal will need to be built on
> >>>> top of internal APIs for constructing and launching pipeline executions.
> >>>> We'll have the option to expose these in the future.
> >>>>
> >>>> It would be worthwhile to understand the use cases in more depth before
> >>>> exposing these, because APIs are one-way doors and can be costly to
> >>>> maintain.
> >>>>
> >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
> >>>> khalidmammadov9@gmail.com> wrote:
> >>>>
> >>>>> Looks great!
> >>>>> QQ: will user able to run this pipeline from normal code? I.e. can I
> >>>>> trigger a pipeline from *driver* code based on some condition etc. or
> >>>>> it must be executed via separate shell command ?
> >>>>> As a background Databricks imposes similar limitation where as you
> >>>>> cannot run normal Spark code and DLT on the same cluster for some reason
> >>>>> and forces to use two clusters increasing the cost and latency.
> >>>>>
> >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
> >>>>>
> >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
> >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
> >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
> >>>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
> >>>>>> ].
> >>>>>>
> >>>>>> The SPIP proposes extending Spark's lazy, declarative execution model
> >>>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
> >>>>>> It introduces the ability to compose multiple transformations into a single
> >>>>>> declarative dataflow graph.
> >>>>>>
> >>>>>> Declarative pipelines aim to simplify the development and management
> >>>>>> of data pipelines, by removing the need for manual orchestration of
> >>>>>> dependencies and making it possible to catch many errors before any
> >>>>>> execution steps are launched.
> >>>>>>
> >>>>>> Declarative pipelines can include both batch and streaming
> >>>>>> computations, leveraging Structured Streaming for stream processing and new
> >>>>>> materialized view syntax for batch processing. Tight integration with Spark
> >>>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
> >>>>>> achievable with more generic frameworks.
> >>>>>>
> >>>>>> Let us know what you think!
> >>>>>>
> >>>>>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Hyukjin Kwon <gu...@apache.org>.
+1
I am actually pretty excited to have this. Happy to see this being proposed.
On Wed, 9 Apr 2025 at 01:55, Chao Sun <su...@apache.org> wrote:
> +1. Super excited about this effort!
>
> On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com> wrote:
>
>> +1 I support this SPIP because it simplifies data pipeline management and
>> enhances error detection.
>>
>>
>> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com> wrote:
>>
>>> Excited to see this heading toward open source — materialized views and
>>> other features will bring a lot of value.
>>> +1 (non-binding)
>>>
>>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
>>>
>>>> Hi Khalid – the CLI in the current proposal will need to be built on
>>>> top of internal APIs for constructing and launching pipeline executions.
>>>> We'll have the option to expose these in the future.
>>>>
>>>> It would be worthwhile to understand the use cases in more depth before
>>>> exposing these, because APIs are one-way doors and can be costly to
>>>> maintain.
>>>>
>>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>>> khalidmammadov9@gmail.com> wrote:
>>>>
>>>>> Looks great!
>>>>> QQ: will user able to run this pipeline from normal code? I.e. can I
>>>>> trigger a pipeline from *driver* code based on some condition etc. or
>>>>> it must be executed via separate shell command ?
>>>>> As a background Databricks imposes similar limitation where as you
>>>>> cannot run normal Spark code and DLT on the same cluster for some reason
>>>>> and forces to use two clusters increasing the cost and latency.
>>>>>
>>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>>>>>
>>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>>>>>> ].
>>>>>>
>>>>>> The SPIP proposes extending Spark's lazy, declarative execution model
>>>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>>>>>> It introduces the ability to compose multiple transformations into a single
>>>>>> declarative dataflow graph.
>>>>>>
>>>>>> Declarative pipelines aim to simplify the development and management
>>>>>> of data pipelines, by removing the need for manual orchestration of
>>>>>> dependencies and making it possible to catch many errors before any
>>>>>> execution steps are launched.
>>>>>>
>>>>>> Declarative pipelines can include both batch and streaming
>>>>>> computations, leveraging Structured Streaming for stream processing and new
>>>>>> materialized view syntax for batch processing. Tight integration with Spark
>>>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
>>>>>> achievable with more generic frameworks.
>>>>>>
>>>>>> Let us know what you think!
>>>>>>
>>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Chao Sun <su...@apache.org>.
+1. Super excited about this effort!
On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <hu...@gmail.com> wrote:
> +1 I support this SPIP because it simplifies data pipeline management and
> enhances error detection.
>
>
> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com> wrote:
>
>> Excited to see this heading toward open source — materialized views and
>> other features will bring a lot of value.
>> +1 (non-binding)
>>
>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
>>
>>> Hi Khalid – the CLI in the current proposal will need to be built on top
>>> of internal APIs for constructing and launching pipeline executions. We'll
>>> have the option to expose these in the future.
>>>
>>> It would be worthwhile to understand the use cases in more depth before
>>> exposing these, because APIs are one-way doors and can be costly to
>>> maintain.
>>>
>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>>> khalidmammadov9@gmail.com> wrote:
>>>
>>>> Looks great!
>>>> QQ: will user able to run this pipeline from normal code? I.e. can I
>>>> trigger a pipeline from *driver* code based on some condition etc. or
>>>> it must be executed via separate shell command ?
>>>> As a background Databricks imposes similar limitation where as you
>>>> cannot run normal Spark code and DLT on the same cluster for some reason
>>>> and forces to use two clusters increasing the cost and latency.
>>>>
>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>>>>
>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>>>>> ].
>>>>>
>>>>> The SPIP proposes extending Spark's lazy, declarative execution model
>>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>>>>> It introduces the ability to compose multiple transformations into a single
>>>>> declarative dataflow graph.
>>>>>
>>>>> Declarative pipelines aim to simplify the development and management
>>>>> of data pipelines, by removing the need for manual orchestration of
>>>>> dependencies and making it possible to catch many errors before any
>>>>> execution steps are launched.
>>>>>
>>>>> Declarative pipelines can include both batch and streaming
>>>>> computations, leveraging Structured Streaming for stream processing and new
>>>>> materialized view syntax for batch processing. Tight integration with Spark
>>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
>>>>> achievable with more generic frameworks.
>>>>>
>>>>> Let us know what you think!
>>>>>
>>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by huaxin gao <hu...@gmail.com>.
+1 I support this SPIP because it simplifies data pipeline management and
enhances error detection.
On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dk...@gmail.com> wrote:
> Excited to see this heading toward open source — materialized views and
> other features will bring a lot of value.
> +1 (non-binding)
>
> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
>
>> Hi Khalid – the CLI in the current proposal will need to be built on top
>> of internal APIs for constructing and launching pipeline executions. We'll
>> have the option to expose these in the future.
>>
>> It would be worthwhile to understand the use cases in more depth before
>> exposing these, because APIs are one-way doors and can be costly to
>> maintain.
>>
>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> khalidmammadov9@gmail.com> wrote:
>>
>>> Looks great!
>>> QQ: will user able to run this pipeline from normal code? I.e. can I
>>> trigger a pipeline from *driver* code based on some condition etc. or
>>> it must be executed via separate shell command ?
>>> As a background Databricks imposes similar limitation where as you
>>> cannot run normal Spark code and DLT on the same cluster for some reason
>>> and forces to use two clusters increasing the cost and latency.
>>>
>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>>>
>>>> Hi all – starting a discussion thread for a SPIP that I've been working
>>>> on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>>>> ].
>>>>
>>>> The SPIP proposes extending Spark's lazy, declarative execution model
>>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>>>> It introduces the ability to compose multiple transformations into a single
>>>> declarative dataflow graph.
>>>>
>>>> Declarative pipelines aim to simplify the development and management of
>>>> data pipelines, by removing the need for manual orchestration of
>>>> dependencies and making it possible to catch many errors before any
>>>> execution steps are launched.
>>>>
>>>> Declarative pipelines can include both batch and streaming
>>>> computations, leveraging Structured Streaming for stream processing and new
>>>> materialized view syntax for batch processing. Tight integration with Spark
>>>> SQL's analyzer enables deeper analysis and earlier error detection than is
>>>> achievable with more generic frameworks.
>>>>
>>>> Let us know what you think!
>>>>
>>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Dilip Biswal <dk...@gmail.com>.
Excited to see this heading toward open source — materialized views and
other features will bring a lot of value.
+1 (non-binding)
On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org> wrote:
> Hi Khalid – the CLI in the current proposal will need to be built on top
> of internal APIs for constructing and launching pipeline executions. We'll
> have the option to expose these in the future.
>
> It would be worthwhile to understand the use cases in more depth before
> exposing these, because APIs are one-way doors and can be costly to
> maintain.
>
> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <kh...@gmail.com>
> wrote:
>
>> Looks great!
>> QQ: will user able to run this pipeline from normal code? I.e. can I
>> trigger a pipeline from *driver* code based on some condition etc. or it
>> must be executed via separate shell command ?
>> As a background Databricks imposes similar limitation where as you cannot
>> run normal Spark code and DLT on the same cluster for some reason and
>> forces to use two clusters increasing the cost and latency.
>>
>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>>
>>> Hi all – starting a discussion thread for a SPIP that I've been working
>>> on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>>> ].
>>>
>>> The SPIP proposes extending Spark's lazy, declarative execution model
>>> beyond single queries, to pipelines that keep multiple datasets up to date.
>>> It introduces the ability to compose multiple transformations into a single
>>> declarative dataflow graph.
>>>
>>> Declarative pipelines aim to simplify the development and management of
>>> data pipelines, by removing the need for manual orchestration of
>>> dependencies and making it possible to catch many errors before any
>>> execution steps are launched.
>>>
>>> Declarative pipelines can include both batch and streaming computations,
>>> leveraging Structured Streaming for stream processing and new materialized
>>> view syntax for batch processing. Tight integration with Spark SQL's
>>> analyzer enables deeper analysis and earlier error detection than is
>>> achievable with more generic frameworks.
>>>
>>> Let us know what you think!
>>>
>>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Sandy Ryza <sa...@apache.org>.
Hi Khalid – the CLI in the current proposal will need to be built on top of
internal APIs for constructing and launching pipeline executions. We'll
have the option to expose these in the future.
It would be worthwhile to understand the use cases in more depth before
exposing these, because APIs are one-way doors and can be costly to
maintain.
On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <kh...@gmail.com>
wrote:
> Looks great!
> QQ: will user able to run this pipeline from normal code? I.e. can I
> trigger a pipeline from *driver* code based on some condition etc. or it
> must be executed via separate shell command ?
> As a background Databricks imposes similar limitation where as you cannot
> run normal Spark code and DLT on the same cluster for some reason and
> forces to use two clusters increasing the cost and latency.
>
> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>
>> Hi all – starting a discussion thread for a SPIP that I've been working
>> on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
>> ].
>>
>> The SPIP proposes extending Spark's lazy, declarative execution model
>> beyond single queries, to pipelines that keep multiple datasets up to date.
>> It introduces the ability to compose multiple transformations into a single
>> declarative dataflow graph.
>>
>> Declarative pipelines aim to simplify the development and management of
>> data pipelines, by removing the need for manual orchestration of
>> dependencies and making it possible to catch many errors before any
>> execution steps are launched.
>>
>> Declarative pipelines can include both batch and streaming computations,
>> leveraging Structured Streaming for stream processing and new materialized
>> view syntax for batch processing. Tight integration with Spark SQL's
>> analyzer enables deeper analysis and earlier error detection than is
>> achievable with more generic frameworks.
>>
>> Let us know what you think!
>>
>>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Khalid Mammadov <kh...@gmail.com>.
Looks great!
QQ: will user able to run this pipeline from normal code? I.e. can I
trigger a pipeline from *driver* code based on some condition etc. or it
must be executed via separate shell command ?
As a background Databricks imposes similar limitation where as you cannot
run normal Spark code and DLT on the same cluster for some reason and
forces to use two clusters increasing the cost and latency.
On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
> Hi all – starting a discussion thread for a SPIP that I've been working on
> with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA
> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>
> ].
>
> The SPIP proposes extending Spark's lazy, declarative execution model
> beyond single queries, to pipelines that keep multiple datasets up to date.
> It introduces the ability to compose multiple transformations into a single
> declarative dataflow graph.
>
> Declarative pipelines aim to simplify the development and management of
> data pipelines, by removing the need for manual orchestration of
> dependencies and making it possible to catch many errors before any
> execution steps are launched.
>
> Declarative pipelines can include both batch and streaming computations,
> leveraging Structured Streaming for stream processing and new materialized
> view syntax for batch processing. Tight integration with Spark SQL's
> analyzer enables deeper analysis and earlier error detection than is
> achievable with more generic frameworks.
>
> Let us know what you think!
>
>
Re: [DISCUSS] SPIP: Declarative Pipelines
Posted by Nicholas Chammas <ni...@gmail.com>.
There are many projects in the Spark ecosystem — like Deequ and Great Expectations — that are focused on expressing and enforcing data quality checks.
In the more complex cases, these checks do not fit the scope of the checks that a typical data source may support (i.e. PK, FK, CHECK), so these projects have designed their own APIs for expressing various constraints, plus their own analyzers for figuring out how to run all the checks with as little repeated scanning or computation as possible.
It’s a bit forward looking since this is still a proposal, but I wonder if the Pipelines API will eventually be able to address this kind of use case directly. I believe arbitrary data constraints can be expressed naturally as materialized views in a pipeline <https://nchammas.com/writing/query-language-constraint-language>, and a hypothetical Pipelines API could look something like this:
@pipelines.assertion
def sufficient_on_call_coverage():
return (
spark.table(“doctors")
.where(col("on_call"))
.select(count("*") >= 1)
)
With visibility into the dependencies of a given assertion or constraint, a Pipeline could perhaps figure out how to enforce it without wasting resources rereading or recomputing stuff across nodes in the graph.
Again, this is a bit forward looking, but is this kind of idea “on topic” for the Pipelines API?
Nick
> On Apr 5, 2025, at 5:30 PM, Sandy Ryza <sa...@apache.org> wrote:
>
> Hi all – starting a discussion thread for a SPIP that I've been working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0>].
>
> The SPIP proposes extending Spark's lazy, declarative execution model beyond single queries, to pipelines that keep multiple datasets up to date. It introduces the ability to compose multiple transformations into a single declarative dataflow graph.
>
> Declarative pipelines aim to simplify the development and management of data pipelines, by removing the need for manual orchestration of dependencies and making it possible to catch many errors before any execution steps are launched.
>
> Declarative pipelines can include both batch and streaming computations, leveraging Structured Streaming for stream processing and new materialized view syntax for batch processing. Tight integration with Spark SQL's analyzer enables deeper analysis and earlier error detection than is achievable with more generic frameworks.
>
> Let us know what you think!
>