You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Vladimir Golubev <vv...@gmail.com> on 2024/09/30 13:44:26 UTC

[VOTE] Single-pass Analyzer for Catalyst

Hi all,

I’d like to start a vote for a single-pass Analyzer for the Catalyst
project. This project will introduce a new analysis framework to the
Catalyst, which will eventually replace the fixed-point one.

Please refer to the SPIP jira:
https://issues.apache.org/jira/browse/SPARK-49834

[ ] +1: Accept the proposal
[ ] +0
[ ] -1: I don’t think this is a good idea because …

Thanks!
Vladimir

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Yang Jie <ya...@apache.org>.
+1, Thanks

Jie Yang

On 2024/10/01 03:26:40 John Zhuge wrote:
> +1 (non-binding)
> 
> On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
> <ge...@databricks.com.invalid> wrote:
> 
> > +1
> >
> > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >
> >> +1 (non-binding), promising proposal!
> >>
> >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
> >>
> >>> Thank you for the swift clarification, Reynold and Xiao.
> >>>
> >>> It seems that the Target Version was set mistakenly initially.
> >>>
> >>> I removed the `Target Version` from the SPIP JIRA.
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-49834
> >>>
> >>> I'm switching my cast to +1 for this SPIP vote.
> >>>
> >>> Thanks,
> >>> Dongjoon.
> >>>
> >>> On 2024/09/30 22:55:41 Xiao Li wrote:
> >>> > +1 in support of the direction of the Single-pass Analyzer for
> >>> Catalyst.
> >>> >
> >>> > I think we should not have a target version for the new Catalyst
> >>> SPARK-49834
> >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be
> >>> a
> >>> > blocker for Spark 4.0. When implementing the new analyzer, the code
> >>> changes
> >>> > must not affect users of the existing analyzer to avoid any user-facing
> >>> > impacts.
> >>> >
> >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
> >>> >
> >>> > > I don't actually "lead" this. But I don't think this needs to target
> >>> a
> >>> > > specific Spark version given it should not have any user facing
> >>> > > consequences?
> >>> > >
> >>> > >
> >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
> >>> wrote:
> >>> > >
> >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
> >>> > >>
> >>> > >> I'm wondering if this is really achievable goal for Apache Spark
> >>> 4.0.0.
> >>> > >>
> >>> > >> If it's expected that we are unable to deliver it, shall we
> >>> postpone this
> >>> > >> vote until 4.1.0 planning?
> >>> > >>
> >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> >>> > >>
> >>> > >> -1 from my side.
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Dongjoon.
> >>> > >>
> >>> > >>
> >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> >>> > >> > +1
> >>> > >> >
> >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
> >>> <rxin@databricks.com.invalid
> >>> > >> >
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > +1
> >>> > >> > >
> >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
> >>> vvdr.glb@gmail.com>
> >>> > >> > > wrote:
> >>> > >> > >
> >>> > >> > >> Hi all,
> >>> > >> > >>
> >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
> >>> Catalyst
> >>> > >> > >> project. This project will introduce a new analysis framework
> >>> to the
> >>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
> >>> > >> > >>
> >>> > >> > >> Please refer to the SPIP jira:
> >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> >>> > >> > >>
> >>> > >> > >> [ ] +1: Accept the proposal
> >>> > >> > >> [ ] +0
> >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
> >>> > >> > >>
> >>> > >> > >> Thanks!
> >>> > >> > >>
> >>> > >> > >> Vladimir
> >>> > >> > >>
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >>
> >>> ---------------------------------------------------------------------
> >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>> > >>
> >>> > >>
> >>> >
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >>>
> 
> -- 
> John Zhuge
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by "L. C. Hsieh" <vi...@gmail.com>.
+1

On Thu, Oct 3, 2024 at 7:31 AM Wenchen Fan <cl...@gmail.com> wrote:
>
> +1
>
> On Wed, Oct 2, 2024 at 7:50 AM Peter Toth <pe...@gmail.com> wrote:
>>
>> +1
>>
>>
>> On Tue, Oct 1, 2024, 08:33 Yang Jie <ya...@apache.org> wrote:
>>>
>>> +1, Thanks
>>>
>>> Jie Yang
>>>
>>> On 2024/10/01 03:26:40 John Zhuge wrote:
>>> > +1 (non-binding)
>>> >
>>> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
>>> > <ge...@databricks.com.invalid> wrote:
>>> >
>>> > > +1
>>> > >
>>> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <ka...@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> +1 (non-binding), promising proposal!
>>> > >>
>>> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>>> > >>
>>> > >>> Thank you for the swift clarification, Reynold and Xiao.
>>> > >>>
>>> > >>> It seems that the Target Version was set mistakenly initially.
>>> > >>>
>>> > >>> I removed the `Target Version` from the SPIP JIRA.
>>> > >>>
>>> > >>> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >>>
>>> > >>> I'm switching my cast to +1 for this SPIP vote.
>>> > >>>
>>> > >>> Thanks,
>>> > >>> Dongjoon.
>>> > >>>
>>> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
>>> > >>> > +1 in support of the direction of the Single-pass Analyzer for
>>> > >>> Catalyst.
>>> > >>> >
>>> > >>> > I think we should not have a target version for the new Catalyst
>>> > >>> SPARK-49834
>>> > >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be
>>> > >>> a
>>> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the code
>>> > >>> changes
>>> > >>> > must not affect users of the existing analyzer to avoid any user-facing
>>> > >>> > impacts.
>>> > >>> >
>>> > >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>>> > >>> >
>>> > >>> > > I don't actually "lead" this. But I don't think this needs to target
>>> > >>> a
>>> > >>> > > specific Spark version given it should not have any user facing
>>> > >>> > > consequences?
>>> > >>> > >
>>> > >>> > >
>>> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
>>> > >>> wrote:
>>> > >>> > >
>>> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>>> > >>> > >>
>>> > >>> > >> I'm wondering if this is really achievable goal for Apache Spark
>>> > >>> 4.0.0.
>>> > >>> > >>
>>> > >>> > >> If it's expected that we are unable to deliver it, shall we
>>> > >>> postpone this
>>> > >>> > >> vote until 4.1.0 planning?
>>> > >>> > >>
>>> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>> > >>> > >>
>>> > >>> > >> -1 from my side.
>>> > >>> > >>
>>> > >>> > >> Thanks,
>>> > >>> > >> Dongjoon.
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>> > >>> > >> > +1
>>> > >>> > >> >
>>> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>> > >>> <rxin@databricks.com.invalid
>>> > >>> > >> >
>>> > >>> > >> > wrote:
>>> > >>> > >> >
>>> > >>> > >> > > +1
>>> > >>> > >> > >
>>> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>>> > >>> vvdr.glb@gmail.com>
>>> > >>> > >> > > wrote:
>>> > >>> > >> > >
>>> > >>> > >> > >> Hi all,
>>> > >>> > >> > >>
>>> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
>>> > >>> Catalyst
>>> > >>> > >> > >> project. This project will introduce a new analysis framework
>>> > >>> to the
>>> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
>>> > >>> > >> > >>
>>> > >>> > >> > >> Please refer to the SPIP jira:
>>> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >>> > >> > >>
>>> > >>> > >> > >> [ ] +1: Accept the proposal
>>> > >>> > >> > >> [ ] +0
>>> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>>> > >>> > >> > >>
>>> > >>> > >> > >> Thanks!
>>> > >>> > >> > >>
>>> > >>> > >> > >> Vladimir
>>> > >>> > >> > >>
>>> > >>> > >> > >
>>> > >>> > >> >
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> ---------------------------------------------------------------------
>>> > >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> >
>>> > >>>
>>> > >>> ---------------------------------------------------------------------
>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>>
>>> > >>>
>>> >
>>> > --
>>> > John Zhuge
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Mich Talebzadeh <mi...@gmail.com>.
+1 on the assumption that we should phase this release on an incremental
basis. Probably will take us to end of release 5.

HTH

Mich Talebzadeh,

Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>


 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".


On Thu, 3 Oct 2024 at 21:50, Ángel <an...@gmail.com> wrote:

> +1
>
> El jue, 3 oct 2024, 20:06, Wenchen Fan <cl...@gmail.com> escribió:
>
>> +1
>>
>> On Wed, Oct 2, 2024 at 7:50 AM Peter Toth <pe...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Tue, Oct 1, 2024, 08:33 Yang Jie <ya...@apache.org> wrote:
>>>
>>>> +1, Thanks
>>>>
>>>> Jie Yang
>>>>
>>>> On 2024/10/01 03:26:40 John Zhuge wrote:
>>>> > +1 (non-binding)
>>>> >
>>>> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
>>>> > <ge...@databricks.com.invalid> wrote:
>>>> >
>>>> > > +1
>>>> > >
>>>> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
>>>> kabhwan.opensource@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > >> +1 (non-binding), promising proposal!
>>>> > >>
>>>> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이
>>>> 작성:
>>>> > >>
>>>> > >>> Thank you for the swift clarification, Reynold and Xiao.
>>>> > >>>
>>>> > >>> It seems that the Target Version was set mistakenly initially.
>>>> > >>>
>>>> > >>> I removed the `Target Version` from the SPIP JIRA.
>>>> > >>>
>>>> > >>> https://issues.apache.org/jira/browse/SPARK-49834
>>>> > >>>
>>>> > >>> I'm switching my cast to +1 for this SPIP vote.
>>>> > >>>
>>>> > >>> Thanks,
>>>> > >>> Dongjoon.
>>>> > >>>
>>>> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
>>>> > >>> > +1 in support of the direction of the Single-pass Analyzer for
>>>> > >>> Catalyst.
>>>> > >>> >
>>>> > >>> > I think we should not have a target version for the new Catalyst
>>>> > >>> SPARK-49834
>>>> > >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should
>>>> not be
>>>> > >>> a
>>>> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the
>>>> code
>>>> > >>> changes
>>>> > >>> > must not affect users of the existing analyzer to avoid any
>>>> user-facing
>>>> > >>> > impacts.
>>>> > >>> >
>>>> > >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一
>>>> 15:39写道:
>>>> > >>> >
>>>> > >>> > > I don't actually "lead" this. But I don't think this needs to
>>>> target
>>>> > >>> a
>>>> > >>> > > specific Spark version given it should not have any user
>>>> facing
>>>> > >>> > > consequences?
>>>> > >>> > >
>>>> > >>> > >
>>>> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <
>>>> dongjoon@apache.org>
>>>> > >>> wrote:
>>>> > >>> > >
>>>> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>>>> > >>> > >>
>>>> > >>> > >> I'm wondering if this is really achievable goal for Apache
>>>> Spark
>>>> > >>> 4.0.0.
>>>> > >>> > >>
>>>> > >>> > >> If it's expected that we are unable to deliver it, shall we
>>>> > >>> postpone this
>>>> > >>> > >> vote until 4.1.0 planning?
>>>> > >>> > >>
>>>> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0
>>>> explicitly,
>>>> > >>> > >>
>>>> > >>> > >> -1 from my side.
>>>> > >>> > >>
>>>> > >>> > >> Thanks,
>>>> > >>> > >> Dongjoon.
>>>> > >>> > >>
>>>> > >>> > >>
>>>> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>>> > >>> > >> > +1
>>>> > >>> > >> >
>>>> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>>> > >>> <rxin@databricks.com.invalid
>>>> > >>> > >> >
>>>> > >>> > >> > wrote:
>>>> > >>> > >> >
>>>> > >>> > >> > > +1
>>>> > >>> > >> > >
>>>> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>>>> > >>> vvdr.glb@gmail.com>
>>>> > >>> > >> > > wrote:
>>>> > >>> > >> > >
>>>> > >>> > >> > >> Hi all,
>>>> > >>> > >> > >>
>>>> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for
>>>> the
>>>> > >>> Catalyst
>>>> > >>> > >> > >> project. This project will introduce a new analysis
>>>> framework
>>>> > >>> to the
>>>> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point
>>>> one.
>>>> > >>> > >> > >>
>>>> > >>> > >> > >> Please refer to the SPIP jira:
>>>> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>>> > >>> > >> > >>
>>>> > >>> > >> > >> [ ] +1: Accept the proposal
>>>> > >>> > >> > >> [ ] +0
>>>> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>>>> > >>> > >> > >>
>>>> > >>> > >> > >> Thanks!
>>>> > >>> > >> > >>
>>>> > >>> > >> > >> Vladimir
>>>> > >>> > >> > >>
>>>> > >>> > >> > >
>>>> > >>> > >> >
>>>> > >>> > >>
>>>> > >>> > >>
>>>> > >>>
>>>> ---------------------------------------------------------------------
>>>> > >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >>> > >>
>>>> > >>> > >>
>>>> > >>> >
>>>> > >>>
>>>> > >>>
>>>> ---------------------------------------------------------------------
>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >>>
>>>> > >>>
>>>> >
>>>> > --
>>>> > John Zhuge
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Ángel <an...@gmail.com>.
+1

El jue, 3 oct 2024, 20:06, Wenchen Fan <cl...@gmail.com> escribió:

> +1
>
> On Wed, Oct 2, 2024 at 7:50 AM Peter Toth <pe...@gmail.com> wrote:
>
>> +1
>>
>> On Tue, Oct 1, 2024, 08:33 Yang Jie <ya...@apache.org> wrote:
>>
>>> +1, Thanks
>>>
>>> Jie Yang
>>>
>>> On 2024/10/01 03:26:40 John Zhuge wrote:
>>> > +1 (non-binding)
>>> >
>>> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
>>> > <ge...@databricks.com.invalid> wrote:
>>> >
>>> > > +1
>>> > >
>>> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
>>> kabhwan.opensource@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> +1 (non-binding), promising proposal!
>>> > >>
>>> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>>> > >>
>>> > >>> Thank you for the swift clarification, Reynold and Xiao.
>>> > >>>
>>> > >>> It seems that the Target Version was set mistakenly initially.
>>> > >>>
>>> > >>> I removed the `Target Version` from the SPIP JIRA.
>>> > >>>
>>> > >>> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >>>
>>> > >>> I'm switching my cast to +1 for this SPIP vote.
>>> > >>>
>>> > >>> Thanks,
>>> > >>> Dongjoon.
>>> > >>>
>>> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
>>> > >>> > +1 in support of the direction of the Single-pass Analyzer for
>>> > >>> Catalyst.
>>> > >>> >
>>> > >>> > I think we should not have a target version for the new Catalyst
>>> > >>> SPARK-49834
>>> > >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should
>>> not be
>>> > >>> a
>>> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the
>>> code
>>> > >>> changes
>>> > >>> > must not affect users of the existing analyzer to avoid any
>>> user-facing
>>> > >>> > impacts.
>>> > >>> >
>>> > >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>>> > >>> >
>>> > >>> > > I don't actually "lead" this. But I don't think this needs to
>>> target
>>> > >>> a
>>> > >>> > > specific Spark version given it should not have any user facing
>>> > >>> > > consequences?
>>> > >>> > >
>>> > >>> > >
>>> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <
>>> dongjoon@apache.org>
>>> > >>> wrote:
>>> > >>> > >
>>> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>>> > >>> > >>
>>> > >>> > >> I'm wondering if this is really achievable goal for Apache
>>> Spark
>>> > >>> 4.0.0.
>>> > >>> > >>
>>> > >>> > >> If it's expected that we are unable to deliver it, shall we
>>> > >>> postpone this
>>> > >>> > >> vote until 4.1.0 planning?
>>> > >>> > >>
>>> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0
>>> explicitly,
>>> > >>> > >>
>>> > >>> > >> -1 from my side.
>>> > >>> > >>
>>> > >>> > >> Thanks,
>>> > >>> > >> Dongjoon.
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>> > >>> > >> > +1
>>> > >>> > >> >
>>> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>> > >>> <rxin@databricks.com.invalid
>>> > >>> > >> >
>>> > >>> > >> > wrote:
>>> > >>> > >> >
>>> > >>> > >> > > +1
>>> > >>> > >> > >
>>> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>>> > >>> vvdr.glb@gmail.com>
>>> > >>> > >> > > wrote:
>>> > >>> > >> > >
>>> > >>> > >> > >> Hi all,
>>> > >>> > >> > >>
>>> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for
>>> the
>>> > >>> Catalyst
>>> > >>> > >> > >> project. This project will introduce a new analysis
>>> framework
>>> > >>> to the
>>> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point
>>> one.
>>> > >>> > >> > >>
>>> > >>> > >> > >> Please refer to the SPIP jira:
>>> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >>> > >> > >>
>>> > >>> > >> > >> [ ] +1: Accept the proposal
>>> > >>> > >> > >> [ ] +0
>>> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>>> > >>> > >> > >>
>>> > >>> > >> > >> Thanks!
>>> > >>> > >> > >>
>>> > >>> > >> > >> Vladimir
>>> > >>> > >> > >>
>>> > >>> > >> > >
>>> > >>> > >> >
>>> > >>> > >>
>>> > >>> > >>
>>> > >>>
>>> ---------------------------------------------------------------------
>>> > >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> >
>>> > >>>
>>> > >>>
>>> ---------------------------------------------------------------------
>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>>
>>> > >>>
>>> >
>>> > --
>>> > John Zhuge
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Wenchen Fan <cl...@gmail.com>.
+1

On Wed, Oct 2, 2024 at 7:50 AM Peter Toth <pe...@gmail.com> wrote:

> +1
>
> On Tue, Oct 1, 2024, 08:33 Yang Jie <ya...@apache.org> wrote:
>
>> +1, Thanks
>>
>> Jie Yang
>>
>> On 2024/10/01 03:26:40 John Zhuge wrote:
>> > +1 (non-binding)
>> >
>> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
>> > <ge...@databricks.com.invalid> wrote:
>> >
>> > > +1
>> > >
>> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com>
>> > > wrote:
>> > >
>> > >> +1 (non-binding), promising proposal!
>> > >>
>> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>> > >>
>> > >>> Thank you for the swift clarification, Reynold and Xiao.
>> > >>>
>> > >>> It seems that the Target Version was set mistakenly initially.
>> > >>>
>> > >>> I removed the `Target Version` from the SPIP JIRA.
>> > >>>
>> > >>> https://issues.apache.org/jira/browse/SPARK-49834
>> > >>>
>> > >>> I'm switching my cast to +1 for this SPIP vote.
>> > >>>
>> > >>> Thanks,
>> > >>> Dongjoon.
>> > >>>
>> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
>> > >>> > +1 in support of the direction of the Single-pass Analyzer for
>> > >>> Catalyst.
>> > >>> >
>> > >>> > I think we should not have a target version for the new Catalyst
>> > >>> SPARK-49834
>> > >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should
>> not be
>> > >>> a
>> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the
>> code
>> > >>> changes
>> > >>> > must not affect users of the existing analyzer to avoid any
>> user-facing
>> > >>> > impacts.
>> > >>> >
>> > >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>> > >>> >
>> > >>> > > I don't actually "lead" this. But I don't think this needs to
>> target
>> > >>> a
>> > >>> > > specific Spark version given it should not have any user facing
>> > >>> > > consequences?
>> > >>> > >
>> > >>> > >
>> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <
>> dongjoon@apache.org>
>> > >>> wrote:
>> > >>> > >
>> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>> > >>> > >>
>> > >>> > >> I'm wondering if this is really achievable goal for Apache
>> Spark
>> > >>> 4.0.0.
>> > >>> > >>
>> > >>> > >> If it's expected that we are unable to deliver it, shall we
>> > >>> postpone this
>> > >>> > >> vote until 4.1.0 planning?
>> > >>> > >>
>> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0
>> explicitly,
>> > >>> > >>
>> > >>> > >> -1 from my side.
>> > >>> > >>
>> > >>> > >> Thanks,
>> > >>> > >> Dongjoon.
>> > >>> > >>
>> > >>> > >>
>> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>> > >>> > >> > +1
>> > >>> > >> >
>> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>> > >>> <rxin@databricks.com.invalid
>> > >>> > >> >
>> > >>> > >> > wrote:
>> > >>> > >> >
>> > >>> > >> > > +1
>> > >>> > >> > >
>> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>> > >>> vvdr.glb@gmail.com>
>> > >>> > >> > > wrote:
>> > >>> > >> > >
>> > >>> > >> > >> Hi all,
>> > >>> > >> > >>
>> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for
>> the
>> > >>> Catalyst
>> > >>> > >> > >> project. This project will introduce a new analysis
>> framework
>> > >>> to the
>> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point
>> one.
>> > >>> > >> > >>
>> > >>> > >> > >> Please refer to the SPIP jira:
>> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>> > >>> > >> > >>
>> > >>> > >> > >> [ ] +1: Accept the proposal
>> > >>> > >> > >> [ ] +0
>> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>> > >>> > >> > >>
>> > >>> > >> > >> Thanks!
>> > >>> > >> > >>
>> > >>> > >> > >> Vladimir
>> > >>> > >> > >>
>> > >>> > >> > >
>> > >>> > >> >
>> > >>> > >>
>> > >>> > >>
>> > >>>
>> ---------------------------------------------------------------------
>> > >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >>> > >>
>> > >>> > >>
>> > >>> >
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------
>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >>>
>> > >>>
>> >
>> > --
>> > John Zhuge
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Peter Toth <pe...@gmail.com>.
+1

On Tue, Oct 1, 2024, 08:33 Yang Jie <ya...@apache.org> wrote:

> +1, Thanks
>
> Jie Yang
>
> On 2024/10/01 03:26:40 John Zhuge wrote:
> > +1 (non-binding)
> >
> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
> > <ge...@databricks.com.invalid> wrote:
> >
> > > +1
> > >
> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
> kabhwan.opensource@gmail.com>
> > > wrote:
> > >
> > >> +1 (non-binding), promising proposal!
> > >>
> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
> > >>
> > >>> Thank you for the swift clarification, Reynold and Xiao.
> > >>>
> > >>> It seems that the Target Version was set mistakenly initially.
> > >>>
> > >>> I removed the `Target Version` from the SPIP JIRA.
> > >>>
> > >>> https://issues.apache.org/jira/browse/SPARK-49834
> > >>>
> > >>> I'm switching my cast to +1 for this SPIP vote.
> > >>>
> > >>> Thanks,
> > >>> Dongjoon.
> > >>>
> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
> > >>> > +1 in support of the direction of the Single-pass Analyzer for
> > >>> Catalyst.
> > >>> >
> > >>> > I think we should not have a target version for the new Catalyst
> > >>> SPARK-49834
> > >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should
> not be
> > >>> a
> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the code
> > >>> changes
> > >>> > must not affect users of the existing analyzer to avoid any
> user-facing
> > >>> > impacts.
> > >>> >
> > >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
> > >>> >
> > >>> > > I don't actually "lead" this. But I don't think this needs to
> target
> > >>> a
> > >>> > > specific Spark version given it should not have any user facing
> > >>> > > consequences?
> > >>> > >
> > >>> > >
> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <
> dongjoon@apache.org>
> > >>> wrote:
> > >>> > >
> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
> > >>> > >>
> > >>> > >> I'm wondering if this is really achievable goal for Apache Spark
> > >>> 4.0.0.
> > >>> > >>
> > >>> > >> If it's expected that we are unable to deliver it, shall we
> > >>> postpone this
> > >>> > >> vote until 4.1.0 planning?
> > >>> > >>
> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> > >>> > >>
> > >>> > >> -1 from my side.
> > >>> > >>
> > >>> > >> Thanks,
> > >>> > >> Dongjoon.
> > >>> > >>
> > >>> > >>
> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> > >>> > >> > +1
> > >>> > >> >
> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
> > >>> <rxin@databricks.com.invalid
> > >>> > >> >
> > >>> > >> > wrote:
> > >>> > >> >
> > >>> > >> > > +1
> > >>> > >> > >
> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
> > >>> vvdr.glb@gmail.com>
> > >>> > >> > > wrote:
> > >>> > >> > >
> > >>> > >> > >> Hi all,
> > >>> > >> > >>
> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
> > >>> Catalyst
> > >>> > >> > >> project. This project will introduce a new analysis
> framework
> > >>> to the
> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point
> one.
> > >>> > >> > >>
> > >>> > >> > >> Please refer to the SPIP jira:
> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> > >>> > >> > >>
> > >>> > >> > >> [ ] +1: Accept the proposal
> > >>> > >> > >> [ ] +0
> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
> > >>> > >> > >>
> > >>> > >> > >> Thanks!
> > >>> > >> > >>
> > >>> > >> > >> Vladimir
> > >>> > >> > >>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> > >>
> > >>> ---------------------------------------------------------------------
> > >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >>> > >>
> > >>> > >>
> > >>> >
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >>>
> > >>>
> >
> > --
> > John Zhuge
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Yang Jie <ya...@apache.org>.
+1, Thanks

Jie Yang

On 2024/10/01 03:26:40 John Zhuge wrote:
> +1 (non-binding)
> 
> On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
> <ge...@databricks.com.invalid> wrote:
> 
> > +1
> >
> > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >
> >> +1 (non-binding), promising proposal!
> >>
> >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
> >>
> >>> Thank you for the swift clarification, Reynold and Xiao.
> >>>
> >>> It seems that the Target Version was set mistakenly initially.
> >>>
> >>> I removed the `Target Version` from the SPIP JIRA.
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-49834
> >>>
> >>> I'm switching my cast to +1 for this SPIP vote.
> >>>
> >>> Thanks,
> >>> Dongjoon.
> >>>
> >>> On 2024/09/30 22:55:41 Xiao Li wrote:
> >>> > +1 in support of the direction of the Single-pass Analyzer for
> >>> Catalyst.
> >>> >
> >>> > I think we should not have a target version for the new Catalyst
> >>> SPARK-49834
> >>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be
> >>> a
> >>> > blocker for Spark 4.0. When implementing the new analyzer, the code
> >>> changes
> >>> > must not affect users of the existing analyzer to avoid any user-facing
> >>> > impacts.
> >>> >
> >>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
> >>> >
> >>> > > I don't actually "lead" this. But I don't think this needs to target
> >>> a
> >>> > > specific Spark version given it should not have any user facing
> >>> > > consequences?
> >>> > >
> >>> > >
> >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
> >>> wrote:
> >>> > >
> >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
> >>> > >>
> >>> > >> I'm wondering if this is really achievable goal for Apache Spark
> >>> 4.0.0.
> >>> > >>
> >>> > >> If it's expected that we are unable to deliver it, shall we
> >>> postpone this
> >>> > >> vote until 4.1.0 planning?
> >>> > >>
> >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> >>> > >>
> >>> > >> -1 from my side.
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Dongjoon.
> >>> > >>
> >>> > >>
> >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> >>> > >> > +1
> >>> > >> >
> >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
> >>> <rxin@databricks.com.invalid
> >>> > >> >
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > +1
> >>> > >> > >
> >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
> >>> vvdr.glb@gmail.com>
> >>> > >> > > wrote:
> >>> > >> > >
> >>> > >> > >> Hi all,
> >>> > >> > >>
> >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
> >>> Catalyst
> >>> > >> > >> project. This project will introduce a new analysis framework
> >>> to the
> >>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
> >>> > >> > >>
> >>> > >> > >> Please refer to the SPIP jira:
> >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> >>> > >> > >>
> >>> > >> > >> [ ] +1: Accept the proposal
> >>> > >> > >> [ ] +0
> >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
> >>> > >> > >>
> >>> > >> > >> Thanks!
> >>> > >> > >>
> >>> > >> > >> Vladimir
> >>> > >> > >>
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >>
> >>> ---------------------------------------------------------------------
> >>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>> > >>
> >>> > >>
> >>> >
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >>>
> 
> -- 
> John Zhuge
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Vladimir Golubev <vv...@gmail.com>.
Folks, thanks for voting!

Yes, the target version was set by my mistake, Dongjoon. Sorry for that.

Vladimir.

On Tue, Oct 1, 2024, 05:28 John Zhuge <jz...@apache.org> wrote:

> +1 (non-binding)
>
> On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
> <ge...@databricks.com.invalid> wrote:
>
>> +1
>>
>> On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> +1 (non-binding), promising proposal!
>>>
>>> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>>>
>>>> Thank you for the swift clarification, Reynold and Xiao.
>>>>
>>>> It seems that the Target Version was set mistakenly initially.
>>>>
>>>> I removed the `Target Version` from the SPIP JIRA.
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-49834
>>>>
>>>> I'm switching my cast to +1 for this SPIP vote.
>>>>
>>>> Thanks,
>>>> Dongjoon.
>>>>
>>>> On 2024/09/30 22:55:41 Xiao Li wrote:
>>>> > +1 in support of the direction of the Single-pass Analyzer for
>>>> Catalyst.
>>>> >
>>>> > I think we should not have a target version for the new Catalyst
>>>> SPARK-49834
>>>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not
>>>> be a
>>>> > blocker for Spark 4.0. When implementing the new analyzer, the code
>>>> changes
>>>> > must not affect users of the existing analyzer to avoid any
>>>> user-facing
>>>> > impacts.
>>>> >
>>>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>>>> >
>>>> > > I don't actually "lead" this. But I don't think this needs to
>>>> target a
>>>> > > specific Spark version given it should not have any user facing
>>>> > > consequences?
>>>> > >
>>>> > >
>>>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>>>> > >>
>>>> > >> I'm wondering if this is really achievable goal for Apache Spark
>>>> 4.0.0.
>>>> > >>
>>>> > >> If it's expected that we are unable to deliver it, shall we
>>>> postpone this
>>>> > >> vote until 4.1.0 planning?
>>>> > >>
>>>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>>> > >>
>>>> > >> -1 from my side.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Dongjoon.
>>>> > >>
>>>> > >>
>>>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>>> > >> > +1
>>>> > >> >
>>>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>>> <rxin@databricks.com.invalid
>>>> > >> >
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > > +1
>>>> > >> > >
>>>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>>>> vvdr.glb@gmail.com>
>>>> > >> > > wrote:
>>>> > >> > >
>>>> > >> > >> Hi all,
>>>> > >> > >>
>>>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
>>>> Catalyst
>>>> > >> > >> project. This project will introduce a new analysis framework
>>>> to the
>>>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
>>>> > >> > >>
>>>> > >> > >> Please refer to the SPIP jira:
>>>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>>> > >> > >>
>>>> > >> > >> [ ] +1: Accept the proposal
>>>> > >> > >> [ ] +0
>>>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>>>> > >> > >>
>>>> > >> > >> Thanks!
>>>> > >> > >>
>>>> > >> > >> Vladimir
>>>> > >> > >>
>>>> > >> > >
>>>> > >> >
>>>> > >>
>>>> > >>
>>>> ---------------------------------------------------------------------
>>>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >>
>>>> > >>
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>
> --
> John Zhuge
>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by John Zhuge <jz...@apache.org>.
+1 (non-binding)

On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
<ge...@databricks.com.invalid> wrote:

> +1
>
> On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> +1 (non-binding), promising proposal!
>>
>> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>>
>>> Thank you for the swift clarification, Reynold and Xiao.
>>>
>>> It seems that the Target Version was set mistakenly initially.
>>>
>>> I removed the `Target Version` from the SPIP JIRA.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-49834
>>>
>>> I'm switching my cast to +1 for this SPIP vote.
>>>
>>> Thanks,
>>> Dongjoon.
>>>
>>> On 2024/09/30 22:55:41 Xiao Li wrote:
>>> > +1 in support of the direction of the Single-pass Analyzer for
>>> Catalyst.
>>> >
>>> > I think we should not have a target version for the new Catalyst
>>> SPARK-49834
>>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be
>>> a
>>> > blocker for Spark 4.0. When implementing the new analyzer, the code
>>> changes
>>> > must not affect users of the existing analyzer to avoid any user-facing
>>> > impacts.
>>> >
>>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>>> >
>>> > > I don't actually "lead" this. But I don't think this needs to target
>>> a
>>> > > specific Spark version given it should not have any user facing
>>> > > consequences?
>>> > >
>>> > >
>>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>> > >
>>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>>> > >>
>>> > >> I'm wondering if this is really achievable goal for Apache Spark
>>> 4.0.0.
>>> > >>
>>> > >> If it's expected that we are unable to deliver it, shall we
>>> postpone this
>>> > >> vote until 4.1.0 planning?
>>> > >>
>>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>> > >>
>>> > >> -1 from my side.
>>> > >>
>>> > >> Thanks,
>>> > >> Dongjoon.
>>> > >>
>>> > >>
>>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>> > >> > +1
>>> > >> >
>>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>> <rxin@databricks.com.invalid
>>> > >> >
>>> > >> > wrote:
>>> > >> >
>>> > >> > > +1
>>> > >> > >
>>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>>> vvdr.glb@gmail.com>
>>> > >> > > wrote:
>>> > >> > >
>>> > >> > >> Hi all,
>>> > >> > >>
>>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
>>> Catalyst
>>> > >> > >> project. This project will introduce a new analysis framework
>>> to the
>>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
>>> > >> > >>
>>> > >> > >> Please refer to the SPIP jira:
>>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >> > >>
>>> > >> > >> [ ] +1: Accept the proposal
>>> > >> > >> [ ] +0
>>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>>> > >> > >>
>>> > >> > >> Thanks!
>>> > >> > >>
>>> > >> > >> Vladimir
>>> > >> > >>
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >>
>>> ---------------------------------------------------------------------
>>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>
>>> > >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

-- 
John Zhuge

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Gengliang Wang <ge...@databricks.com.INVALID>.
+1

On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> +1 (non-binding), promising proposal!
>
> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:
>
>> Thank you for the swift clarification, Reynold and Xiao.
>>
>> It seems that the Target Version was set mistakenly initially.
>>
>> I removed the `Target Version` from the SPIP JIRA.
>>
>> https://issues.apache.org/jira/browse/SPARK-49834
>>
>> I'm switching my cast to +1 for this SPIP vote.
>>
>> Thanks,
>> Dongjoon.
>>
>> On 2024/09/30 22:55:41 Xiao Li wrote:
>> > +1 in support of the direction of the Single-pass Analyzer for Catalyst.
>> >
>> > I think we should not have a target version for the new Catalyst
>> SPARK-49834
>> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be a
>> > blocker for Spark 4.0. When implementing the new analyzer, the code
>> changes
>> > must not affect users of the existing analyzer to avoid any user-facing
>> > impacts.
>> >
>> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
>> >
>> > > I don't actually "lead" this. But I don't think this needs to target a
>> > > specific Spark version given it should not have any user facing
>> > > consequences?
>> > >
>> > >
>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>> > >
>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
>> > >>
>> > >> I'm wondering if this is really achievable goal for Apache Spark
>> 4.0.0.
>> > >>
>> > >> If it's expected that we are unable to deliver it, shall we postpone
>> this
>> > >> vote until 4.1.0 planning?
>> > >>
>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>> > >>
>> > >> -1 from my side.
>> > >>
>> > >> Thanks,
>> > >> Dongjoon.
>> > >>
>> > >>
>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>> > >> > +1
>> > >> >
>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>> <rxin@databricks.com.invalid
>> > >> >
>> > >> > wrote:
>> > >> >
>> > >> > > +1
>> > >> > >
>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
>> vvdr.glb@gmail.com>
>> > >> > > wrote:
>> > >> > >
>> > >> > >> Hi all,
>> > >> > >>
>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
>> Catalyst
>> > >> > >> project. This project will introduce a new analysis framework
>> to the
>> > >> > >> Catalyst, which will eventually replace the fixed-point one.
>> > >> > >>
>> > >> > >> Please refer to the SPIP jira:
>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
>> > >> > >>
>> > >> > >> [ ] +1: Accept the proposal
>> > >> > >> [ ] +0
>> > >> > >> [ ] -1: I don’t think this is a good idea because …
>> > >> > >>
>> > >> > >> Thanks!
>> > >> > >>
>> > >> > >> Vladimir
>> > >> > >>
>> > >> > >
>> > >> >
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >>
>> > >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Jungtaek Lim <ka...@gmail.com>.
+1 (non-binding), promising proposal!

2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun <do...@apache.org>님이 작성:

> Thank you for the swift clarification, Reynold and Xiao.
>
> It seems that the Target Version was set mistakenly initially.
>
> I removed the `Target Version` from the SPIP JIRA.
>
> https://issues.apache.org/jira/browse/SPARK-49834
>
> I'm switching my cast to +1 for this SPIP vote.
>
> Thanks,
> Dongjoon.
>
> On 2024/09/30 22:55:41 Xiao Li wrote:
> > +1 in support of the direction of the Single-pass Analyzer for Catalyst.
> >
> > I think we should not have a target version for the new Catalyst
> SPARK-49834
> > <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be a
> > blocker for Spark 4.0. When implementing the new analyzer, the code
> changes
> > must not affect users of the existing analyzer to avoid any user-facing
> > impacts.
> >
> > Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
> >
> > > I don't actually "lead" this. But I don't think this needs to target a
> > > specific Spark version given it should not have any user facing
> > > consequences?
> > >
> > >
> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
> wrote:
> > >
> > >> Thank you for leading this, Vladimir, Reynold, Herman.
> > >>
> > >> I'm wondering if this is really achievable goal for Apache Spark
> 4.0.0.
> > >>
> > >> If it's expected that we are unable to deliver it, shall we postpone
> this
> > >> vote until 4.1.0 planning?
> > >>
> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> > >>
> > >> -1 from my side.
> > >>
> > >> Thanks,
> > >> Dongjoon.
> > >>
> > >>
> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> > >> > +1
> > >> >
> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
> <rxin@databricks.com.invalid
> > >> >
> > >> > wrote:
> > >> >
> > >> > > +1
> > >> > >
> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
> vvdr.glb@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Hi all,
> > >> > >>
> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
> Catalyst
> > >> > >> project. This project will introduce a new analysis framework to
> the
> > >> > >> Catalyst, which will eventually replace the fixed-point one.
> > >> > >>
> > >> > >> Please refer to the SPIP jira:
> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> > >> > >>
> > >> > >> [ ] +1: Accept the proposal
> > >> > >> [ ] +0
> > >> > >> [ ] -1: I don’t think this is a good idea because …
> > >> > >>
> > >> > >> Thanks!
> > >> > >>
> > >> > >> Vladimir
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >>
> > >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for the swift clarification, Reynold and Xiao.

It seems that the Target Version was set mistakenly initially.

I removed the `Target Version` from the SPIP JIRA.

https://issues.apache.org/jira/browse/SPARK-49834

I'm switching my cast to +1 for this SPIP vote.

Thanks,
Dongjoon.

On 2024/09/30 22:55:41 Xiao Li wrote:
> +1 in support of the direction of the Single-pass Analyzer for Catalyst.
> 
> I think we should not have a target version for the new Catalyst SPARK-49834
> <https://issues.apache.org/jira/browse/SPARK-49834>. It should not be a
> blocker for Spark 4.0. When implementing the new analyzer, the code changes
> must not affect users of the existing analyzer to avoid any user-facing
> impacts.
> 
> Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:
> 
> > I don't actually "lead" this. But I don't think this needs to target a
> > specific Spark version given it should not have any user facing
> > consequences?
> >
> >
> > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org> wrote:
> >
> >> Thank you for leading this, Vladimir, Reynold, Herman.
> >>
> >> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
> >>
> >> If it's expected that we are unable to deliver it, shall we postpone this
> >> vote until 4.1.0 planning?
> >>
> >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> >>
> >> -1 from my side.
> >>
> >> Thanks,
> >> Dongjoon.
> >>
> >>
> >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> >> > +1
> >> >
> >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rxin@databricks.com.invalid
> >> >
> >> > wrote:
> >> >
> >> > > +1
> >> > >
> >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hi all,
> >> > >>
> >> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
> >> > >> project. This project will introduce a new analysis framework to the
> >> > >> Catalyst, which will eventually replace the fixed-point one.
> >> > >>
> >> > >> Please refer to the SPIP jira:
> >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> >> > >>
> >> > >> [ ] +1: Accept the proposal
> >> > >> [ ] +0
> >> > >> [ ] -1: I don’t think this is a good idea because …
> >> > >>
> >> > >> Thanks!
> >> > >>
> >> > >> Vladimir
> >> > >>
> >> > >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
> >>
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Xiao Li <ga...@gmail.com>.
+1 in support of the direction of the Single-pass Analyzer for Catalyst.

I think we should not have a target version for the new Catalyst SPARK-49834
<https://issues.apache.org/jira/browse/SPARK-49834>. It should not be a
blocker for Spark 4.0. When implementing the new analyzer, the code changes
must not affect users of the existing analyzer to avoid any user-facing
impacts.

Reynold Xin <rx...@databricks.com.invalid> 于2024年9月30日周一 15:39写道:

> I don't actually "lead" this. But I don't think this needs to target a
> specific Spark version given it should not have any user facing
> consequences?
>
>
> On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for leading this, Vladimir, Reynold, Herman.
>>
>> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
>>
>> If it's expected that we are unable to deliver it, shall we postpone this
>> vote until 4.1.0 planning?
>>
>> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>
>> -1 from my side.
>>
>> Thanks,
>> Dongjoon.
>>
>>
>> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>> > +1
>> >
>> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rxin@databricks.com.invalid
>> >
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi all,
>> > >>
>> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
>> > >> project. This project will introduce a new analysis framework to the
>> > >> Catalyst, which will eventually replace the fixed-point one.
>> > >>
>> > >> Please refer to the SPIP jira:
>> > >> https://issues.apache.org/jira/browse/SPARK-49834
>> > >>
>> > >> [ ] +1: Accept the proposal
>> > >> [ ] +0
>> > >> [ ] -1: I don’t think this is a good idea because …
>> > >>
>> > >> Thanks!
>> > >>
>> > >> Vladimir
>> > >>
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Mridul Muralidharan <mr...@gmail.com>.
+1

Regards,
Mridul

On Mon, Sep 30, 2024 at 5:39 PM Reynold Xin <rx...@databricks.com.invalid>
wrote:

> I don't actually "lead" this. But I don't think this needs to target a
> specific Spark version given it should not have any user facing
> consequences?
>
>
> On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for leading this, Vladimir, Reynold, Herman.
>>
>> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
>>
>> If it's expected that we are unable to deliver it, shall we postpone this
>> vote until 4.1.0 planning?
>>
>> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>
>> -1 from my side.
>>
>> Thanks,
>> Dongjoon.
>>
>>
>> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>> > +1
>> >
>> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rxin@databricks.com.invalid
>> >
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi all,
>> > >>
>> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
>> > >> project. This project will introduce a new analysis framework to the
>> > >> Catalyst, which will eventually replace the fixed-point one.
>> > >>
>> > >> Please refer to the SPIP jira:
>> > >> https://issues.apache.org/jira/browse/SPARK-49834
>> > >>
>> > >> [ ] +1: Accept the proposal
>> > >> [ ] +0
>> > >> [ ] -1: I don’t think this is a good idea because …
>> > >>
>> > >> Thanks!
>> > >>
>> > >> Vladimir
>> > >>
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Vladimir Golubev <vv...@gmail.com>.
Hi Mich! Thank you for this input.

Yes, this is exactly the approach I would propose too. Putting the new
analyzer under a flag and making the tests pass for both implementations is
crucial. We need to compare the logical (analyzed) plans and ensure that
they are identical.

On Thu, Oct 3, 2024 at 7:41 PM Mich Talebzadeh <mi...@gmail.com>
wrote:

>
> With a project manager hat on and having read the SPIP
> <https://issues.apache.org/jira/browse/SPARK-49834>
>
> This proposed single-pass Analyzer framework does potentially offer
> significant long-term benefits in terms of efficiency, maintenance, and
> stability, especially for large or complex queries. However, the rewrite
> involves substantial challenges, including the complexity of the
> transition, the resource cost, and the risk of breaking edge cases or
> existing workflows during the migration period. The key trade-off is
> between the upfront complexity and development time versus the potential
> long-term gains in performance, predictability, and ease of maintenance.
> Phasing the implementation could be an effective method to balance the
> risks and rewards. It allows the community to gradually transition to the
> new framework while mitigating potential disruptions. This is a
> proposal that we can consider
>
> Phase 1: Experimental Opt-In
>    - Introduce the single-pass Analyzer framework as an experimental
> feature.
>    - Allow users to opt-in through a configuration setting say (
> *`spark.sql.analyzer.singlePass.enabled=true*`), so the developers can
> start testing their workflows with it.
>    - Maintain the existing fixed-point Analyzer as the default to ensure
> stability for current users.
>    - Gradually build out coverage for common SQL and DataFrame operations.
>
> Phase 2: Expanded Operator Coverage
>    - Incrementally support more SQL operators, expressions, and DataFrame
> functionality as feedback and testing reveal areas of improvement.
>    - Ensure unit and integration tests run against both frameworks to
> maintain backward compatibility.
>    - Provide detailed documentation and migration guides so users are
> aware of the differences and can adjust their code if needed.
>
> Phase 3: Deprecation of the Fixed-Point Analyzer
>    - Once the single-pass Analyzer has full coverage and has been tested
> extensively in production environments, deprecate the old fixed-point
> framework.
>    - Offer a transition period where both frameworks are supported to give
> users time to adjust.
>
> Phase 4: Full Transition and Removal of Fixed-Point Framework
>    - After sufficient testing and user adoption, make the single-pass
> Analyzer the default and eventually remove the old framework in a future
> major release (say Spark 5.0).
>
> Benefits of Phasing:
> - Risk Mitigation: The phased approach allows gradual adoption, reducing
> the risk of breaking existing workloads or workflows. It ensures there is
> plenty of time for testing and feedback before a full transition.
> - Early User Feedback: Users can test the new framework in their
> environments early and provide feedback, allowing developers to address
> edge cases before it becomes the default.
> - Controlled Rollout: Phasing ensures that any unforeseen issues can be
> addressed incrementally without large disruptions to Spark deployments.
>
> This approach will hopefully ensure a smooth transition to the new
> framework while balancing the trade-offs of complexity, resource
> availability and long-term gains.
>
> HTH
>
> Mich Talebzadeh,
>
> Architect | Data Engineer | Data Science | Financial Crime
> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
> London <https://en.wikipedia.org/wiki/Imperial_College_London>
> London, United Kingdom
>
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner  <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>
>
> On Mon, 30 Sept 2024 at 23:38, Reynold Xin <rx...@databricks.com.invalid>
> wrote:
>
>> I don't actually "lead" this. But I don't think this needs to target a
>> specific Spark version given it should not have any user facing
>> consequences?
>>
>>
>> On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you for leading this, Vladimir, Reynold, Herman.
>>>
>>> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
>>>
>>> If it's expected that we are unable to deliver it, shall we postpone
>>> this vote until 4.1.0 planning?
>>>
>>> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>>
>>> -1 from my side.
>>>
>>> Thanks,
>>> Dongjoon.
>>>
>>>
>>> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>>> > +1
>>> >
>>> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
>>> <rx...@databricks.com.invalid>
>>> > wrote:
>>> >
>>> > > +1
>>> > >
>>> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vvdr.glb@gmail.com
>>> >
>>> > > wrote:
>>> > >
>>> > >> Hi all,
>>> > >>
>>> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
>>> > >> project. This project will introduce a new analysis framework to the
>>> > >> Catalyst, which will eventually replace the fixed-point one.
>>> > >>
>>> > >> Please refer to the SPIP jira:
>>> > >> https://issues.apache.org/jira/browse/SPARK-49834
>>> > >>
>>> > >> [ ] +1: Accept the proposal
>>> > >> [ ] +0
>>> > >> [ ] -1: I don’t think this is a good idea because …
>>> > >>
>>> > >> Thanks!
>>> > >>
>>> > >> Vladimir
>>> > >>
>>> > >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Mich Talebzadeh <mi...@gmail.com>.
With a project manager hat on and having read the SPIP
<https://issues.apache.org/jira/browse/SPARK-49834>

This proposed single-pass Analyzer framework does potentially offer
significant long-term benefits in terms of efficiency, maintenance, and
stability, especially for large or complex queries. However, the rewrite
involves substantial challenges, including the complexity of the
transition, the resource cost, and the risk of breaking edge cases or
existing workflows during the migration period. The key trade-off is
between the upfront complexity and development time versus the potential
long-term gains in performance, predictability, and ease of maintenance.
Phasing the implementation could be an effective method to balance the
risks and rewards. It allows the community to gradually transition to the
new framework while mitigating potential disruptions. This is a
proposal that we can consider

Phase 1: Experimental Opt-In
   - Introduce the single-pass Analyzer framework as an experimental
feature.
   - Allow users to opt-in through a configuration setting say (
*`spark.sql.analyzer.singlePass.enabled=true*`), so the developers can
start testing their workflows with it.
   - Maintain the existing fixed-point Analyzer as the default to ensure
stability for current users.
   - Gradually build out coverage for common SQL and DataFrame operations.

Phase 2: Expanded Operator Coverage
   - Incrementally support more SQL operators, expressions, and DataFrame
functionality as feedback and testing reveal areas of improvement.
   - Ensure unit and integration tests run against both frameworks to
maintain backward compatibility.
   - Provide detailed documentation and migration guides so users are aware
of the differences and can adjust their code if needed.

Phase 3: Deprecation of the Fixed-Point Analyzer
   - Once the single-pass Analyzer has full coverage and has been tested
extensively in production environments, deprecate the old fixed-point
framework.
   - Offer a transition period where both frameworks are supported to give
users time to adjust.

Phase 4: Full Transition and Removal of Fixed-Point Framework
   - After sufficient testing and user adoption, make the single-pass
Analyzer the default and eventually remove the old framework in a future
major release (say Spark 5.0).

Benefits of Phasing:
- Risk Mitigation: The phased approach allows gradual adoption, reducing
the risk of breaking existing workloads or workflows. It ensures there is
plenty of time for testing and feedback before a full transition.
- Early User Feedback: Users can test the new framework in their
environments early and provide feedback, allowing developers to address
edge cases before it becomes the default.
- Controlled Rollout: Phasing ensures that any unforeseen issues can be
addressed incrementally without large disruptions to Spark deployments.

This approach will hopefully ensure a smooth transition to the new
framework while balancing the trade-offs of complexity, resource
availability and long-term gains.

HTH

Mich Talebzadeh,

Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>


 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".


On Mon, 30 Sept 2024 at 23:38, Reynold Xin <rx...@databricks.com.invalid>
wrote:

> I don't actually "lead" this. But I don't think this needs to target a
> specific Spark version given it should not have any user facing
> consequences?
>
>
> On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you for leading this, Vladimir, Reynold, Herman.
>>
>> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
>>
>> If it's expected that we are unable to deliver it, shall we postpone this
>> vote until 4.1.0 planning?
>>
>> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>>
>> -1 from my side.
>>
>> Thanks,
>> Dongjoon.
>>
>>
>> On 2024/09/30 17:51:24 Herman van Hovell wrote:
>> > +1
>> >
>> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rxin@databricks.com.invalid
>> >
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi all,
>> > >>
>> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
>> > >> project. This project will introduce a new analysis framework to the
>> > >> Catalyst, which will eventually replace the fixed-point one.
>> > >>
>> > >> Please refer to the SPIP jira:
>> > >> https://issues.apache.org/jira/browse/SPARK-49834
>> > >>
>> > >> [ ] +1: Accept the proposal
>> > >> [ ] +0
>> > >> [ ] -1: I don’t think this is a good idea because …
>> > >>
>> > >> Thanks!
>> > >>
>> > >> Vladimir
>> > >>
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Reynold Xin <rx...@databricks.com.INVALID>.
I don't actually "lead" this. But I don't think this needs to target a
specific Spark version given it should not have any user facing
consequences?


On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <do...@apache.org> wrote:

> Thank you for leading this, Vladimir, Reynold, Herman.
>
> I'm wondering if this is really achievable goal for Apache Spark 4.0.0.
>
> If it's expected that we are unable to deliver it, shall we postpone this
> vote until 4.1.0 planning?
>
> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
>
> -1 from my side.
>
> Thanks,
> Dongjoon.
>
>
> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> > +1
> >
> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rxin@databricks.com.invalid
> >
> > wrote:
> >
> > > +1
> > >
> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
> > >> project. This project will introduce a new analysis framework to the
> > >> Catalyst, which will eventually replace the fixed-point one.
> > >>
> > >> Please refer to the SPIP jira:
> > >> https://issues.apache.org/jira/browse/SPARK-49834
> > >>
> > >> [ ] +1: Accept the proposal
> > >> [ ] +0
> > >> [ ] -1: I don’t think this is a good idea because …
> > >>
> > >> Thanks!
> > >>
> > >> Vladimir
> > >>
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for leading this, Vladimir, Reynold, Herman.

I'm wondering if this is really achievable goal for Apache Spark 4.0.0.

If it's expected that we are unable to deliver it, shall we postpone this vote until 4.1.0 planning?

Anyway, since SPARK-49834 has a target version 4.0.0 explicitly, 

-1 from my side.

Thanks,
Dongjoon.


On 2024/09/30 17:51:24 Herman van Hovell wrote:
> +1
> 
> On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rx...@databricks.com.invalid>
> wrote:
> 
> > +1
> >
> > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> I’d like to start a vote for a single-pass Analyzer for the Catalyst
> >> project. This project will introduce a new analysis framework to the
> >> Catalyst, which will eventually replace the fixed-point one.
> >>
> >> Please refer to the SPIP jira:
> >> https://issues.apache.org/jira/browse/SPARK-49834
> >>
> >> [ ] +1: Accept the proposal
> >> [ ] +0
> >> [ ] -1: I don’t think this is a good idea because …
> >>
> >> Thanks!
> >>
> >> Vladimir
> >>
> >
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Herman van Hovell <he...@databricks.com.INVALID>.
+1

On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin <rx...@databricks.com.invalid>
wrote:

> +1
>
> On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I’d like to start a vote for a single-pass Analyzer for the Catalyst
>> project. This project will introduce a new analysis framework to the
>> Catalyst, which will eventually replace the fixed-point one.
>>
>> Please refer to the SPIP jira:
>> https://issues.apache.org/jira/browse/SPARK-49834
>>
>> [ ] +1: Accept the proposal
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thanks!
>>
>> Vladimir
>>
>

Re: [VOTE] Single-pass Analyzer for Catalyst

Posted by Reynold Xin <rx...@databricks.com.INVALID>.
+1

On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <vv...@gmail.com> wrote:

> Hi all,
>
> I’d like to start a vote for a single-pass Analyzer for the Catalyst
> project. This project will introduce a new analysis framework to the
> Catalyst, which will eventually replace the fixed-point one.
>
> Please refer to the SPIP jira:
> https://issues.apache.org/jira/browse/SPARK-49834
>
> [ ] +1: Accept the proposal
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thanks!
>
> Vladimir
>