You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by John Zhuge <jz...@apache.org> on 2021/05/24 18:21:51 UTC
[VOTE] SPIP: Catalog API for view metadata
Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
(SPIP).
The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.
The full SPIP doc is here:
https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
update the PR for review.
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
Holden has graciously agreed to shepherd the SPIP. Thanks!
On Thu, Feb 10, 2022 at 9:19 AM John Zhuge <jz...@apache.org> wrote:
> The vote is now closed and the vote passes. Thank you to everyone who took
> the time to review and vote on this SPIP. I’m looking forward to adding
> this feature to the next Spark release. The tracking JIRA is
> https://issues.apache.org/jira/browse/SPARK-31357.
>
> The tally is:
>
> +1s:
>
> Walaa Eldin Moustafa
> Erik Krogen
> Holden Karau (binding)
> Ryan Blue
> Chao Sun
> L C Hsieh (binding)
> Huaxin Gao
> Yufei Gu
> Terry Kim
> Jacky Lee
> Wenchen Fan (binding)
>
> 0s:
>
> -1s:
>
> On Mon, Feb 7, 2022 at 10:04 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> +1 (binding)
>>
>> On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:
>>
>>> +1 (non-binding). Thanks John!
>>> It's great to see ViewCatalog moving on, it's a nice feature.
>>>
>>> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>>>
>>>> +1 (non-binding). Thanks John!
>>>>
>>>> Terry
>>>>
>>>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>> Best,
>>>>>
>>>>> Yufei
>>>>>
>>>>> `This is not a contribution`
>>>>>
>>>>>
>>>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>>>> >
>>>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>>>> >
>>>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>> >>
>>>>>>> >> +1 for the SPIP. I think it's well designed and it has worked
>>>>>>> quite well at Netflix for a long time.
>>>>>>> >>
>>>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi Spark community,
>>>>>>> >>>
>>>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>>>> (SPIP).
>>>>>>> >>>
>>>>>>> >>> The proposal is to add a ViewCatalog interface that can be used
>>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>>> >>>
>>>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>>>> >>>
>>>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> >>> [ ] +0
>>>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>>>> >>>
>>>>>>> >>> Thanks!
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Ryan Blue
>>>>>>> >> Tabular
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>
> --
> John Zhuge
>
--
John Zhuge
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
The vote is now closed and the vote passes. Thank you to everyone who took
the time to review and vote on this SPIP. I’m looking forward to adding
this feature to the next Spark release. The tracking JIRA is
https://issues.apache.org/jira/browse/SPARK-31357.
The tally is:
+1s:
Walaa Eldin Moustafa
Erik Krogen
Holden Karau (binding)
Ryan Blue
Chao Sun
L C Hsieh (binding)
Huaxin Gao
Yufei Gu
Terry Kim
Jacky Lee
Wenchen Fan (binding)
0s:
-1s:
On Mon, Feb 7, 2022 at 10:04 PM Wenchen Fan <cl...@gmail.com> wrote:
> +1 (binding)
>
> On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:
>
>> +1 (non-binding). Thanks John!
>> It's great to see ViewCatalog moving on, it's a nice feature.
>>
>> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>>
>>> +1 (non-binding). Thanks John!
>>>
>>> Terry
>>>
>>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>> Best,
>>>>
>>>> Yufei
>>>>
>>>> `This is not a contribution`
>>>>
>>>>
>>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>>> >
>>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>>> >
>>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>> >>
>>>>>> >> +1 for the SPIP. I think it's well designed and it has worked
>>>>>> quite well at Netflix for a long time.
>>>>>> >>
>>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi Spark community,
>>>>>> >>>
>>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>>> (SPIP).
>>>>>> >>>
>>>>>> >>> The proposal is to add a ViewCatalog interface that can be used
>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>> >>>
>>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>>> >>>
>>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> >>> [ ] +0
>>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>>> >>>
>>>>>> >>> Thanks!
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Ryan Blue
>>>>>> >> Tabular
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
--
John Zhuge
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Wenchen Fan <cl...@gmail.com>.
+1 (binding)
On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:
> +1 (non-binding). Thanks John!
> It's great to see ViewCatalog moving on, it's a nice feature.
>
> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>
>> +1 (non-binding). Thanks John!
>>
>> Terry
>>
>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>> Best,
>>>
>>> Yufei
>>>
>>> `This is not a contribution`
>>>
>>>
>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>> >
>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>> >
>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>> >>
>>>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>>>> well at Netflix for a long time.
>>>>> >>
>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi Spark community,
>>>>> >>>
>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>> (SPIP).
>>>>> >>>
>>>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>> >>>
>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>> >>>
>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>> >>> [ ] +0
>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>> >>>
>>>>> >>> Thanks!
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Ryan Blue
>>>>> >> Tabular
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Jacky Lee <qc...@gmail.com>.
+1 (non-binding). Thanks John!
It's great to see ViewCatalog moving on, it's a nice feature.
Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
> +1 (non-binding). Thanks John!
>
> Terry
>
> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> +1 (non-binding)
>> Best,
>>
>> Yufei
>>
>> `This is not a contribution`
>>
>>
>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>> >
>>>> > +1 (non-binding). Looking forward to this feature!
>>>> >
>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>> >>
>>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>>> well at Netflix for a long time.
>>>> >>
>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>>> >>>
>>>> >>> Hi Spark community,
>>>> >>>
>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>> (SPIP).
>>>> >>>
>>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>> >>>
>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>> >>>
>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>> >>> [ ] +0
>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>> >>>
>>>> >>> Thanks!
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Ryan Blue
>>>> >> Tabular
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Terry Kim <yu...@gmail.com>.
+1 (non-binding). Thanks John!
Terry
On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
> +1 (non-binding)
> Best,
>
> Yufei
>
> `This is not a contribution`
>
>
> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>> >
>>> > +1 (non-binding). Looking forward to this feature!
>>> >
>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>> >>
>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>> well at Netflix for a long time.
>>> >>
>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>> >>>
>>> >>> Hi Spark community,
>>> >>>
>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>> (SPIP).
>>> >>>
>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>> load, create, alter, and drop views in DataSourceV2.
>>> >>>
>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>> >>>
>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>> >>> [ ] +0
>>> >>> [ ] -1: I don’t think this is a good idea because …
>>> >>>
>>> >>> Thanks!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Ryan Blue
>>> >> Tabular
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Yufei Gu <fl...@gmail.com>.
+1 (non-binding)
Best,
Yufei
`This is not a contribution`
On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com> wrote:
> +1 (non-binding)
>
> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>
>> +1
>>
>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>> >
>> > +1 (non-binding). Looking forward to this feature!
>> >
>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>> >>
>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>> well at Netflix for a long time.
>> >>
>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>> >>>
>> >>> Hi Spark community,
>> >>>
>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>> (SPIP).
>> >>>
>> >>> The proposal is to add a ViewCatalog interface that can be used to
>> load, create, alter, and drop views in DataSourceV2.
>> >>>
>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>> >>>
>> >>> [ ] +1: Accept the proposal as an official SPIP
>> >>> [ ] +0
>> >>> [ ] -1: I don’t think this is a good idea because …
>> >>>
>> >>> Thanks!
>> >>
>> >>
>> >>
>> >> --
>> >> Ryan Blue
>> >> Tabular
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by huaxin gao <hu...@gmail.com>.
+1 (non-binding)
On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
> +1
>
> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
> >
> > +1 (non-binding). Looking forward to this feature!
> >
> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
> >>
> >> +1 for the SPIP. I think it's well designed and it has worked quite
> well at Netflix for a long time.
> >>
> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
> >>>
> >>> Hi Spark community,
> >>>
> >>> I’d like to restart the vote for the ViewCatalog design proposal
> (SPIP).
> >>>
> >>> The proposal is to add a ViewCatalog interface that can be used to
> load, create, alter, and drop views in DataSourceV2.
> >>>
> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
> >>>
> >>> [ ] +1: Accept the proposal as an official SPIP
> >>> [ ] +0
> >>> [ ] -1: I don’t think this is a good idea because …
> >>>
> >>> Thanks!
> >>
> >>
> >>
> >> --
> >> Ryan Blue
> >> Tabular
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by "L. C. Hsieh" <vi...@gmail.com>.
+1
On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>
> +1 (non-binding). Looking forward to this feature!
>
> On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>> +1 for the SPIP. I think it's well designed and it has worked quite well at Netflix for a long time.
>>
>> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>> Hi Spark community,
>>>
>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load, create, alter, and drop views in DataSourceV2.
>>>
>>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ryan Blue
>> Tabular
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Chao Sun <su...@apache.org>.
+1 (non-binding). Looking forward to this feature!
On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
> +1 for the SPIP. I think it's well designed and it has worked quite well
> at Netflix for a long time.
>
> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>
>> Hi Spark community,
>>
>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thanks!
>>
>
>
> --
> Ryan Blue
> Tabular
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Ryan Blue <bl...@tabular.io>.
+1 for the SPIP. I think it's well designed and it has worked quite well at
Netflix for a long time.
On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
> Hi Spark community,
>
> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> Please vote on the SPIP until Feb. 9th (Wednesday).
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thanks!
>
--
Ryan Blue
Tabular
[VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
Hi Spark community,
I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.
Please vote on the SPIP until Feb. 9th (Wednesday).
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …
Thanks!
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
Sure Xiao.
Happy Lunar New Year!
On Thu, Feb 3, 2022 at 1:57 PM Xiao Li <ga...@gmail.com> wrote:
> Can we extend the voting window to next Wednesday? This week is a holiday
> week for the lunar new year. AFAIK, many members in Asia are taking the
> whole week off. They might not regularly check the emails.
>
> Also how about starting a separate email thread starting with [VOTE] ?
>
> Happy Lunar New Year!!!
>
> Xiao
>
> Holden Karau <ho...@pigscanfly.ca> 于2022年2月3日周四 12:28写道：
>
>> +1 (binding)
>>
>> On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:
>>
>>> +1 (non-binding)
>>>
>>> Really looking forward to having this natively supported by Spark, so
>>> that we can get rid of our own hacks to tie in a custom view catalog
>>> implementation. I appreciate the care John has put into various parts of
>>> the design and believe this will provide a robust and flexible solution to
>>> this problem faced by various large-scale Spark users.
>>>
>>> Thanks John!
>>>
>>> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Hi Spark community,
>>>>>
>>>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>>>> ).
>>>>>
>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>
>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>> I’ll update the PR <https://github.com/apache/spark/pull/28147> for
>>>>> review.
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>>>> idea. A view catalog will enable us to integrate Coral
>>>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>>>> only do it by maintaining our special version of the
>>>>>> HiveExternalCatalog. Considering that views can be expanded
>>>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>>>> view API can make performance better if performance is the concern.
>>>>>> Further, a catalog can still be both a table and view provider if it
>>>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>>>> ability of providing both. Looking forward to more discussions on this and
>>>>>> making views a powerful tool in Spark.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>>
>>>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>>>
>>>>>>> Looks like we are running in circles. Should we have an online
>>>>>>> meeting to get this sorted out?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>>>> catalog to the v2 API sooner or later.
>>>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>>>> table/view resolution framework.
>>>>>>>> 3. It's better to avoid name conflicts between table and views at
>>>>>>>> the API level, instead of relying on the catalog implementation.
>>>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>>>
>>>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I don't think that it makes sense to discuss a different approach
>>>>>>>>> in the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>>>> purpose of an SPIP.
>>>>>>>>>
>>>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>>>> proposal (SPIP).
>>>>>>>>>>
>>>>>>>>>> The proposal is to add a ViewCatalog interface that can be used
>>>>>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>>>>>>
>>>>>>>>>> The full SPIP doc is here:
>>>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>>>
>>>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is
>>>>>>>>>> approved, I’ll update the PR for review.
>>>>>>>>>>
>>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>>>> [ ] +0
>>>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> John Zhuge
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
--
John Zhuge
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Xiao Li <ga...@gmail.com>.
Can we extend the voting window to next Wednesday? This week is a holiday
week for the lunar new year. AFAIK, many members in Asia are taking the
whole week off. They might not regularly check the emails.
Also how about starting a separate email thread starting with [VOTE] ?
Happy Lunar New Year!!!
Xiao
Holden Karau <ho...@pigscanfly.ca> 于2022年2月3日周四 12:28写道：
> +1 (binding)
>
> On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:
>
>> +1 (non-binding)
>>
>> Really looking forward to having this natively supported by Spark, so
>> that we can get rid of our own hacks to tie in a custom view catalog
>> implementation. I appreciate the care John has put into various parts of
>> the design and believe this will provide a robust and flexible solution to
>> this problem faced by various large-scale Spark users.
>>
>> Thanks John!
>>
>> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Hi Spark community,
>>>>
>>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>>> ).
>>>>
>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>>
>>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don’t think this is a good idea because …
>>>>
>>>> Thanks!
>>>>
>>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>>> idea. A view catalog will enable us to integrate Coral
>>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>>> only do it by maintaining our special version of the
>>>>> HiveExternalCatalog. Considering that views can be expanded
>>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>>> view API can make performance better if performance is the concern.
>>>>> Further, a catalog can still be both a table and view provider if it
>>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>>> ability of providing both. Looking forward to more discussions on this and
>>>>> making views a powerful tool in Spark.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> Looks like we are running in circles. Should we have an online
>>>>>> meeting to get this sorted out?
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>>> catalog to the v2 API sooner or later.
>>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>>> table/view resolution framework.
>>>>>>> 3. It's better to avoid name conflicts between table and views at
>>>>>>> the API level, instead of relying on the catalog implementation.
>>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>>
>>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't think that it makes sense to discuss a different approach
>>>>>>>> in the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>>> purpose of an SPIP.
>>>>>>>>
>>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>>> proposal (SPIP).
>>>>>>>>>
>>>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>>>
>>>>>>>>> The full SPIP doc is here:
>>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>>
>>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>>>> I’ll update the PR for review.
>>>>>>>>>
>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>>> [ ] +0
>>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> John Zhuge
>>>>>>
>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Holden Karau <ho...@pigscanfly.ca>.
+1 (binding)
On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:
> +1 (non-binding)
>
> Really looking forward to having this natively supported by Spark, so that
> we can get rid of our own hacks to tie in a custom view catalog
> implementation. I appreciate the care John has put into various parts of
> the design and believe this will provide a robust and flexible solution to
> this problem faced by various large-scale Spark users.
>
> Thanks John!
>
> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> +1
>>
>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Hi Spark community,
>>>
>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>> ).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load,
>>> create, alter, and drop views in DataSourceV2.
>>>
>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>> Thanks!
>>>
>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>> idea. A view catalog will enable us to integrate Coral
>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>> only do it by maintaining our special version of the
>>>> HiveExternalCatalog. Considering that views can be expanded
>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>> view API can make performance better if performance is the concern.
>>>> Further, a catalog can still be both a table and view provider if it
>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>> ability of providing both. Looking forward to more discussions on this and
>>>> making views a powerful tool in Spark.
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>>
>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Looks like we are running in circles. Should we have an online meeting
>>>>> to get this sorted out?
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>> catalog to the v2 API sooner or later.
>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>> table/view resolution framework.
>>>>>> 3. It's better to avoid name conflicts between table and views at the
>>>>>> API level, instead of relying on the catalog implementation.
>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>
>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't think that it makes sense to discuss a different approach in
>>>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>> purpose of an SPIP.
>>>>>>>
>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>> proposal (SPIP).
>>>>>>>>
>>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>>
>>>>>>>> The full SPIP doc is here:
>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>
>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>>> I’ll update the PR for review.
>>>>>>>>
>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>> [ ] +0
>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>>
>>>
>>> --
>>> John Zhuge
>>>
>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Erik Krogen <xk...@apache.org>.
+1 (non-binding)
Really looking forward to having this natively supported by Spark, so that
we can get rid of our own hacks to tie in a custom view catalog
implementation. I appreciate the care John has put into various parts of
the design and believe this will provide a robust and flexible solution to
this problem faced by various large-scale Spark users.
Thanks John!
On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> +1
>
> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>
>> Hi Spark community,
>>
>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>> ).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thanks!
>>
>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Considering the API aspect, the ViewCatalog API sounds like a good idea.
>>> A view catalog will enable us to integrate Coral
>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>> translation and management layer) very cleanly to Spark. Currently we can
>>> only do it by maintaining our special version of the HiveExternalCatalog.
>>> Considering that views can be expanded syntactically without necessarily
>>> invoking the analyzer, using a dedicated view API can make performance
>>> better if performance is the concern. Further, a catalog can still be both
>>> a table and view provider if it chooses to based on this design, so I do
>>> not think we necessarily lose the ability of providing both. Looking
>>> forward to more discussions on this and making views a powerful tool in
>>> Spark.
>>>
>>> Thanks,
>>> Walaa.
>>>
>>>
>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Looks like we are running in circles. Should we have an online meeting
>>>> to get this sorted out?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>> wrote:
>>>>
>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>>>> to the v2 API sooner or later.
>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>> table/view resolution framework.
>>>>> 3. It's better to avoid name conflicts between table and views at the
>>>>> API level, instead of relying on the catalog implementation.
>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>
>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> I don't think that it makes sense to discuss a different approach in
>>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>>> purpose of an SPIP.
>>>>>>
>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>> proposal (SPIP).
>>>>>>>
>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>
>>>>>>> The full SPIP doc is here:
>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>
>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>> I’ll update the PR for review.
>>>>>>>
>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> [ ] +0
>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>
>>
>> --
>> John Zhuge
>>
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
+1
On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
> Hi Spark community,
>
> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
> ).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
> update the PR <https://github.com/apache/spark/pull/28147> for review.
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thanks!
>
> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <wa...@gmail.com>
> wrote:
>
>> Considering the API aspect, the ViewCatalog API sounds like a good idea.
>> A view catalog will enable us to integrate Coral
>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>> translation and management layer) very cleanly to Spark. Currently we can
>> only do it by maintaining our special version of the HiveExternalCatalog.
>> Considering that views can be expanded syntactically without necessarily
>> invoking the analyzer, using a dedicated view API can make performance
>> better if performance is the concern. Further, a catalog can still be both
>> a table and view provider if it chooses to based on this design, so I do
>> not think we necessarily lose the ability of providing both. Looking
>> forward to more discussions on this and making views a powerful tool in
>> Spark.
>>
>> Thanks,
>> Walaa.
>>
>>
>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Looks like we are running in circles. Should we have an online meeting
>>> to get this sorted out?
>>>
>>> Thanks,
>>> John
>>>
>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>> wrote:
>>>
>>>> OK, then I'd vote for TableViewCatalog, because
>>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>>> to the v2 API sooner or later.
>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>> table/view resolution framework.
>>>> 3. It's better to avoid name conflicts between table and views at the
>>>> API level, instead of relying on the catalog implementation.
>>>> 4. Caching invalidation is always a tricky problem.
>>>>
>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> I don't think that it makes sense to discuss a different approach in
>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>> purpose of an SPIP.
>>>>>
>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>> proposal (SPIP).
>>>>>>
>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>
>>>>>> The full SPIP doc is here:
>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>
>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>> I’ll update the PR for review.
>>>>>>
>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> [ ] +0
>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>
>>> --
>>> John Zhuge
>>>
>>
>
> --
> John Zhuge
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
Hi Spark community,
I’d like to restart the vote for the ViewCatalog design proposal (SPIP
<https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
).
The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.
Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
update the PR <https://github.com/apache/spark/pull/28147> for review.
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …
Thanks!
On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> Considering the API aspect, the ViewCatalog API sounds like a good idea. A
> view catalog will enable us to integrate Coral
> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
> translation and management layer) very cleanly to Spark. Currently we can
> only do it by maintaining our special version of the HiveExternalCatalog.
> Considering that views can be expanded syntactically without necessarily
> invoking the analyzer, using a dedicated view API can make performance
> better if performance is the concern. Further, a catalog can still be both
> a table and view provider if it chooses to based on this design, so I do
> not think we necessarily lose the ability of providing both. Looking
> forward to more discussions on this and making views a powerful tool in
> Spark.
>
> Thanks,
> Walaa.
>
>
> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>
>> Looks like we are running in circles. Should we have an online meeting to
>> get this sorted out?
>>
>> Thanks,
>> John
>>
>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> OK, then I'd vote for TableViewCatalog, because
>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>> to the v2 API sooner or later.
>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>> table/view resolution framework.
>>> 3. It's better to avoid name conflicts between table and views at the
>>> API level, instead of relying on the catalog implementation.
>>> 4. Caching invalidation is always a tricky problem.
>>>
>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> I don't think that it makes sense to discuss a different approach in
>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>> purpose of an SPIP.
>>>>
>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>> proposal (SPIP).
>>>>>
>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>
>>>>> The full SPIP doc is here:
>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>
>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>> I’ll update the PR for review.
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> John Zhuge
>>
>
--
John Zhuge
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Considering the API aspect, the ViewCatalog API sounds like a good idea. A
view catalog will enable us to integrate Coral
<https://engineering.linkedin.com/blog/2020/coral> (our view SQL
translation and management layer) very cleanly to Spark. Currently we can
only do it by maintaining our special version of the HiveExternalCatalog.
Considering that views can be expanded syntactically without necessarily
invoking the analyzer, using a dedicated view API can make performance
better if performance is the concern. Further, a catalog can still be both
a table and view provider if it chooses to based on this design, so I do
not think we necessarily lose the ability of providing both. Looking
forward to more discussions on this and making views a powerful tool in
Spark.
Thanks,
Walaa.
On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
> Looks like we are running in circles. Should we have an online meeting to
> get this sorted out?
>
> Thanks,
> John
>
> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> OK, then I'd vote for TableViewCatalog, because
>> 1. This is how Hive catalog works, and we need to migrate Hive catalog to
>> the v2 API sooner or later.
>> 2. Because of 1, TableViewCatalog is easy to support in the current
>> table/view resolution framework.
>> 3. It's better to avoid name conflicts between table and views at the API
>> level, instead of relying on the catalog implementation.
>> 4. Caching invalidation is always a tricky problem.
>>
>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> I don't think that it makes sense to discuss a different approach in the
>>> PR rather than in the vote. Let's discuss this now since that's the purpose
>>> of an SPIP.
>>>
>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>> proposal (SPIP).
>>>>
>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>>
>>>> The full SPIP doc is here:
>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>
>>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>>> update the PR for review.
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don’t think this is a good idea because …
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> John Zhuge
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by John Zhuge <jz...@apache.org>.
Looks like we are running in circles. Should we have an online meeting to
get this sorted out?
Thanks,
John
On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:
> OK, then I'd vote for TableViewCatalog, because
> 1. This is how Hive catalog works, and we need to migrate Hive catalog to
> the v2 API sooner or later.
> 2. Because of 1, TableViewCatalog is easy to support in the current
> table/view resolution framework.
> 3. It's better to avoid name conflicts between table and views at the API
> level, instead of relying on the catalog implementation.
> 4. Caching invalidation is always a tricky problem.
>
> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> I don't think that it makes sense to discuss a different approach in the
>> PR rather than in the vote. Let's discuss this now since that's the purpose
>> of an SPIP.
>>
>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>> proposal (SPIP).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load,
>>> create, alter, and drop views in DataSourceV2.
>>>
>>> The full SPIP doc is here:
>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>
>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>> update the PR for review.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
--
John Zhuge
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Wenchen Fan <cl...@gmail.com>.
OK, then I'd vote for TableViewCatalog, because
1. This is how Hive catalog works, and we need to migrate Hive catalog to
the v2 API sooner or later.
2. Because of 1, TableViewCatalog is easy to support in the current
table/view resolution framework.
3. It's better to avoid name conflicts between table and views at the API
level, instead of relying on the catalog implementation.
4. Caching invalidation is always a tricky problem.
On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> I don't think that it makes sense to discuss a different approach in the
> PR rather than in the vote. Let's discuss this now since that's the purpose
> of an SPIP.
>
> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>
>> Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
>> (SPIP).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> The full SPIP doc is here:
>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>
>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>> update the PR for review.
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] SPIP: Catalog API for view metadata
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
I don't think that it makes sense to discuss a different approach in the PR
rather than in the vote. Let's discuss this now since that's the purpose of
an SPIP.
On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
> Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
> (SPIP).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> The full SPIP doc is here:
> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>
> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
> update the PR for review.
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
--
Ryan Blue
Software Engineer
Netflix