You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Wenchen Fan <cl...@gmail.com> on 2020/11/10 04:18:43 UTC

Re: SPIP: Catalog API for view metadata

Moving back the discussion to this thread. The current argument is how to
avoid extra RPC calls for catalogs supporting both table and view. There
are several options:
1. ignore it as extra PRC calls are cheap compared to the query execution
2. have a per session cache for loaded table/view
3. have a per query cache for loaded table/view
4. add a new trait TableViewCatalog

I think it's important to avoid perf regression with new APIs. RPC calls
can be significant for short queries. We may also double the RPC
traffic which is bad for the metastore service. Normally I would not
recommend caching as cache invalidation is a hard problem. Personally I
prefer option 4 as it only affects catalogs that support both table and
view, and it fits the hive catalog very well.

On Fri, Sep 4, 2020 at 4:21 PM John Zhuge <jz...@apache.org> wrote:

> SPIP
> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
> has been updated. Please review.
>
> On Thu, Sep 3, 2020 at 9:22 AM John Zhuge <jz...@apache.org> wrote:
>
>> Wenchen, sorry for the delay, I will post an update shortly.
>>
>> On Thu, Sep 3, 2020 at 2:00 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Any updates here? I agree that a new View API is better, but we need a
>>> solution to avoid performance regression. We need to elaborate on the cache
>>> idea.
>>>
>>> On Thu, Aug 20, 2020 at 7:43 AM Ryan Blue <rb...@netflix.com> wrote:
>>>
>>>> I think it is a good idea to keep tables and views separate.
>>>>
>>>> The main two arguments I’ve heard for combining lookup into a single
>>>> function are the ones brought up in this thread. First, an identifier in a
>>>> catalog must be either a view or a table and should not collide. Second, a
>>>> single lookup is more likely to require a single RPC. I think the RPC
>>>> concern is well addressed by caching, which we already do in the Spark
>>>> catalog, so I’ll primarily focus on the first.
>>>>
>>>> Table/view name collision is unlikely to be a problem. Metastores that
>>>> support both today store them in a single namespace, so this is not a
>>>> concern for even a naive implementation that talks to the Hive MetaStore. I
>>>> know that a new metastore catalog could choose to implement both
>>>> ViewCatalog and TableCatalog and store the two sets separately, but that
>>>> would be a very strange choice: if the metastore itself has different
>>>> namespaces for tables and views, then it makes much more sense to expose
>>>> them through separate catalogs because Spark will always prefer one over
>>>> the other.
>>>>
>>>> In a similar line of reasoning, catalogs that expose both views and
>>>> tables are much more rare than catalogs that only expose one. For example,
>>>> v2 catalogs for JDBC and Cassandra expose data through the Table interface
>>>> and implementing ViewCatalog would make little sense. Exposing new data
>>>> sources to Spark requires TableCatalog, not ViewCatalog. View catalogs are
>>>> likely to be the same. Say I have a way to convert Pig statements or some
>>>> other representation into a SQL view. It would make little sense to combine
>>>> that with some other TableCatalog.
>>>>
>>>> I also don’t think there is benefit from an API perspective to justify
>>>> combining the Table and View interfaces. The two share only schema and
>>>> properties, and are handled very differently internally — a View’s SQL
>>>> query is parsed and substituted into the plan, while a Table is wrapped in
>>>> a relation that eventually becomes a Scan node using SupportsRead. A view’s
>>>> SQL also needs additional context to be resolved correctly: the current
>>>> catalog and namespace from the time the view was created.
>>>>
>>>> Query planning is distinct between tables and views, so Spark doesn’t
>>>> benefit from combining them. I think it has actually caused problems that
>>>> both were resolved by the same method in v1: the resolution rule grew
>>>> extremely complicated trying to look up a reference just once because it
>>>> had to parse a view plan and resolve relations within it using the view’s
>>>> context (current database). In contrast, John’s new view substitution rules
>>>> are cleaner and can stay within the substitution batch.
>>>>
>>>> People implementing views would also not benefit from combining the two
>>>> interfaces:
>>>>
>>>>    - There is little overlap between View and Table, only schema and
>>>>    properties
>>>>    - Most catalogs won’t implement both interfaces, so returning a
>>>>    ViewOrTable is more difficult for implementations
>>>>    - TableCatalog assumes that ViewCatalog will be added separately
>>>>    like John proposes, so we would have to break or replace that API
>>>>
>>>> I understand the initial appeal of combining TableCatalog and
>>>> ViewCatalog since it is done that way in the existing interfaces. But I
>>>> think that Hive chose to do that mostly on the fact that the two were
>>>> already stored together, and not because it made sense for users of the
>>>> API, or any other implementer of the API.
>>>>
>>>> rb
>>>>
>>>> On Tue, Aug 18, 2020 at 9:46 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> > AFAIK view schema is only used by DESCRIBE.
>>>>>>
>>>>>> Correction: Spark adds a new Project at the top of the parsed plan
>>>>>> from view, based on the stored schema, to make sure the view schema doesn't
>>>>>> change.
>>>>>>
>>>>>
>>>>> Thanks Wenchen! I thought I forgot something :) Yes it is the
>>>>> validation done in *checkAnalysis*:
>>>>>
>>>>>           // If the view output doesn't have the same number of
>>>>> columns neither with the child
>>>>>           // output, nor with the query column names, throw an
>>>>> AnalysisException.
>>>>>           // If the view's child output can't up cast to the view
>>>>> output,
>>>>>           // throw an AnalysisException, too.
>>>>>
>>>>> The view output comes from the schema:
>>>>>
>>>>>       val child = View(
>>>>>         desc = metadata,
>>>>>         output = metadata.schema.toAttributes,
>>>>>         child = parser.parsePlan(viewText))
>>>>>
>>>>> So it is a validation (here) or cache (in DESCRIBE) nice to have but
>>>>> not "required" or "should be frozen". Thanks Ryan and Burak for pointing
>>>>> that out in SPIP. I will add a new paragraph accordingly.
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> John Zhuge
>>
>
>
> --
> John Zhuge
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Holden has graciously agreed to shepherd the SPIP. Thanks!

On Thu, Feb 10, 2022 at 9:19 AM John Zhuge <jz...@apache.org> wrote:

> The vote is now closed and the vote passes. Thank you to everyone who took
> the time to review and vote on this SPIP. I’m looking forward to adding
> this feature to the next Spark release. The tracking JIRA is
> https://issues.apache.org/jira/browse/SPARK-31357.
>
> The tally is:
>
> +1s:
>
> Walaa Eldin Moustafa
> Erik Krogen
> Holden Karau (binding)
> Ryan Blue
> Chao Sun
> L C Hsieh (binding)
> Huaxin Gao
> Yufei Gu
> Terry Kim
> Jacky Lee
> Wenchen Fan (binding)
>
> 0s:
>
> -1s:
>
> On Mon, Feb 7, 2022 at 10:04 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> +1 (binding)
>>
>> On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:
>>
>>> +1 (non-binding). Thanks John!
>>> It's great to see ViewCatalog moving on, it's a nice feature.
>>>
>>> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>>>
>>>> +1 (non-binding). Thanks John!
>>>>
>>>> Terry
>>>>
>>>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>> Best,
>>>>>
>>>>> Yufei
>>>>>
>>>>> `This is not a contribution`
>>>>>
>>>>>
>>>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>>>> >
>>>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>>>> >
>>>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>> >>
>>>>>>> >> +1 for the SPIP. I think it's well designed and it has worked
>>>>>>> quite well at Netflix for a long time.
>>>>>>> >>
>>>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi Spark community,
>>>>>>> >>>
>>>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>>>> (SPIP).
>>>>>>> >>>
>>>>>>> >>> The proposal is to add a ViewCatalog interface that can be used
>>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>>> >>>
>>>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>>>> >>>
>>>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> >>> [ ] +0
>>>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>>>> >>>
>>>>>>> >>> Thanks!
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Ryan Blue
>>>>>>> >> Tabular
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>
> --
> John Zhuge
>


-- 
John Zhuge

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

The vote is now closed and the vote passes. Thank you to everyone who took
the time to review and vote on this SPIP. I’m looking forward to adding
this feature to the next Spark release. The tracking JIRA is
https://issues.apache.org/jira/browse/SPARK-31357.

The tally is:

+1s:

Walaa Eldin Moustafa
Erik Krogen
Holden Karau (binding)
Ryan Blue
Chao Sun
L C Hsieh (binding)
Huaxin Gao
Yufei Gu
Terry Kim
Jacky Lee
Wenchen Fan (binding)

0s:

-1s:

On Mon, Feb 7, 2022 at 10:04 PM Wenchen Fan <cl...@gmail.com> wrote:

> +1 (binding)
>
> On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:
>
>> +1 (non-binding). Thanks John!
>> It's great to see ViewCatalog moving on, it's a nice feature.
>>
>> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>>
>>> +1 (non-binding). Thanks John!
>>>
>>> Terry
>>>
>>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>> Best,
>>>>
>>>> Yufei
>>>>
>>>> `This is not a contribution`
>>>>
>>>>
>>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>>> >
>>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>>> >
>>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>> >>
>>>>>> >> +1 for the SPIP. I think it's well designed and it has worked
>>>>>> quite well at Netflix for a long time.
>>>>>> >>
>>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi Spark community,
>>>>>> >>>
>>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>>> (SPIP).
>>>>>> >>>
>>>>>> >>> The proposal is to add a ViewCatalog interface that can be used
>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>> >>>
>>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>>> >>>
>>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> >>> [ ] +0
>>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>>> >>>
>>>>>> >>> Thanks!
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Ryan Blue
>>>>>> >> Tabular
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>

-- 
John Zhuge

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Wenchen Fan <cl...@gmail.com>.

+1 (binding)

On Sun, Feb 6, 2022 at 10:27 AM Jacky Lee <qc...@gmail.com> wrote:

> +1 (non-binding). Thanks John!
> It's great to see ViewCatalog moving on, it's a nice feature.
>
> Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：
>
>> +1 (non-binding). Thanks John!
>>
>> Terry
>>
>> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>> Best,
>>>
>>> Yufei
>>>
>>> `This is not a contribution`
>>>
>>>
>>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>>> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>>> >
>>>>> > +1 (non-binding). Looking forward to this feature!
>>>>> >
>>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>> >>
>>>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>>>> well at Netflix for a long time.
>>>>> >>
>>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi Spark community,
>>>>> >>>
>>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>>> (SPIP).
>>>>> >>>
>>>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>> >>>
>>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>>> >>>
>>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>>> >>> [ ] +0
>>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>>> >>>
>>>>> >>> Thanks!
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Ryan Blue
>>>>> >> Tabular
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Jacky Lee <qc...@gmail.com>.

+1 (non-binding). Thanks John!
It's great to see ViewCatalog moving on, it's a nice feature.

Terry Kim <yu...@gmail.com> 于2022年2月5日周六 11:57写道：

> +1 (non-binding). Thanks John!
>
> Terry
>
> On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> +1 (non-binding)
>> Best,
>>
>> Yufei
>>
>> `This is not a contribution`
>>
>>
>> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>>> >
>>>> > +1 (non-binding). Looking forward to this feature!
>>>> >
>>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>>> >>
>>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>>> well at Netflix for a long time.
>>>> >>
>>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>>> >>>
>>>> >>> Hi Spark community,
>>>> >>>
>>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>>> (SPIP).
>>>> >>>
>>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>> >>>
>>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>> >>>
>>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>>> >>> [ ] +0
>>>> >>> [ ] -1: I don’t think this is a good idea because …
>>>> >>>
>>>> >>> Thanks!
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Ryan Blue
>>>> >> Tabular
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Terry Kim <yu...@gmail.com>.

+1 (non-binding). Thanks John!

Terry

On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu <fl...@gmail.com> wrote:

> +1 (non-binding)
> Best,
>
> Yufei
>
> `This is not a contribution`
>
>
> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>>> >
>>> > +1 (non-binding). Looking forward to this feature!
>>> >
>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>> >>
>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>> well at Netflix for a long time.
>>> >>
>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>> >>>
>>> >>> Hi Spark community,
>>> >>>
>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>> (SPIP).
>>> >>>
>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>> load, create, alter, and drop views in DataSourceV2.
>>> >>>
>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>> >>>
>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>> >>> [ ] +0
>>> >>> [ ] -1: I don’t think this is a good idea because …
>>> >>>
>>> >>> Thanks!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Ryan Blue
>>> >> Tabular
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Yufei Gu <fl...@gmail.com>.

+1 (non-binding)
Best,

Yufei

`This is not a contribution`


On Fri, Feb 4, 2022 at 11:54 AM huaxin gao <hu...@gmail.com> wrote:

> +1 (non-binding)
>
> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:
>
>> +1
>>
>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>> >
>> > +1 (non-binding). Looking forward to this feature!
>> >
>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>> >>
>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>> well at Netflix for a long time.
>> >>
>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>> >>>
>> >>> Hi Spark community,
>> >>>
>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>> (SPIP).
>> >>>
>> >>> The proposal is to add a ViewCatalog interface that can be used to
>> load, create, alter, and drop views in DataSourceV2.
>> >>>
>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>> >>>
>> >>> [ ] +1: Accept the proposal as an official SPIP
>> >>> [ ] +0
>> >>> [ ] -1: I don’t think this is a good idea because …
>> >>>
>> >>> Thanks!
>> >>
>> >>
>> >>
>> >> --
>> >> Ryan Blue
>> >> Tabular
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by huaxin gao <hu...@gmail.com>.

+1 (non-binding)

On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh <vi...@gmail.com> wrote:

> +1
>
> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
> >
> > +1 (non-binding). Looking forward to this feature!
> >
> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
> >>
> >> +1 for the SPIP. I think it's well designed and it has worked quite
> well at Netflix for a long time.
> >>
> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
> >>>
> >>> Hi Spark community,
> >>>
> >>> I’d like to restart the vote for the ViewCatalog design proposal
> (SPIP).
> >>>
> >>> The proposal is to add a ViewCatalog interface that can be used to
> load, create, alter, and drop views in DataSourceV2.
> >>>
> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
> >>>
> >>> [ ] +1: Accept the proposal as an official SPIP
> >>> [ ] +0
> >>> [ ] -1: I don’t think this is a good idea because …
> >>>
> >>> Thanks!
> >>
> >>
> >>
> >> --
> >> Ryan Blue
> >> Tabular
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by "L. C. Hsieh" <vi...@gmail.com>.

+1

On Thu, Feb 3, 2022 at 7:25 PM Chao Sun <su...@apache.org> wrote:
>
> +1 (non-binding). Looking forward to this feature!
>
> On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>> +1 for the SPIP. I think it's well designed and it has worked quite well at Netflix for a long time.
>>
>> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>> Hi Spark community,
>>>
>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load, create, alter, and drop views in DataSourceV2.
>>>
>>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ryan Blue
>> Tabular

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Chao Sun <su...@apache.org>.

+1 (non-binding). Looking forward to this feature!

On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue <bl...@tabular.io> wrote:

> +1 for the SPIP. I think it's well designed and it has worked quite well
> at Netflix for a long time.
>
> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:
>
>> Hi Spark community,
>>
>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thanks!
>>
>
>
> --
> Ryan Blue
> Tabular
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Ryan Blue <bl...@tabular.io>.

+1 for the SPIP. I think it's well designed and it has worked quite well at
Netflix for a long time.

On Thu, Feb 3, 2022 at 2:04 PM John Zhuge <jz...@apache.org> wrote:

> Hi Spark community,
>
> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> Please vote on the SPIP until Feb. 9th (Wednesday).
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thanks!
>


-- 
Ryan Blue
Tabular

[VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Hi Spark community,

I’d like to restart the vote for the ViewCatalog design proposal (SPIP).

The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.

Please vote on the SPIP until Feb. 9th (Wednesday).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

Thanks!

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Sure Xiao.

Happy Lunar New Year!

On Thu, Feb 3, 2022 at 1:57 PM Xiao Li <ga...@gmail.com> wrote:

> Can we extend the voting window to next Wednesday? This week is a holiday
> week for the lunar new year. AFAIK, many members in Asia are taking the
> whole week off. They might not regularly check the emails.
>
> Also how about starting a separate email thread starting with [VOTE] ?
>
> Happy Lunar New Year!!!
>
> Xiao
>
> Holden Karau <ho...@pigscanfly.ca> 于2022年2月3日周四 12:28写道：
>
>> +1 (binding)
>>
>> On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:
>>
>>> +1 (non-binding)
>>>
>>> Really looking forward to having this natively supported by Spark, so
>>> that we can get rid of our own hacks to tie in a custom view catalog
>>> implementation. I appreciate the care John has put into various parts of
>>> the design and believe this will provide a robust and flexible solution to
>>> this problem faced by various large-scale Spark users.
>>>
>>> Thanks John!
>>>
>>> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Hi Spark community,
>>>>>
>>>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>>>> ).
>>>>>
>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>
>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>> I’ll update the PR <https://github.com/apache/spark/pull/28147> for
>>>>> review.
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>>>> idea. A view catalog will enable us to integrate Coral
>>>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>>>> only do it by maintaining our special version of the
>>>>>> HiveExternalCatalog. Considering that views can be expanded
>>>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>>>> view API can make performance better if performance is the concern.
>>>>>> Further, a catalog can still be both a table and view provider if it
>>>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>>>> ability of providing both. Looking forward to more discussions on this and
>>>>>> making views a powerful tool in Spark.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>>
>>>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>>>
>>>>>>> Looks like we are running in circles. Should we have an online
>>>>>>> meeting to get this sorted out?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>>>> catalog to the v2 API sooner or later.
>>>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>>>> table/view resolution framework.
>>>>>>>> 3. It's better to avoid name conflicts between table and views at
>>>>>>>> the API level, instead of relying on the catalog implementation.
>>>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>>>
>>>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I don't think that it makes sense to discuss a different approach
>>>>>>>>> in the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>>>> purpose of an SPIP.
>>>>>>>>>
>>>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>>>> proposal (SPIP).
>>>>>>>>>>
>>>>>>>>>> The proposal is to add a ViewCatalog interface that can be used
>>>>>>>>>> to load, create, alter, and drop views in DataSourceV2.
>>>>>>>>>>
>>>>>>>>>> The full SPIP doc is here:
>>>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>>>
>>>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is
>>>>>>>>>> approved, I’ll update the PR for review.
>>>>>>>>>>
>>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>>>> [ ] +0
>>>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Software Engineer
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> John Zhuge
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

-- 
John Zhuge

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Xiao Li <ga...@gmail.com>.

Can we extend the voting window to next Wednesday? This week is a holiday
week for the lunar new year. AFAIK, many members in Asia are taking the
whole week off. They might not regularly check the emails.

Also how about starting a separate email thread starting with [VOTE] ?

Happy Lunar New Year!!!

Xiao

Holden Karau <ho...@pigscanfly.ca> 于2022年2月3日周四 12:28写道：

> +1 (binding)
>
> On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:
>
>> +1 (non-binding)
>>
>> Really looking forward to having this natively supported by Spark, so
>> that we can get rid of our own hacks to tie in a custom view catalog
>> implementation. I appreciate the care John has put into various parts of
>> the design and believe this will provide a robust and flexible solution to
>> this problem faced by various large-scale Spark users.
>>
>> Thanks John!
>>
>> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Hi Spark community,
>>>>
>>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>>> ).
>>>>
>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>>
>>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don’t think this is a good idea because …
>>>>
>>>> Thanks!
>>>>
>>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>>> idea. A view catalog will enable us to integrate Coral
>>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>>> only do it by maintaining our special version of the
>>>>> HiveExternalCatalog. Considering that views can be expanded
>>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>>> view API can make performance better if performance is the concern.
>>>>> Further, a catalog can still be both a table and view provider if it
>>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>>> ability of providing both. Looking forward to more discussions on this and
>>>>> making views a powerful tool in Spark.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> Looks like we are running in circles. Should we have an online
>>>>>> meeting to get this sorted out?
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>>> catalog to the v2 API sooner or later.
>>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>>> table/view resolution framework.
>>>>>>> 3. It's better to avoid name conflicts between table and views at
>>>>>>> the API level, instead of relying on the catalog implementation.
>>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>>
>>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't think that it makes sense to discuss a different approach
>>>>>>>> in the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>>> purpose of an SPIP.
>>>>>>>>
>>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>>> proposal (SPIP).
>>>>>>>>>
>>>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>>>
>>>>>>>>> The full SPIP doc is here:
>>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>>
>>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>>>> I’ll update the PR for review.
>>>>>>>>>
>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>>> [ ] +0
>>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Software Engineer
>>>>>>>> Netflix
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> John Zhuge
>>>>>>
>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Holden Karau <ho...@pigscanfly.ca>.

+1 (binding)

On Thu, Feb 3, 2022 at 2:26 PM Erik Krogen <xk...@apache.org> wrote:

> +1 (non-binding)
>
> Really looking forward to having this natively supported by Spark, so that
> we can get rid of our own hacks to tie in a custom view catalog
> implementation. I appreciate the care John has put into various parts of
> the design and believe this will provide a robust and flexible solution to
> this problem faced by various large-scale Spark users.
>
> Thanks John!
>
> On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> +1
>>
>> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Hi Spark community,
>>>
>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>>> ).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load,
>>> create, alter, and drop views in DataSourceV2.
>>>
>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>> Thanks!
>>>
>>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> Considering the API aspect, the ViewCatalog API sounds like a good
>>>> idea. A view catalog will enable us to integrate Coral
>>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>>> translation and management layer) very cleanly to Spark. Currently we can
>>>> only do it by maintaining our special version of the
>>>> HiveExternalCatalog. Considering that views can be expanded
>>>> syntactically without necessarily invoking the analyzer, using a dedicated
>>>> view API can make performance better if performance is the concern.
>>>> Further, a catalog can still be both a table and view provider if it
>>>> chooses to based on this design, so I do not think we necessarily lose the
>>>> ability of providing both. Looking forward to more discussions on this and
>>>> making views a powerful tool in Spark.
>>>>
>>>> Thanks,
>>>> Walaa.
>>>>
>>>>
>>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Looks like we are running in circles. Should we have an online meeting
>>>>> to get this sorted out?
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>>> 1. This is how Hive catalog works, and we need to migrate Hive
>>>>>> catalog to the v2 API sooner or later.
>>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>>> table/view resolution framework.
>>>>>> 3. It's better to avoid name conflicts between table and views at the
>>>>>> API level, instead of relying on the catalog implementation.
>>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>>
>>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't think that it makes sense to discuss a different approach in
>>>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>>>> purpose of an SPIP.
>>>>>>>
>>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>>> proposal (SPIP).
>>>>>>>>
>>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>>
>>>>>>>> The full SPIP doc is here:
>>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>>
>>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>>> I’ll update the PR for review.
>>>>>>>>
>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>> [ ] +0
>>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>>
>>>
>>> --
>>> John Zhuge
>>>
>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Erik Krogen <xk...@apache.org>.

+1 (non-binding)

Really looking forward to having this natively supported by Spark, so that
we can get rid of our own hacks to tie in a custom view catalog
implementation. I appreciate the care John has put into various parts of
the design and believe this will provide a robust and flexible solution to
this problem faced by various large-scale Spark users.

Thanks John!

On Thu, Feb 3, 2022 at 11:22 AM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> +1
>
> On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:
>
>> Hi Spark community,
>>
>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
>> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
>> ).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>> update the PR <https://github.com/apache/spark/pull/28147> for review.
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thanks!
>>
>> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Considering the API aspect, the ViewCatalog API sounds like a good idea.
>>> A view catalog will enable us to integrate Coral
>>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>>> translation and management layer) very cleanly to Spark. Currently we can
>>> only do it by maintaining our special version of the HiveExternalCatalog.
>>> Considering that views can be expanded syntactically without necessarily
>>> invoking the analyzer, using a dedicated view API can make performance
>>> better if performance is the concern. Further, a catalog can still be both
>>> a table and view provider if it chooses to based on this design, so I do
>>> not think we necessarily lose the ability of providing both. Looking
>>> forward to more discussions on this and making views a powerful tool in
>>> Spark.
>>>
>>> Thanks,
>>> Walaa.
>>>
>>>
>>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Looks like we are running in circles. Should we have an online meeting
>>>> to get this sorted out?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>>> wrote:
>>>>
>>>>> OK, then I'd vote for TableViewCatalog, because
>>>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>>>> to the v2 API sooner or later.
>>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>>> table/view resolution framework.
>>>>> 3. It's better to avoid name conflicts between table and views at the
>>>>> API level, instead of relying on the catalog implementation.
>>>>> 4. Caching invalidation is always a tricky problem.
>>>>>
>>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> I don't think that it makes sense to discuss a different approach in
>>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>>> purpose of an SPIP.
>>>>>>
>>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>>> proposal (SPIP).
>>>>>>>
>>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>>
>>>>>>> The full SPIP doc is here:
>>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>>
>>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>>> I’ll update the PR for review.
>>>>>>>
>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> [ ] +0
>>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>
>>
>> --
>> John Zhuge
>>
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Walaa Eldin Moustafa <wa...@gmail.com>.

+1

On Thu, Feb 3, 2022 at 11:19 AM John Zhuge <jz...@apache.org> wrote:

> Hi Spark community,
>
> I’d like to restart the vote for the ViewCatalog design proposal (SPIP
> <https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
> ).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
> update the PR <https://github.com/apache/spark/pull/28147> for review.
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thanks!
>
> On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <wa...@gmail.com>
> wrote:
>
>> Considering the API aspect, the ViewCatalog API sounds like a good idea.
>> A view catalog will enable us to integrate Coral
>> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
>> translation and management layer) very cleanly to Spark. Currently we can
>> only do it by maintaining our special version of the HiveExternalCatalog.
>> Considering that views can be expanded syntactically without necessarily
>> invoking the analyzer, using a dedicated view API can make performance
>> better if performance is the concern. Further, a catalog can still be both
>> a table and view provider if it chooses to based on this design, so I do
>> not think we necessarily lose the ability of providing both. Looking
>> forward to more discussions on this and making views a powerful tool in
>> Spark.
>>
>> Thanks,
>> Walaa.
>>
>>
>> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Looks like we are running in circles. Should we have an online meeting
>>> to get this sorted out?
>>>
>>> Thanks,
>>> John
>>>
>>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com>
>>> wrote:
>>>
>>>> OK, then I'd vote for TableViewCatalog, because
>>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>>> to the v2 API sooner or later.
>>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>>> table/view resolution framework.
>>>> 3. It's better to avoid name conflicts between table and views at the
>>>> API level, instead of relying on the catalog implementation.
>>>> 4. Caching invalidation is always a tricky problem.
>>>>
>>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> I don't think that it makes sense to discuss a different approach in
>>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>>> purpose of an SPIP.
>>>>>
>>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>>> proposal (SPIP).
>>>>>>
>>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>>
>>>>>> The full SPIP doc is here:
>>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>>
>>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>>> I’ll update the PR for review.
>>>>>>
>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> [ ] +0
>>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
>>>
>>> --
>>> John Zhuge
>>>
>>
>
> --
> John Zhuge
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Hi Spark community,

I’d like to restart the vote for the ViewCatalog design proposal (SPIP
<https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing>
).

The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
update the PR <https://github.com/apache/spark/pull/28147> for review.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

Thanks!

On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:

> Considering the API aspect, the ViewCatalog API sounds like a good idea. A
> view catalog will enable us to integrate Coral
> <https://engineering.linkedin.com/blog/2020/coral> (our view SQL
> translation and management layer) very cleanly to Spark. Currently we can
> only do it by maintaining our special version of the HiveExternalCatalog.
> Considering that views can be expanded syntactically without necessarily
> invoking the analyzer, using a dedicated view API can make performance
> better if performance is the concern. Further, a catalog can still be both
> a table and view provider if it chooses to based on this design, so I do
> not think we necessarily lose the ability of providing both. Looking
> forward to more discussions on this and making views a powerful tool in
> Spark.
>
> Thanks,
> Walaa.
>
>
> On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:
>
>> Looks like we are running in circles. Should we have an online meeting to
>> get this sorted out?
>>
>> Thanks,
>> John
>>
>> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> OK, then I'd vote for TableViewCatalog, because
>>> 1. This is how Hive catalog works, and we need to migrate Hive catalog
>>> to the v2 API sooner or later.
>>> 2. Because of 1, TableViewCatalog is easy to support in the current
>>> table/view resolution framework.
>>> 3. It's better to avoid name conflicts between table and views at the
>>> API level, instead of relying on the catalog implementation.
>>> 4. Caching invalidation is always a tricky problem.
>>>
>>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> I don't think that it makes sense to discuss a different approach in
>>>> the PR rather than in the vote. Let's discuss this now since that's the
>>>> purpose of an SPIP.
>>>>
>>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>>> proposal (SPIP).
>>>>>
>>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>>> load, create, alter, and drop views in DataSourceV2.
>>>>>
>>>>> The full SPIP doc is here:
>>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>>
>>>>> Please vote on the SPIP in the next 72 hours. Once it is approved,
>>>>> I’ll update the PR for review.
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don’t think this is a good idea because …
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>
>> --
>> John Zhuge
>>
>

-- 
John Zhuge

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Walaa Eldin Moustafa <wa...@gmail.com>.

Considering the API aspect, the ViewCatalog API sounds like a good idea. A
view catalog will enable us to integrate Coral
<https://engineering.linkedin.com/blog/2020/coral> (our view SQL
translation and management layer) very cleanly to Spark. Currently we can
only do it by maintaining our special version of the HiveExternalCatalog.
Considering that views can be expanded syntactically without necessarily
invoking the analyzer, using a dedicated view API can make performance
better if performance is the concern. Further, a catalog can still be both
a table and view provider if it chooses to based on this design, so I do
not think we necessarily lose the ability of providing both. Looking
forward to more discussions on this and making views a powerful tool in
Spark.

Thanks,
Walaa.

On Wed, May 26, 2021 at 9:54 AM John Zhuge <jz...@apache.org> wrote:

> Looks like we are running in circles. Should we have an online meeting to
> get this sorted out?
>
> Thanks,
> John
>
> On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> OK, then I'd vote for TableViewCatalog, because
>> 1. This is how Hive catalog works, and we need to migrate Hive catalog to
>> the v2 API sooner or later.
>> 2. Because of 1, TableViewCatalog is easy to support in the current
>> table/view resolution framework.
>> 3. It's better to avoid name conflicts between table and views at the API
>> level, instead of relying on the catalog implementation.
>> 4. Caching invalidation is always a tricky problem.
>>
>> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> I don't think that it makes sense to discuss a different approach in the
>>> PR rather than in the vote. Let's discuss this now since that's the purpose
>>> of an SPIP.
>>>
>>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>>> proposal (SPIP).
>>>>
>>>> The proposal is to add a ViewCatalog interface that can be used to
>>>> load, create, alter, and drop views in DataSourceV2.
>>>>
>>>> The full SPIP doc is here:
>>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>>
>>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>>> update the PR for review.
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don’t think this is a good idea because …
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> John Zhuge
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Looks like we are running in circles. Should we have an online meeting to
get this sorted out?

Thanks,
John

On Wed, May 26, 2021 at 12:01 AM Wenchen Fan <cl...@gmail.com> wrote:

> OK, then I'd vote for TableViewCatalog, because
> 1. This is how Hive catalog works, and we need to migrate Hive catalog to
> the v2 API sooner or later.
> 2. Because of 1, TableViewCatalog is easy to support in the current
> table/view resolution framework.
> 3. It's better to avoid name conflicts between table and views at the API
> level, instead of relying on the catalog implementation.
> 4. Caching invalidation is always a tricky problem.
>
> On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> I don't think that it makes sense to discuss a different approach in the
>> PR rather than in the vote. Let's discuss this now since that's the purpose
>> of an SPIP.
>>
>> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> Hi everyone, I’d like to start a vote for the ViewCatalog design
>>> proposal (SPIP).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load,
>>> create, alter, and drop views in DataSourceV2.
>>>
>>> The full SPIP doc is here:
>>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>>
>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>> update the PR for review.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
John Zhuge

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Wenchen Fan <cl...@gmail.com>.

OK, then I'd vote for TableViewCatalog, because
1. This is how Hive catalog works, and we need to migrate Hive catalog to
the v2 API sooner or later.
2. Because of 1, TableViewCatalog is easy to support in the current
table/view resolution framework.
3. It's better to avoid name conflicts between table and views at the API
level, instead of relying on the catalog implementation.
4. Caching invalidation is always a tricky problem.

On Tue, May 25, 2021 at 3:09 AM Ryan Blue <rb...@netflix.com.invalid> wrote:

> I don't think that it makes sense to discuss a different approach in the
> PR rather than in the vote. Let's discuss this now since that's the purpose
> of an SPIP.
>
> On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:
>
>> Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
>> (SPIP).
>>
>> The proposal is to add a ViewCatalog interface that can be used to load,
>> create, alter, and drop views in DataSourceV2.
>>
>> The full SPIP doc is here:
>> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>>
>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>> update the PR for review.
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [VOTE] SPIP: Catalog API for view metadata

Posted by Ryan Blue <rb...@netflix.com.INVALID>.

I don't think that it makes sense to discuss a different approach in the PR
rather than in the vote. Let's discuss this now since that's the purpose of
an SPIP.

On Mon, May 24, 2021 at 11:22 AM John Zhuge <jz...@apache.org> wrote:

> Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
> (SPIP).
>
> The proposal is to add a ViewCatalog interface that can be used to load,
> create, alter, and drop views in DataSourceV2.
>
> The full SPIP doc is here:
> https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing
>
> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
> update the PR for review.
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>


-- 
Ryan Blue
Software Engineer
Netflix

[VOTE] SPIP: Catalog API for view metadata

Posted by John Zhuge <jz...@apache.org>.

Hi everyone, I’d like to start a vote for the ViewCatalog design proposal
(SPIP).

The proposal is to add a ViewCatalog interface that can be used to load,
create, alter, and drop views in DataSourceV2.

The full SPIP doc is here:
https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
update the PR for review.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …