You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Ryan Blue <rb...@netflix.com.INVALID> on 2019/02/28 00:27:24 UTC
[VOTE] SPIP: Spark API for Table Metadata
Hi everyone,
In the last DSv2 sync, the consensus was that the table metadata SPIP was
ready to bring up for a vote. Now that the multi-catalog identifier SPIP
vote has passed, I'd like to start one for the table metadata API,
TableCatalog.
The proposal is for adding a TableCatalog interface that will be used by v2
plans. That interface has methods to load, create, drop, alter, refresh,
rename, and check existence for tables. It also specifies the set of
metadata used to configure tables: schema, partitioning, and key-value
properties. For more information, please read the SPIP proposal doc
<https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>
.
Please vote in the next 3 days.
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...
Thanks!
--
Ryan Blue
Software Engineer
Netflix
[RESULT] [VOTE] SPIP: Spark API for Table Metadata
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
This vote passes with 10 +1 votes (4 from PMC members) and no -1 or +0
votes.
Thanks for reviewing the proposal and voting, everyone!
On Sat, Mar 2, 2019 at 10:13 PM Felix Cheung <fe...@hotmail.com>
wrote:
> +1
>
> We reviewed and discussed this for quite some time.
>
>
> ------------------------------
> *From:* Xiao Li <li...@databricks.com>
> *Sent:* Friday, March 1, 2019 7:32 PM
> *To:* Wenchen Fan
> *Cc:* Reynold Xin; Ryan Blue; Spark Dev List
> *Subject:* Re: [VOTE] SPIP: Spark API for Table Metadata
>
> +1
>
> Thanks for your efforts! The high-level direction is fine to me. Let us
> discuss the details of APIs in the design stage and PR reviews.
>
> Xiao
>
> On Fri, Mar 1, 2019 at 6:21 PM Wenchen Fan <cl...@gmail.com> wrote:
>
>> +1, thanks for making it clear that this SPIP focuses on
>> high-level direction!
>>
>> On Sat, Mar 2, 2019 at 9:35 AM Reynold Xin <rx...@databricks.com> wrote:
>>
>>> Thanks Ryan. +1.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue <rb...@netflix.com> wrote:
>>>
>>>> Actually, I went ahead and removed the confusing section. There is no
>>>> public API in the doc now, so that it is clear that it isn't a relevant
>>>> part of this vote.
>>>>
>>>> On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rb...@netflix.com> wrote:
>>>>
>>>> I moved the public API to the "Implementation Sketch" section. That API
>>>> is not an important part of this, as that section notes.
>>>>
>>>> I completely agree that SPIPs should be high-level and that the
>>>> specifics, like method names, are not hard requirements. The proposal was
>>>> more of a sketch, but I was asked by Xiao in the DSv2 sync to make sure the
>>>> list of methods was complete. I think as long as we have agreement that the
>>>> intent is not to make the exact names binding, we should be okay.
>>>>
>>>> I can remove the user-facing API sketch, but I'd prefer to leave it in
>>>> the sketch section so we have it documented somewhere.
>>>>
>>>> On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com> wrote:
>>>>
>>>> Ryan - can you take the public user facing API part out of that SPIP?
>>>>
>>>> In general it'd be better to have the SPIPs be higher level, and put
>>>> the detailed APIs in a separate doc. Alternatively, put them in the SPIP
>>>> but explicitly vote on the high level stuff and not the detailed APIs.
>>>>
>>>> I don't want to get to a situation in which two months later the
>>>> identical APIs were committed with the justification that they were voted
>>>> on a while ago. In this case, it's even more serious because while I think
>>>> we all have consensus on the higher level internal API, not much discussion
>>>> has happened with the user-facing API and we should just leave that out
>>>> explicitly.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <
>>>> anthony.young-garner@cloudera.com.invalid> wrote:
>>>>
>>>> +1 (non-binding)
>>>>
>>>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>> +1 (non-binding)
>>>>
>>>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>>
>>>> *From:*Jamison Bennett <ja...@cloudera.com.INVALID>
>>>> *Date:*Thursday, February 28, 2019 at 8:28 AM
>>>> *To:*Ryan Blue <rb...@netflix.com>, Spark Dev List <
>>>> dev@spark.apache.org>
>>>> *Subject:*Re: [VOTE] SPIP: Spark API for Table Metadata
>>>>
>>>>
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>> *Jamison Bennett*
>>>>
>>>> Cloudera Software Engineer
>>>>
>>>> jamison.bennett@cloudera.com
>>>>
>>>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>>>> russell.spitzer@gmail.com> wrote:
>>>>
>>>> +1 (non-binding)
>>>>
>>>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>>
>>>>
>>>> In the last DSv2 sync, the consensus was that the table metadata SPIP
>>>> was ready to bring up for a vote. Now that the multi-catalog identifier
>>>> SPIP vote has passed, I'd like to start one for the table metadata API,
>>>> TableCatalog.
>>>>
>>>>
>>>>
>>>> The proposal is for adding a TableCatalog interface that will be used
>>>> by v2 plans. That interface has methods to load, create, drop, alter,
>>>> refresh, rename, and check existence for tables. It also specifies the set
>>>> of metadata used to configure tables: schema, partitioning, and key-value
>>>> properties. For more information, please read the SPIP proposal doc
>>>> [docs.google.com]
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>>>> .
>>>>
>>>>
>>>>
>>>> Please vote in the next 3 days.
>>>>
>>>>
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>
>>>> [ ] +0
>>>>
>>>> [ ] -1: I don't think this is a good idea because ...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Ryan Blue
>>>>
>>>> Software Engineer
>>>>
>>>> Netflix
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Ryan Blue
>>>>
>>>> Software Engineer
>>>>
>>>> Netflix
>>>>
>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Felix Cheung <fe...@hotmail.com>.
+1
We reviewed and discussed this for quite some time.
________________________________
From: Xiao Li <li...@databricks.com>
Sent: Friday, March 1, 2019 7:32 PM
To: Wenchen Fan
Cc: Reynold Xin; Ryan Blue; Spark Dev List
Subject: Re: [VOTE] SPIP: Spark API for Table Metadata
+1
Thanks for your efforts! The high-level direction is fine to me. Let us discuss the details of APIs in the design stage and PR reviews.
Xiao
On Fri, Mar 1, 2019 at 6:21 PM Wenchen Fan <cl...@gmail.com>> wrote:
+1, thanks for making it clear that this SPIP focuses on high-level direction!
On Sat, Mar 2, 2019 at 9:35 AM Reynold Xin <rx...@databricks.com>> wrote:
Thanks Ryan. +1.
On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue <rb...@netflix.com>> wrote:
Actually, I went ahead and removed the confusing section. There is no public API in the doc now, so that it is clear that it isn't a relevant part of this vote.
On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rb...@netflix.com>> wrote:
I moved the public API to the "Implementation Sketch" section. That API is not an important part of this, as that section notes.
I completely agree that SPIPs should be high-level and that the specifics, like method names, are not hard requirements. The proposal was more of a sketch, but I was asked by Xiao in the DSv2 sync to make sure the list of methods was complete. I think as long as we have agreement that the intent is not to make the exact names binding, we should be okay.
I can remove the user-facing API sketch, but I'd prefer to leave it in the sketch section so we have it documented somewhere.
On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com>> wrote:
Ryan - can you take the public user facing API part out of that SPIP?
In general it'd be better to have the SPIPs be higher level, and put the detailed APIs in a separate doc. Alternatively, put them in the SPIP but explicitly vote on the high level stuff and not the detailed APIs.
I don't want to get to a situation in which two months later the identical APIs were committed with the justification that they were voted on a while ago. In this case, it's even more serious because while I think we all have consensus on the higher level internal API, not much discussion has happened with the user-facing API and we should just leave that out explicitly.
On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <an...@cloudera.com.invalid>> wrote:
+1 (non-binding)
On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org>> wrote:
+1 (non-binding)
On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com>> wrote:
+1 (non-binding)
From:Jamison Bennett <ja...@cloudera.com.INVALID>>
Date:Thursday, February 28, 2019 at 8:28 AM
To:Ryan Blue <rb...@netflix.com>>, Spark Dev List <de...@spark.apache.org>>
Subject:Re: [VOTE] SPIP: Spark API for Table Metadata
+1 (non-binding)
Jamison Bennett
Cloudera Software Engineer
jamison.bennett@cloudera.com<ma...@cloudera.com>
515 Congress Ave, Suite 1212 | Austin, TX | 78701
On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>> wrote:
+1 (non-binding)
On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <ru...@gmail.com>> wrote:
+1 (non-binding)
On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>> wrote:
Hi everyone,
In the last DSv2 sync, the consensus was that the table metadata SPIP was ready to bring up for a vote. Now that the multi-catalog identifier SPIP vote has passed, I'd like to start one for the table metadata API, TableCatalog.
The proposal is for adding a TableCatalog interface that will be used by v2 plans. That interface has methods to load, create, drop, alter, refresh, rename, and check existence for tables. It also specifies the set of metadata used to configure tables: schema, partitioning, and key-value properties. For more information, please read the SPIP proposal doc [docs.google.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>.
Please vote in the next 3 days.
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...
Thanks!
--
Ryan Blue
Software Engineer
Netflix
--
Ryan Blue
Software Engineer
Netflix
--
John Zhuge
--
Ryan Blue
Software Engineer
Netflix
--
Ryan Blue
Software Engineer
Netflix
--
[https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Xiao Li <li...@databricks.com>.
+1
Thanks for your efforts! The high-level direction is fine to me. Let us
discuss the details of APIs in the design stage and PR reviews.
Xiao
On Fri, Mar 1, 2019 at 6:21 PM Wenchen Fan <cl...@gmail.com> wrote:
> +1, thanks for making it clear that this SPIP focuses on
> high-level direction!
>
> On Sat, Mar 2, 2019 at 9:35 AM Reynold Xin <rx...@databricks.com> wrote:
>
>> Thanks Ryan. +1.
>>
>>
>>
>>
>> On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue <rb...@netflix.com> wrote:
>>
>>> Actually, I went ahead and removed the confusing section. There is no
>>> public API in the doc now, so that it is clear that it isn't a relevant
>>> part of this vote.
>>>
>>> On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rb...@netflix.com> wrote:
>>>
>>> I moved the public API to the "Implementation Sketch" section. That API
>>> is not an important part of this, as that section notes.
>>>
>>> I completely agree that SPIPs should be high-level and that the
>>> specifics, like method names, are not hard requirements. The proposal was
>>> more of a sketch, but I was asked by Xiao in the DSv2 sync to make sure the
>>> list of methods was complete. I think as long as we have agreement that the
>>> intent is not to make the exact names binding, we should be okay.
>>>
>>> I can remove the user-facing API sketch, but I'd prefer to leave it in
>>> the sketch section so we have it documented somewhere.
>>>
>>> On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com> wrote:
>>>
>>> Ryan - can you take the public user facing API part out of that SPIP?
>>>
>>> In general it'd be better to have the SPIPs be higher level, and put the
>>> detailed APIs in a separate doc. Alternatively, put them in the SPIP but
>>> explicitly vote on the high level stuff and not the detailed APIs.
>>>
>>> I don't want to get to a situation in which two months later the
>>> identical APIs were committed with the justification that they were voted
>>> on a while ago. In this case, it's even more serious because while I think
>>> we all have consensus on the higher level internal API, not much discussion
>>> has happened with the user-facing API and we should just leave that out
>>> explicitly.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <
>>> anthony.young-garner@cloudera.com.invalid> wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
>>> *Date: *Thursday, February 28, 2019 at 8:28 AM
>>> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <
>>> dev@spark.apache.org>
>>> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>>>
>>>
>>>
>>> +1 (non-binding)
>>>
>>>
>>> *Jamison Bennett*
>>>
>>> Cloudera Software Engineer
>>>
>>> jamison.bennett@cloudera.com
>>>
>>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>>
>>>
>>> In the last DSv2 sync, the consensus was that the table metadata SPIP
>>> was ready to bring up for a vote. Now that the multi-catalog identifier
>>> SPIP vote has passed, I'd like to start one for the table metadata API,
>>> TableCatalog.
>>>
>>>
>>>
>>> The proposal is for adding a TableCatalog interface that will be used by
>>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>>> rename, and check existence for tables. It also specifies the set of
>>> metadata used to configure tables: schema, partitioning, and key-value
>>> properties. For more information, please read the SPIP proposal doc
>>> [docs.google.com]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>>> .
>>>
>>>
>>>
>>> Please vote in the next 3 days.
>>>
>>>
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>>
>>> [ ] +0
>>>
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>>
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>>
>>> Ryan Blue
>>>
>>> Software Engineer
>>>
>>> Netflix
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Ryan Blue
>>>
>>> Software Engineer
>>>
>>> Netflix
>>>
>>>
>>>
>>> --
>>> John Zhuge
>>>
>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Wenchen Fan <cl...@gmail.com>.
+1, thanks for making it clear that this SPIP focuses on
high-level direction!
On Sat, Mar 2, 2019 at 9:35 AM Reynold Xin <rx...@databricks.com> wrote:
> Thanks Ryan. +1.
>
>
>
>
> On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue <rb...@netflix.com> wrote:
>
>> Actually, I went ahead and removed the confusing section. There is no
>> public API in the doc now, so that it is clear that it isn't a relevant
>> part of this vote.
>>
>> On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>> I moved the public API to the "Implementation Sketch" section. That API
>> is not an important part of this, as that section notes.
>>
>> I completely agree that SPIPs should be high-level and that the
>> specifics, like method names, are not hard requirements. The proposal was
>> more of a sketch, but I was asked by Xiao in the DSv2 sync to make sure the
>> list of methods was complete. I think as long as we have agreement that the
>> intent is not to make the exact names binding, we should be okay.
>>
>> I can remove the user-facing API sketch, but I'd prefer to leave it in
>> the sketch section so we have it documented somewhere.
>>
>> On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com> wrote:
>>
>> Ryan - can you take the public user facing API part out of that SPIP?
>>
>> In general it'd be better to have the SPIPs be higher level, and put the
>> detailed APIs in a separate doc. Alternatively, put them in the SPIP but
>> explicitly vote on the high level stuff and not the detailed APIs.
>>
>> I don't want to get to a situation in which two months later the
>> identical APIs were committed with the justification that they were voted
>> on a while ago. In this case, it's even more serious because while I think
>> we all have consensus on the higher level internal API, not much discussion
>> has happened with the user-facing API and we should just leave that out
>> explicitly.
>>
>>
>>
>>
>>
>>
>> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <
>> anthony.young-garner@cloudera.com.invalid> wrote:
>>
>> +1 (non-binding)
>>
>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
>>
>> +1 (non-binding)
>>
>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
>> *Date: *Thursday, February 28, 2019 at 8:28 AM
>> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <dev@spark.apache.org
>> >
>> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>>
>>
>>
>> +1 (non-binding)
>>
>>
>> *Jamison Bennett*
>>
>> Cloudera Software Engineer
>>
>> jamison.bennett@cloudera.com
>>
>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>
>>
>>
>>
>>
>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>> +1 (non-binding)
>>
>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> Hi everyone,
>>
>>
>>
>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>> vote has passed, I'd like to start one for the table metadata API,
>> TableCatalog.
>>
>>
>>
>> The proposal is for adding a TableCatalog interface that will be used by
>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>> rename, and check existence for tables. It also specifies the set of
>> metadata used to configure tables: schema, partitioning, and key-value
>> properties. For more information, please read the SPIP proposal doc
>> [docs.google.com]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>> .
>>
>>
>>
>> Please vote in the next 3 days.
>>
>>
>>
>> [ ] +1: Accept the proposal as an official SPIP
>>
>> [ ] +0
>>
>> [ ] -1: I don't think this is a good idea because ...
>>
>>
>>
>>
>>
>> Thanks!
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>>
>> --
>> John Zhuge
>>
>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Reynold Xin <rx...@databricks.com>.
Thanks Ryan. +1.
On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue < rblue@netflix.com > wrote:
>
> Actually, I went ahead and removed the confusing section. There is no
> public API in the doc now, so that it is clear that it isn't a relevant
> part of this vote.
>
> On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue < rblue@ netflix. com (
> rblue@netflix.com ) > wrote:
>
>
>> I moved the public API to the "Implementation Sketch" section. That API is
>> not an important part of this, as that section notes.
>>
>>
>> I completely agree that SPIPs should be high-level and that the specifics,
>> like method names, are not hard requirements. The proposal was more of a
>> sketch, but I was asked by Xiao in the DSv2 sync to make sure the list of
>> methods was complete. I think as long as we have agreement that the intent
>> is not to make the exact names binding, we should be okay.
>>
>>
>> I can remove the user-facing API sketch, but I'd prefer to leave it in the
>> sketch section so we have it documented somewhere.
>>
>> On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin < rxin@ databricks. com (
>> rxin@databricks.com ) > wrote:
>>
>>
>>> Ryan - can you take the public user facing API part out of that SPIP?
>>>
>>>
>>>
>>> In general it'd be better to have the SPIPs be higher level, and put the
>>> detailed APIs in a separate doc. Alternatively, put them in the SPIP but
>>> explicitly vote on the high level stuff and not the detailed APIs.
>>>
>>>
>>>
>>> I don't want to get to a situation in which two months later the identical
>>> APIs were committed with the justification that they were voted on a while
>>> ago. In this case, it's even more serious because while I think we all
>>> have consensus on the higher level internal API, not much discussion has
>>> happened with the user-facing API and we should just leave that out
>>> explicitly.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner < anthony. young-garner@
>>> cloudera. com. invalid ( anthony.young-garner@cloudera.com.invalid ) > wrote:
>>>
>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge < jzhuge@ apache. org (
>>>> jzhuge@apache.org ) > wrote:
>>>>
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>>
>>>>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah < mcheah@ palantir. com (
>>>>> mcheah@palantir.com ) > wrote:
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Jamison Bennett < jamison. bennett@ cloudera. com. INVALID (
>>>>>> jamison.bennett@cloudera.com.INVALID ) >
>>>>>> *Date:* Thursday, February 28, 2019 at 8:28 AM
>>>>>> *To:* Ryan Blue < rblue@ netflix. com ( rblue@netflix.com ) >, Spark Dev
>>>>>> List < dev@ spark. apache. org ( dev@spark.apache.org ) >
>>>>>> *Subject:* Re: [VOTE] SPIP: Spark API for Table Metadata
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jamison Bennett*
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cloudera Software Engineer
>>>>>>
>>>>>>
>>>>>>
>>>>>> jamison. bennett@ cloudera. com ( jamison.bennett@cloudera.com )
>>>>>>
>>>>>>
>>>>>>
>>>>>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue < rblue@ netflix. com. invalid (
>>>>>> rblue@netflix.com.invalid ) > wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer < russell. spitzer@ gmail.
>>>>>>> com ( russell.spitzer@gmail.com ) > wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> +1 (non-binding)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue < rblue@ netflix. com. invalid (
>>>>>>>> rblue@netflix.com.invalid ) > wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>>>>>>>>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>>>>>>>>> vote has passed, I'd like to start one for the table metadata API,
>>>>>>>>> TableCatalog.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The proposal is for adding a TableCatalog interface that will be used by
>>>>>>>>> v2 plans. That interface has methods to load, create, drop, alter,
>>>>>>>>> refresh, rename, and check existence for tables. It also specifies the set
>>>>>>>>> of metadata used to configure tables: schema, partitioning, and key-value
>>>>>>>>> properties. For more information, please read the SPIP proposal doc [docs.
>>>>>>>>> google. com] (
>>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=
>>>>>>>>> ).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please vote in the next 3 days.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [ ] +0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ryan Blue
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Software Engineer
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Netflix
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ryan Blue
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Software Engineer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Netflix
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Actually, I went ahead and removed the confusing section. There is no
public API in the doc now, so that it is clear that it isn't a relevant
part of this vote.
On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rb...@netflix.com> wrote:
> I moved the public API to the "Implementation Sketch" section. That API is
> not an important part of this, as that section notes.
>
> I completely agree that SPIPs should be high-level and that the specifics,
> like method names, are not hard requirements. The proposal was more of a
> sketch, but I was asked by Xiao in the DSv2 sync to make sure the list of
> methods was complete. I think as long as we have agreement that the intent
> is not to make the exact names binding, we should be okay.
>
> I can remove the user-facing API sketch, but I'd prefer to leave it in the
> sketch section so we have it documented somewhere.
>
> On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com> wrote:
>
>> Ryan - can you take the public user facing API part out of that SPIP?
>>
>> In general it'd be better to have the SPIPs be higher level, and put the
>> detailed APIs in a separate doc. Alternatively, put them in the SPIP but
>> explicitly vote on the high level stuff and not the detailed APIs.
>>
>> I don't want to get to a situation in which two months later the
>> identical APIs were committed with the justification that they were voted
>> on a while ago. In this case, it's even more serious because while I think
>> we all have consensus on the higher level internal API, not much discussion
>> has happened with the user-facing API and we should just leave that out
>> explicitly.
>>
>>
>>
>>
>>
>>
>> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <
>> anthony.young-garner@cloudera.com.invalid> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
>>> *Date: *Thursday, February 28, 2019 at 8:28 AM
>>> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <
>>> dev@spark.apache.org>
>>> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>>>
>>>
>>>
>>> +1 (non-binding)
>>>
>>>
>>> *Jamison Bennett*
>>>
>>> Cloudera Software Engineer
>>>
>>> jamison.bennett@cloudera.com
>>>
>>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>>
>>>
>>> In the last DSv2 sync, the consensus was that the table metadata SPIP
>>> was ready to bring up for a vote. Now that the multi-catalog identifier
>>> SPIP vote has passed, I'd like to start one for the table metadata API,
>>> TableCatalog.
>>>
>>>
>>>
>>> The proposal is for adding a TableCatalog interface that will be used by
>>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>>> rename, and check existence for tables. It also specifies the set of
>>> metadata used to configure tables: schema, partitioning, and key-value
>>> properties. For more information, please read the SPIP proposal doc
>>> [docs.google.com]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>>> .
>>>
>>>
>>>
>>> Please vote in the next 3 days.
>>>
>>>
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>>
>>> [ ] +0
>>>
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>>
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>>
>>> Ryan Blue
>>>
>>> Software Engineer
>>>
>>> Netflix
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Ryan Blue
>>>
>>> Software Engineer
>>>
>>> Netflix
>>>
>>>
>>>
>>> --
>>> John Zhuge
>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
I moved the public API to the "Implementation Sketch" section. That API is
not an important part of this, as that section notes.
I completely agree that SPIPs should be high-level and that the specifics,
like method names, are not hard requirements. The proposal was more of a
sketch, but I was asked by Xiao in the DSv2 sync to make sure the list of
methods was complete. I think as long as we have agreement that the intent
is not to make the exact names binding, we should be okay.
I can remove the user-facing API sketch, but I'd prefer to leave it in the
sketch section so we have it documented somewhere.
On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rx...@databricks.com> wrote:
> Ryan - can you take the public user facing API part out of that SPIP?
>
> In general it'd be better to have the SPIPs be higher level, and put the
> detailed APIs in a separate doc. Alternatively, put them in the SPIP but
> explicitly vote on the high level stuff and not the detailed APIs.
>
> I don't want to get to a situation in which two months later the identical
> APIs were committed with the justification that they were voted on a while
> ago. In this case, it's even more serious because while I think we all have
> consensus on the higher level internal API, not much discussion has
> happened with the user-facing API and we should just leave that out
> explicitly.
>
>
>
>
>
>
> On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <
> anthony.young-garner@cloudera.com.invalid> wrote:
>
>> +1 (non-binding)
>>
>> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
>>
>> +1 (non-binding)
>>
>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
>> *Date: *Thursday, February 28, 2019 at 8:28 AM
>> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <dev@spark.apache.org
>> >
>> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>>
>>
>>
>> +1 (non-binding)
>>
>>
>> *Jamison Bennett*
>>
>> Cloudera Software Engineer
>>
>> jamison.bennett@cloudera.com
>>
>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>
>>
>>
>>
>>
>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>> +1 (non-binding)
>>
>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> Hi everyone,
>>
>>
>>
>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>> vote has passed, I'd like to start one for the table metadata API,
>> TableCatalog.
>>
>>
>>
>> The proposal is for adding a TableCatalog interface that will be used by
>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>> rename, and check existence for tables. It also specifies the set of
>> metadata used to configure tables: schema, partitioning, and key-value
>> properties. For more information, please read the SPIP proposal doc
>> [docs.google.com]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>> .
>>
>>
>>
>> Please vote in the next 3 days.
>>
>>
>>
>> [ ] +1: Accept the proposal as an official SPIP
>>
>> [ ] +0
>>
>> [ ] -1: I don't think this is a good idea because ...
>>
>>
>>
>>
>>
>> Thanks!
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>>
>> --
>> John Zhuge
>>
>>
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Reynold Xin <rx...@databricks.com>.
Ryan - can you take the public user facing API part out of that SPIP?
In general it'd be better to have the SPIPs be higher level, and put the detailed APIs in a separate doc. Alternatively, put them in the SPIP but explicitly vote on the high level stuff and not the detailed APIs.
I don't want to get to a situation in which two months later the identical APIs were committed with the justification that they were voted on a while ago. In this case, it's even more serious because while I think we all have consensus on the higher level internal API, not much discussion has happened with the user-facing API and we should just leave that out explicitly.
On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner < anthony.young-garner@cloudera.com.invalid > wrote:
>
> +1 (non-binding)
>
>
> On Thu, Feb 28, 2019 at 5:54 PM John Zhuge < jzhuge@ apache. org (
> jzhuge@apache.org ) > wrote:
>
>
>> +1 (non-binding)
>>
>>
>> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah < mcheah@ palantir. com (
>> mcheah@palantir.com ) > wrote:
>>
>>
>>>
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Jamison Bennett < jamison. bennett@ cloudera. com. INVALID (
>>> jamison.bennett@cloudera.com.INVALID ) >
>>> *Date:* Thursday, February 28, 2019 at 8:28 AM
>>> *To:* Ryan Blue < rblue@ netflix. com ( rblue@netflix.com ) >, Spark Dev
>>> List < dev@ spark. apache. org ( dev@spark.apache.org ) >
>>> *Subject:* Re: [VOTE] SPIP: Spark API for Table Metadata
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Jamison Bennett*
>>>
>>>
>>>
>>> Cloudera Software Engineer
>>>
>>>
>>>
>>> jamison. bennett@ cloudera. com ( jamison.bennett@cloudera.com )
>>>
>>>
>>>
>>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue < rblue@ netflix. com. invalid (
>>> rblue@netflix.com.invalid ) > wrote:
>>>
>>>
>>>
>>>>
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer < russell. spitzer@ gmail.
>>>> com ( russell.spitzer@gmail.com ) > wrote:
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue < rblue@ netflix. com. invalid (
>>>>> rblue@netflix.com.invalid ) > wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>>>>>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>>>>>> vote has passed, I'd like to start one for the table metadata API,
>>>>>> TableCatalog.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The proposal is for adding a TableCatalog interface that will be used by
>>>>>> v2 plans. That interface has methods to load, create, drop, alter,
>>>>>> refresh, rename, and check existence for tables. It also specifies the set
>>>>>> of metadata used to configure tables: schema, partitioning, and key-value
>>>>>> properties. For more information, please read the SPIP proposal doc [docs.
>>>>>> google. com] (
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=
>>>>>> ).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Please vote in the next 3 days.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [ ] +0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ryan Blue
>>>>>>
>>>>>>
>>>>>>
>>>>>> Software Engineer
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Netflix
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>> Ryan Blue
>>>>
>>>>
>>>>
>>>> Software Engineer
>>>>
>>>>
>>>>
>>>>
>>>> Netflix
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>> --
>> John Zhuge
>>
>
>
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Anthony Young-Garner <an...@cloudera.com.INVALID>.
+1 (non-binding)
On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jz...@apache.org> wrote:
> +1 (non-binding)
>
> On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
>
>> +1 (non-binding)
>>
>>
>>
>> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
>> *Date: *Thursday, February 28, 2019 at 8:28 AM
>> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <dev@spark.apache.org
>> >
>> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>>
>>
>>
>> +1 (non-binding)
>>
>>
>> *Jamison Bennett*
>>
>> Cloudera Software Engineer
>>
>> jamison.bennett@cloudera.com
>>
>> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>>
>>
>>
>>
>>
>> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>> +1 (non-binding)
>>
>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> Hi everyone,
>>
>>
>>
>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>> vote has passed, I'd like to start one for the table metadata API,
>> TableCatalog.
>>
>>
>>
>> The proposal is for adding a TableCatalog interface that will be used by
>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>> rename, and check existence for tables. It also specifies the set of
>> metadata used to configure tables: schema, partitioning, and key-value
>> properties. For more information, please read the SPIP proposal doc
>> [docs.google.com]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
>> .
>>
>>
>>
>> Please vote in the next 3 days.
>>
>>
>>
>> [ ] +1: Accept the proposal as an official SPIP
>>
>> [ ] +0
>>
>> [ ] -1: I don't think this is a good idea because ...
>>
>>
>>
>>
>>
>> Thanks!
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>>
>>
>> --
>>
>> Ryan Blue
>>
>> Software Engineer
>>
>> Netflix
>>
>>
>
> --
> John Zhuge
>
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by John Zhuge <jz...@apache.org>.
+1 (non-binding)
On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mc...@palantir.com> wrote:
> +1 (non-binding)
>
>
>
> *From: *Jamison Bennett <ja...@cloudera.com.INVALID>
> *Date: *Thursday, February 28, 2019 at 8:28 AM
> *To: *Ryan Blue <rb...@netflix.com>, Spark Dev List <de...@spark.apache.org>
> *Subject: *Re: [VOTE] SPIP: Spark API for Table Metadata
>
>
>
> +1 (non-binding)
>
>
> *Jamison Bennett*
>
> Cloudera Software Engineer
>
> jamison.bennett@cloudera.com
>
> 515 Congress Ave, Suite 1212 | Austin, TX | 78701
>
>
>
>
>
> On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
> +1 (non-binding)
>
>
>
> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
> +1 (non-binding)
>
> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>
> Hi everyone,
>
>
>
> In the last DSv2 sync, the consensus was that the table metadata SPIP was
> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
> vote has passed, I'd like to start one for the table metadata API,
> TableCatalog.
>
>
>
> The proposal is for adding a TableCatalog interface that will be used by
> v2 plans. That interface has methods to load, create, drop, alter, refresh,
> rename, and check existence for tables. It also specifies the set of
> metadata used to configure tables: schema, partitioning, and key-value
> properties. For more information, please read the SPIP proposal doc
> [docs.google.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>
> .
>
>
>
> Please vote in the next 3 days.
>
>
>
> [ ] +1: Accept the proposal as an official SPIP
>
> [ ] +0
>
> [ ] -1: I don't think this is a good idea because ...
>
>
>
>
>
> Thanks!
>
>
>
> --
>
> Ryan Blue
>
> Software Engineer
>
> Netflix
>
>
>
>
> --
>
> Ryan Blue
>
> Software Engineer
>
> Netflix
>
>
--
John Zhuge
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Matt Cheah <mc...@palantir.com>.
+1 (non-binding)
From: Jamison Bennett <ja...@cloudera.com.INVALID>
Date: Thursday, February 28, 2019 at 8:28 AM
To: Ryan Blue <rb...@netflix.com>, Spark Dev List <de...@spark.apache.org>
Subject: Re: [VOTE] SPIP: Spark API for Table Metadata
+1 (non-binding)
Jamison Bennett
Cloudera Software Engineer
jamison.bennett@cloudera.com
515 Congress Ave, Suite 1212 | Austin, TX | 78701
On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
+1 (non-binding)
On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <ru...@gmail.com> wrote:
+1 (non-binding)
On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
Hi everyone,
In the last DSv2 sync, the consensus was that the table metadata SPIP was ready to bring up for a vote. Now that the multi-catalog identifier SPIP vote has passed, I'd like to start one for the table metadata API, TableCatalog.
The proposal is for adding a TableCatalog interface that will be used by v2 plans. That interface has methods to load, create, drop, alter, refresh, rename, and check existence for tables. It also specifies the set of metadata used to configure tables: schema, partitioning, and key-value properties. For more information, please read the SPIP proposal doc [docs.google.com].
Please vote in the next 3 days.
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...
Thanks!
--
Ryan Blue
Software Engineer
Netflix
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Jamison Bennett <ja...@cloudera.com.INVALID>.
+1 (non-binding)
Jamison Bennett
Cloudera Software Engineer
jamison.bennett@cloudera.com
515 Congress Ave, Suite 1212 | Austin, TX | 78701
On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rb...@netflix.com.invalid>
wrote:
> +1 (non-binding)
>
> On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> +1 (non-binding)
>>
>> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> In the last DSv2 sync, the consensus was that the table metadata SPIP
>>> was ready to bring up for a vote. Now that the multi-catalog identifier
>>> SPIP vote has passed, I'd like to start one for the table metadata API,
>>> TableCatalog.
>>>
>>> The proposal is for adding a TableCatalog interface that will be used by
>>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>>> rename, and check existence for tables. It also specifies the set of
>>> metadata used to configure tables: schema, partitioning, and key-value
>>> properties. For more information, please read the SPIP proposal doc
>>> <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>
>>> .
>>>
>>> Please vote in the next 3 days.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>>
>>> Thanks!
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
+1 (non-binding)
On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <ru...@gmail.com>
wrote:
> +1 (non-binding)
>
> On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>
>> Hi everyone,
>>
>> In the last DSv2 sync, the consensus was that the table metadata SPIP was
>> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
>> vote has passed, I'd like to start one for the table metadata API,
>> TableCatalog.
>>
>> The proposal is for adding a TableCatalog interface that will be used by
>> v2 plans. That interface has methods to load, create, drop, alter, refresh,
>> rename, and check existence for tables. It also specifies the set of
>> metadata used to configure tables: schema, partitioning, and key-value
>> properties. For more information, please read the SPIP proposal doc
>> <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>
>> .
>>
>> Please vote in the next 3 days.
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don't think this is a good idea because ...
>>
>>
>> Thanks!
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] SPIP: Spark API for Table Metadata
Posted by Russell Spitzer <ru...@gmail.com>.
+1 (non-binding)
On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Hi everyone,
>
> In the last DSv2 sync, the consensus was that the table metadata SPIP was
> ready to bring up for a vote. Now that the multi-catalog identifier SPIP
> vote has passed, I'd like to start one for the table metadata API,
> TableCatalog.
>
> The proposal is for adding a TableCatalog interface that will be used by
> v2 plans. That interface has methods to load, create, drop, alter, refresh,
> rename, and check existence for tables. It also specifies the set of
> metadata used to configure tables: schema, partitioning, and key-value
> properties. For more information, please read the SPIP proposal doc
> <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>
> .
>
> Please vote in the next 3 days.
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
>
>
> Thanks!
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>