You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Jack Ye <ye...@gmail.com> on 2021/08/29 07:19:58 UTC
Snapshot tagging, branching and retention
Hi everyone,
Recently I have published PR 2961 - add snapshot tags interface (
https://github.com/apache/iceberg/pull/2961) and received a lot of great
feedback. I have summarized everything in the discussions and put up a
design to discuss the path forward around snapshot tagging, branching and
retention:
https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
Any feedback around the doc would be much appreciated!
Also, to facilitate future changes in Iceberg spec, it would be very
helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
https://github.com/apache/iceberg/pull/2957) which would make changing
TableMetadata much simpler.
Thanks,
Jack Ye
Re: Snapshot tagging, branching and retention
Posted by Kyle Bendickson <ky...@tabular.io>.
Thanks for collecting these notes as well as for the proposal, Jack. Have
been traveling today so I couldn't attend.
Will be looking out for the new PR.
Best,
Kyle Bendickson (@kbendick)
On Mon, Oct 18, 2021 at 9:58 AM Jack Ye <ye...@gmail.com> wrote:
> Thanks to everyone who came to the meeting for the discussion. Here is the
> meeting note:
> https://docs.google.com/document/d/1yVxvgQfGDUdKsr6j60jL54LKZSUBvLy9QEQstVNrWYQ/edit#
>
> As the next step, I will proceed with implementation in the current open
> PR https://github.com/apache/iceberg/pull/3104, and also publish a new PR
> to document all the spec changes.
>
> Best,
> Jack Ye
>
> On Wed, Oct 13, 2021 at 8:23 PM Jack Ye <ye...@gmail.com> wrote:
>
>> Sure, I will take note and publish it to this thread.
>> -Jack
>>
>> On Wed, Oct 13, 2021 at 7:18 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Is it possible to maintain a meeting note for this and publish it to the
>>> mail list because I don't think everybody could attend this meeting ?
>>>
>>> Thanks.
>>>
>>> On Thu, Oct 14, 2021 at 2:00 AM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Based on some offline discussions with different people around
>>>> availability, we will hold the meeting on Monday 10/18 9am PDT.
>>>>
>>>> Here is the meeting link: meet.google.com/ubj-kvfm-ehg
>>>>
>>>> I have added all the people in this thread to the invite. Feel free to
>>>> also forward the meeting to anyone else interested.
>>>>
>>>> Best,
>>>> Jack Ye
>>>>
>>>> On Mon, Oct 11, 2021 at 8:53 AM Eduard Tudenhoefner <ed...@dremio.com>
>>>> wrote:
>>>>
>>>>> Hey Jack,
>>>>>
>>>>> would this week on Wednesday work for you from 9 to 10am PDT?
>>>>>
>>>>> On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> We have had a few iterations of the design doc with various people,
>>>>>> thanks for all the feedback. I am thinking about a meeting to finalize the
>>>>>> design and move forward with implementation.
>>>>>>
>>>>>> Considering the various time zones, I propose we choose any time from
>>>>>> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>>>>>>
>>>>>> If anyone is interested in joining, please let me know the preferred
>>>>>> time slot.
>>>>>>
>>>>>> Best,
>>>>>> Jack Ye
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <
>>>>>> eduard@dremio.com> wrote:
>>>>>>
>>>>>>> Nice work Jack, the proposal looks really good.
>>>>>>>
>>>>>>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Recently I have published PR 2961 - add snapshot tags interface (
>>>>>>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>>>>>>> great feedback. I have summarized everything in the discussions and put up
>>>>>>>> a design to discuss the path forward around snapshot tagging, branching and
>>>>>>>> retention:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>>>>>>
>>>>>>>> Any feedback around the doc would be much appreciated!
>>>>>>>>
>>>>>>>> Also, to facilitate future changes in Iceberg spec, it would be
>>>>>>>> very helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>>>>>>> https://github.com/apache/iceberg/pull/2957) which would make
>>>>>>>> changing TableMetadata much simpler.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jack Ye
>>>>>>>>
>>>>>>>
Re: Snapshot tagging, branching and retention
Posted by Jack Ye <ye...@gmail.com>.
Thanks to everyone who came to the meeting for the discussion. Here is the
meeting note:
https://docs.google.com/document/d/1yVxvgQfGDUdKsr6j60jL54LKZSUBvLy9QEQstVNrWYQ/edit#
As the next step, I will proceed with implementation in the current open PR
https://github.com/apache/iceberg/pull/3104, and also publish a new PR to
document all the spec changes.
Best,
Jack Ye
On Wed, Oct 13, 2021 at 8:23 PM Jack Ye <ye...@gmail.com> wrote:
> Sure, I will take note and publish it to this thread.
> -Jack
>
> On Wed, Oct 13, 2021 at 7:18 PM OpenInx <op...@gmail.com> wrote:
>
>> Is it possible to maintain a meeting note for this and publish it to the
>> mail list because I don't think everybody could attend this meeting ?
>>
>> Thanks.
>>
>> On Thu, Oct 14, 2021 at 2:00 AM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Based on some offline discussions with different people around
>>> availability, we will hold the meeting on Monday 10/18 9am PDT.
>>>
>>> Here is the meeting link: meet.google.com/ubj-kvfm-ehg
>>>
>>> I have added all the people in this thread to the invite. Feel free to
>>> also forward the meeting to anyone else interested.
>>>
>>> Best,
>>> Jack Ye
>>>
>>> On Mon, Oct 11, 2021 at 8:53 AM Eduard Tudenhoefner <ed...@dremio.com>
>>> wrote:
>>>
>>>> Hey Jack,
>>>>
>>>> would this week on Wednesday work for you from 9 to 10am PDT?
>>>>
>>>> On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> We have had a few iterations of the design doc with various people,
>>>>> thanks for all the feedback. I am thinking about a meeting to finalize the
>>>>> design and move forward with implementation.
>>>>>
>>>>> Considering the various time zones, I propose we choose any time from
>>>>> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>>>>>
>>>>> If anyone is interested in joining, please let me know the preferred
>>>>> time slot.
>>>>>
>>>>> Best,
>>>>> Jack Ye
>>>>>
>>>>>
>>>>> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <
>>>>> eduard@dremio.com> wrote:
>>>>>
>>>>>> Nice work Jack, the proposal looks really good.
>>>>>>
>>>>>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Recently I have published PR 2961 - add snapshot tags interface (
>>>>>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>>>>>> great feedback. I have summarized everything in the discussions and put up
>>>>>>> a design to discuss the path forward around snapshot tagging, branching and
>>>>>>> retention:
>>>>>>>
>>>>>>>
>>>>>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>>>>>
>>>>>>> Any feedback around the doc would be much appreciated!
>>>>>>>
>>>>>>> Also, to facilitate future changes in Iceberg spec, it would be very
>>>>>>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>>>>>> https://github.com/apache/iceberg/pull/2957) which would make
>>>>>>> changing TableMetadata much simpler.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jack Ye
>>>>>>>
>>>>>>
Re: Snapshot tagging, branching and retention
Posted by Jack Ye <ye...@gmail.com>.
Sure, I will take note and publish it to this thread.
-Jack
On Wed, Oct 13, 2021 at 7:18 PM OpenInx <op...@gmail.com> wrote:
> Is it possible to maintain a meeting note for this and publish it to the
> mail list because I don't think everybody could attend this meeting ?
>
> Thanks.
>
> On Thu, Oct 14, 2021 at 2:00 AM Jack Ye <ye...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Based on some offline discussions with different people around
>> availability, we will hold the meeting on Monday 10/18 9am PDT.
>>
>> Here is the meeting link: meet.google.com/ubj-kvfm-ehg
>>
>> I have added all the people in this thread to the invite. Feel free to
>> also forward the meeting to anyone else interested.
>>
>> Best,
>> Jack Ye
>>
>> On Mon, Oct 11, 2021 at 8:53 AM Eduard Tudenhoefner <ed...@dremio.com>
>> wrote:
>>
>>> Hey Jack,
>>>
>>> would this week on Wednesday work for you from 9 to 10am PDT?
>>>
>>> On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> We have had a few iterations of the design doc with various people,
>>>> thanks for all the feedback. I am thinking about a meeting to finalize the
>>>> design and move forward with implementation.
>>>>
>>>> Considering the various time zones, I propose we choose any time from
>>>> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>>>>
>>>> If anyone is interested in joining, please let me know the preferred
>>>> time slot.
>>>>
>>>> Best,
>>>> Jack Ye
>>>>
>>>>
>>>> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>> wrote:
>>>>
>>>>> Nice work Jack, the proposal looks really good.
>>>>>
>>>>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Recently I have published PR 2961 - add snapshot tags interface (
>>>>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>>>>> great feedback. I have summarized everything in the discussions and put up
>>>>>> a design to discuss the path forward around snapshot tagging, branching and
>>>>>> retention:
>>>>>>
>>>>>>
>>>>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>>>>
>>>>>> Any feedback around the doc would be much appreciated!
>>>>>>
>>>>>> Also, to facilitate future changes in Iceberg spec, it would be very
>>>>>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>>>>> https://github.com/apache/iceberg/pull/2957) which would make
>>>>>> changing TableMetadata much simpler.
>>>>>>
>>>>>> Thanks,
>>>>>> Jack Ye
>>>>>>
>>>>>
Re: Snapshot tagging, branching and retention
Posted by OpenInx <op...@gmail.com>.
Is it possible to maintain a meeting note for this and publish it to the
mail list because I don't think everybody could attend this meeting ?
Thanks.
On Thu, Oct 14, 2021 at 2:00 AM Jack Ye <ye...@gmail.com> wrote:
> Hi everyone,
>
> Based on some offline discussions with different people around
> availability, we will hold the meeting on Monday 10/18 9am PDT.
>
> Here is the meeting link: meet.google.com/ubj-kvfm-ehg
>
> I have added all the people in this thread to the invite. Feel free to
> also forward the meeting to anyone else interested.
>
> Best,
> Jack Ye
>
> On Mon, Oct 11, 2021 at 8:53 AM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
>> Hey Jack,
>>
>> would this week on Wednesday work for you from 9 to 10am PDT?
>>
>> On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> We have had a few iterations of the design doc with various people,
>>> thanks for all the feedback. I am thinking about a meeting to finalize the
>>> design and move forward with implementation.
>>>
>>> Considering the various time zones, I propose we choose any time from
>>> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>>>
>>> If anyone is interested in joining, please let me know the preferred
>>> time slot.
>>>
>>> Best,
>>> Jack Ye
>>>
>>>
>>> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <ed...@dremio.com>
>>> wrote:
>>>
>>>> Nice work Jack, the proposal looks really good.
>>>>
>>>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Recently I have published PR 2961 - add snapshot tags interface (
>>>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>>>> great feedback. I have summarized everything in the discussions and put up
>>>>> a design to discuss the path forward around snapshot tagging, branching and
>>>>> retention:
>>>>>
>>>>>
>>>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>>>
>>>>> Any feedback around the doc would be much appreciated!
>>>>>
>>>>> Also, to facilitate future changes in Iceberg spec, it would be very
>>>>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>>>> https://github.com/apache/iceberg/pull/2957) which would make
>>>>> changing TableMetadata much simpler.
>>>>>
>>>>> Thanks,
>>>>> Jack Ye
>>>>>
>>>>
Re: Snapshot tagging, branching and retention
Posted by Jack Ye <ye...@gmail.com>.
Hi everyone,
Based on some offline discussions with different people around
availability, we will hold the meeting on Monday 10/18 9am PDT.
Here is the meeting link: meet.google.com/ubj-kvfm-ehg
I have added all the people in this thread to the invite. Feel free to also
forward the meeting to anyone else interested.
Best,
Jack Ye
On Mon, Oct 11, 2021 at 8:53 AM Eduard Tudenhoefner <ed...@dremio.com>
wrote:
> Hey Jack,
>
> would this week on Wednesday work for you from 9 to 10am PDT?
>
> On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> We have had a few iterations of the design doc with various people,
>> thanks for all the feedback. I am thinking about a meeting to finalize the
>> design and move forward with implementation.
>>
>> Considering the various time zones, I propose we choose any time from
>> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>>
>> If anyone is interested in joining, please let me know the preferred time
>> slot.
>>
>> Best,
>> Jack Ye
>>
>>
>> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <ed...@dremio.com>
>> wrote:
>>
>>> Nice work Jack, the proposal looks really good.
>>>
>>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Recently I have published PR 2961 - add snapshot tags interface (
>>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>>> great feedback. I have summarized everything in the discussions and put up
>>>> a design to discuss the path forward around snapshot tagging, branching and
>>>> retention:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>>
>>>> Any feedback around the doc would be much appreciated!
>>>>
>>>> Also, to facilitate future changes in Iceberg spec, it would be very
>>>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>>> https://github.com/apache/iceberg/pull/2957) which would make changing
>>>> TableMetadata much simpler.
>>>>
>>>> Thanks,
>>>> Jack Ye
>>>>
>>>
Re: Snapshot tagging, branching and retention
Posted by Eduard Tudenhoefner <ed...@dremio.com>.
Hey Jack,
would this week on Wednesday work for you from 9 to 10am PDT?
On Thu, Oct 7, 2021 at 7:41 PM Jack Ye <ye...@gmail.com> wrote:
> Hi everyone,
>
> We have had a few iterations of the design doc with various people, thanks
> for all the feedback. I am thinking about a meeting to finalize the design
> and move forward with implementation.
>
> Considering the various time zones, I propose we choose any time from
> Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
>
> If anyone is interested in joining, please let me know the preferred time
> slot.
>
> Best,
> Jack Ye
>
>
> On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
>> Nice work Jack, the proposal looks really good.
>>
>> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Recently I have published PR 2961 - add snapshot tags interface (
>>> https://github.com/apache/iceberg/pull/2961) and received a lot of
>>> great feedback. I have summarized everything in the discussions and put up
>>> a design to discuss the path forward around snapshot tagging, branching and
>>> retention:
>>>
>>>
>>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>>
>>> Any feedback around the doc would be much appreciated!
>>>
>>> Also, to facilitate future changes in Iceberg spec, it would be very
>>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>>> https://github.com/apache/iceberg/pull/2957) which would make changing
>>> TableMetadata much simpler.
>>>
>>> Thanks,
>>> Jack Ye
>>>
>>
Re: Snapshot tagging, branching and retention
Posted by Jack Ye <ye...@gmail.com>.
Hi everyone,
We have had a few iterations of the design doc with various people, thanks
for all the feedback. I am thinking about a meeting to finalize the design
and move forward with implementation.
Considering the various time zones, I propose we choose any time from
Tuesday (10/12) to Friday (10/15), 8-10am PDT, 1 hour meeting slot.
If anyone is interested in joining, please let me know the preferred time
slot.
Best,
Jack Ye
On Wed, Sep 15, 2021 at 11:29 PM Eduard Tudenhoefner <ed...@dremio.com>
wrote:
> Nice work Jack, the proposal looks really good.
>
> On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Recently I have published PR 2961 - add snapshot tags interface (
>> https://github.com/apache/iceberg/pull/2961) and received a lot of great
>> feedback. I have summarized everything in the discussions and put up a
>> design to discuss the path forward around snapshot tagging, branching and
>> retention:
>>
>>
>> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>>
>> Any feedback around the doc would be much appreciated!
>>
>> Also, to facilitate future changes in Iceberg spec, it would be very
>> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
>> https://github.com/apache/iceberg/pull/2957) which would make changing
>> TableMetadata much simpler.
>>
>> Thanks,
>> Jack Ye
>>
>
Re: Snapshot tagging, branching and retention
Posted by Eduard Tudenhoefner <ed...@dremio.com>.
Nice work Jack, the proposal looks really good.
On Sun, Aug 29, 2021 at 9:20 AM Jack Ye <ye...@gmail.com> wrote:
> Hi everyone,
>
> Recently I have published PR 2961 - add snapshot tags interface (
> https://github.com/apache/iceberg/pull/2961) and received a lot of great
> feedback. I have summarized everything in the discussions and put up a
> design to discuss the path forward around snapshot tagging, branching and
> retention:
>
>
> https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit?usp=sharing
>
> Any feedback around the doc would be much appreciated!
>
> Also, to facilitate future changes in Iceberg spec, it would be very
> helpful to take a look at 2597 - Core: introduce TableMetadataBuilder (
> https://github.com/apache/iceberg/pull/2957) which would make changing
> TableMetadata much simpler.
>
> Thanks,
> Jack Ye
>