You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Ryan Blue <rb...@netflix.com.INVALID> on 2021/01/28 22:14:41 UTC
Sync to discuss secondary index proposal
Hi everyone,
The proposal that Miao wrote about secondary indexes has come up a lot
lately. I think it would be a good time to have a discussion about the
proposal and set some initial goals for what we want to do next. Since
there hasn't been much discussion on the dev list, I'll schedule a sync so
that everyone has a deadline to read the proposal and be ready with
questions. Then we can have a quick summary to start with and a productive
discussion.
First, who is interested in attending? I think that Miao and I are in the
US in PST (UTC-8). I think Paula from IBM in Israel is interested. Anyone
else in a time zone that we should try to include? We can always have two
discussions if we need to include more zones.
Please reply if you're interested so we can get something set up. Thanks!
rb
--
Ryan Blue
Software Engineer
Netflix
Re: Sync to discuss secondary index proposal
Posted by Miao Wang <mi...@adobe.com.INVALID>.
Hi @OpenInx<ma...@gmail.com>,
The code change is based on our internal fork. We need to some refactoring before sending out an open source PR. In addition, since there is no spec defined in Iceberg, the implementation is coupled closely to our code base. That is one of the major reasons that I put my thoughts on building data format and compute engine agnostic into the draft.
Miao
From: OpenInx <op...@gmail.com>
Date: Thursday, January 28, 2021 at 6:31 PM
To: Iceberg Dev List <de...@iceberg.apache.org>, Miao Wang <mi...@adobe.com>
Cc: Ryan Blue <rb...@netflix.com>
Subject: Re: Sync to discuss secondary index proposal
Sorry I sent the wrong link, the secondary index document link is: https://docs.google.com/document/d/1E1ofBQoKRnX04bWT3utgyHQGaHZoelgXosk_UNsTUuQ/edit<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1E1ofBQoKRnX04bWT3utgyHQGaHZoelgXosk_UNsTUuQ%2Fedit&data=04%7C01%7Cmiwang%40adobe.com%7C7d97fc234caf481f4a5508d8c3fe0623%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637474843077023815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QkXoJydi3dCc6xWaQLjfcHsuA%2FZZlKsEXMRDoMEnUWE%3D&reserved=0>
On Fri, Jan 29, 2021 at 10:31 AM OpenInx <op...@gmail.com>> wrote:
Hi
@Miao Wang<ma...@adobe.com> Would you mind to share your current PoC code or PR for this document [1] if possible ? I'd like to understand more details before I get involved in this discussion.
Thanks.
[1]. https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit?ts=601316b0#<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM%2Fedit%3Fts%3D601316b0%23&data=04%7C01%7Cmiwang%40adobe.com%7C7d97fc234caf481f4a5508d8c3fe0623%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637474843077023815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hgiPjL%2B0IqQxCJ9W700snLDeBthvumjQcG0DUNk5tKM%3D&reserved=0>
On Fri, Jan 29, 2021 at 10:16 AM 李响 <wa...@gmail.com>> wrote:
+1, my colleagues and I is at UTC+8
On Fri, Jan 29, 2021 at 9:50 AM OpenInx <op...@gmail.com>> wrote:
+1, my time zone is CST.
On Fri, Jan 29, 2021 at 6:57 AM Xinli shang <sh...@uber.com.invalid> wrote:
I had some earlier discussion with Miao on this. I am still interested in it. My time zone is PST.
On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com>> wrote:
+1, looking forward to the discussion, please include me and Yan (yyanyyyy@gmail.com<ma...@gmail.com>), also in PST.
-Jack
On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <ru...@gmail.com>> wrote:
CST Please :) But I don’t mind waking up early or staying up late as required
On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID>> wrote:
Hi everyone,
The proposal that Miao wrote about secondary indexes has come up a lot lately. I think it would be a good time to have a discussion about the proposal and set some initial goals for what we want to do next. Since there hasn't been much discussion on the dev list, I'll schedule a sync so that everyone has a deadline to read the proposal and be ready with questions. Then we can have a quick summary to start with and a productive discussion.
First, who is interested in attending? I think that Miao and I are in the US in PST (UTC-8). I think Paula from IBM in Israel is interested. Anyone else in a time zone that we should try to include? We can always have two discussions if we need to include more zones.
Please reply if you're interested so we can get something set up. Thanks!
rb
--
Ryan Blue
Software Engineer
Netflix
--
Xinli Shang
--
李响 Xiang Li
手机 cellphone ：+86-136-8113-8972
邮件 e-mail ：waterlx@gmail.com<ma...@gmail.com>
Re: Sync to discuss secondary index proposal
Posted by OpenInx <op...@gmail.com>.
Sorry I sent the wrong link, the secondary index document link is:
https://docs.google.com/document/d/1E1ofBQoKRnX04bWT3utgyHQGaHZoelgXosk_UNsTUuQ/edit
On Fri, Jan 29, 2021 at 10:31 AM OpenInx <op...@gmail.com> wrote:
> Hi
>
> @Miao Wang <mi...@adobe.com> Would you mind to share your current PoC
> code or PR for this document [1] if possible ? I'd like to understand
> more details before I get involved in this discussion.
>
> Thanks.
>
> [1].
> https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit?ts=601316b0#
>
> On Fri, Jan 29, 2021 at 10:16 AM 李响 <wa...@gmail.com> wrote:
>
>> +1, my colleagues and I is at UTC+8
>>
>> On Fri, Jan 29, 2021 at 9:50 AM OpenInx <op...@gmail.com> wrote:
>>
>>> +1, my time zone is CST.
>>>
>>> On Fri, Jan 29, 2021 at 6:57 AM Xinli shang <sh...@uber.com.invalid>
>>> wrote:
>>>
>>>> I had some earlier discussion with Miao on this. I am still
>>>> interested in it. My time zone is PST.
>>>>
>>>> On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>>> +1, looking forward to the discussion, please include me and Yan (
>>>>> yyanyyyy@gmail.com), also in PST.
>>>>> -Jack
>>>>>
>>>>> On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <
>>>>> russell.spitzer@gmail.com> wrote:
>>>>>
>>>>>> CST Please :) But I don’t mind waking up early or staying up late as
>>>>>> required
>>>>>>
>>>>>> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID>
>>>>>> wrote:
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> The proposal that Miao wrote about secondary indexes has come up a
>>>>>> lot lately. I think it would be a good time to have a discussion about the
>>>>>> proposal and set some initial goals for what we want to do next. Since
>>>>>> there hasn't been much discussion on the dev list, I'll schedule a sync so
>>>>>> that everyone has a deadline to read the proposal and be ready with
>>>>>> questions. Then we can have a quick summary to start with and a productive
>>>>>> discussion.
>>>>>>
>>>>>> First, who is interested in attending? I think that Miao and I are in
>>>>>> the US in PST (UTC-8). I think Paula from IBM in Israel is interested.
>>>>>> Anyone else in a time zone that we should try to include? We can always
>>>>>> have two discussions if we need to include more zones.
>>>>>>
>>>>>> Please reply if you're interested so we can get something set up.
>>>>>> Thanks!
>>>>>>
>>>>>> rb
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Xinli Shang
>>>>
>>>
>>
>> --
>>
>> 李响 Xiang Li
>>
>> 手机 cellphone ：+86-136-8113-8972
>> 邮件 e-mail ：waterlx@gmail.com
>>
>
Re: Sync to discuss secondary index proposal
Posted by OpenInx <op...@gmail.com>.
Hi
@Miao Wang <mi...@adobe.com> Would you mind to share your current PoC
code or PR for this document [1] if possible ? I'd like to understand
more details before I get involved in this discussion.
Thanks.
[1].
https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit?ts=601316b0#
On Fri, Jan 29, 2021 at 10:16 AM 李响 <wa...@gmail.com> wrote:
> +1, my colleagues and I is at UTC+8
>
> On Fri, Jan 29, 2021 at 9:50 AM OpenInx <op...@gmail.com> wrote:
>
>> +1, my time zone is CST.
>>
>> On Fri, Jan 29, 2021 at 6:57 AM Xinli shang <sh...@uber.com.invalid>
>> wrote:
>>
>>> I had some earlier discussion with Miao on this. I am still
>>> interested in it. My time zone is PST.
>>>
>>> On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> +1, looking forward to the discussion, please include me and Yan (
>>>> yyanyyyy@gmail.com), also in PST.
>>>> -Jack
>>>>
>>>> On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <
>>>> russell.spitzer@gmail.com> wrote:
>>>>
>>>>> CST Please :) But I don’t mind waking up early or staying up late as
>>>>> required
>>>>>
>>>>> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID>
>>>>> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> The proposal that Miao wrote about secondary indexes has come up a lot
>>>>> lately. I think it would be a good time to have a discussion about the
>>>>> proposal and set some initial goals for what we want to do next. Since
>>>>> there hasn't been much discussion on the dev list, I'll schedule a sync so
>>>>> that everyone has a deadline to read the proposal and be ready with
>>>>> questions. Then we can have a quick summary to start with and a productive
>>>>> discussion.
>>>>>
>>>>> First, who is interested in attending? I think that Miao and I are in
>>>>> the US in PST (UTC-8). I think Paula from IBM in Israel is interested.
>>>>> Anyone else in a time zone that we should try to include? We can always
>>>>> have two discussions if we need to include more zones.
>>>>>
>>>>> Please reply if you're interested so we can get something set up.
>>>>> Thanks!
>>>>>
>>>>> rb
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Xinli Shang
>>>
>>
>
> --
>
> 李响 Xiang Li
>
> 手机 cellphone ：+86-136-8113-8972
> 邮件 e-mail ：waterlx@gmail.com
>
Re: Sync to discuss secondary index proposal
Posted by 李响 <wa...@gmail.com>.
+1, my colleagues and I is at UTC+8
On Fri, Jan 29, 2021 at 9:50 AM OpenInx <op...@gmail.com> wrote:
> +1, my time zone is CST.
>
> On Fri, Jan 29, 2021 at 6:57 AM Xinli shang <sh...@uber.com.invalid>
> wrote:
>
>> I had some earlier discussion with Miao on this. I am still interested in
>> it. My time zone is PST.
>>
>> On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com> wrote:
>>
>>> +1, looking forward to the discussion, please include me and Yan (
>>> yyanyyyy@gmail.com), also in PST.
>>> -Jack
>>>
>>> On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>>> CST Please :) But I don’t mind waking up early or staying up late as
>>>> required
>>>>
>>>> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID>
>>>> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> The proposal that Miao wrote about secondary indexes has come up a lot
>>>> lately. I think it would be a good time to have a discussion about the
>>>> proposal and set some initial goals for what we want to do next. Since
>>>> there hasn't been much discussion on the dev list, I'll schedule a sync so
>>>> that everyone has a deadline to read the proposal and be ready with
>>>> questions. Then we can have a quick summary to start with and a productive
>>>> discussion.
>>>>
>>>> First, who is interested in attending? I think that Miao and I are in
>>>> the US in PST (UTC-8). I think Paula from IBM in Israel is interested.
>>>> Anyone else in a time zone that we should try to include? We can always
>>>> have two discussions if we need to include more zones.
>>>>
>>>> Please reply if you're interested so we can get something set up.
>>>> Thanks!
>>>>
>>>> rb
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>>
>>>>
>>
>> --
>> Xinli Shang
>>
>
--
李响 Xiang Li
手机 cellphone ：+86-136-8113-8972
邮件 e-mail ：waterlx@gmail.com
Re: Sync to discuss secondary index proposal
Posted by OpenInx <op...@gmail.com>.
+1, my time zone is CST.
On Fri, Jan 29, 2021 at 6:57 AM Xinli shang <sh...@uber.com.invalid> wrote:
> I had some earlier discussion with Miao on this. I am still interested in
> it. My time zone is PST.
>
> On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com> wrote:
>
>> +1, looking forward to the discussion, please include me and Yan (
>> yyanyyyy@gmail.com), also in PST.
>> -Jack
>>
>> On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> CST Please :) But I don’t mind waking up early or staying up late as
>>> required
>>>
>>> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID>
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>> The proposal that Miao wrote about secondary indexes has come up a lot
>>> lately. I think it would be a good time to have a discussion about the
>>> proposal and set some initial goals for what we want to do next. Since
>>> there hasn't been much discussion on the dev list, I'll schedule a sync so
>>> that everyone has a deadline to read the proposal and be ready with
>>> questions. Then we can have a quick summary to start with and a productive
>>> discussion.
>>>
>>> First, who is interested in attending? I think that Miao and I are in
>>> the US in PST (UTC-8). I think Paula from IBM in Israel is interested.
>>> Anyone else in a time zone that we should try to include? We can always
>>> have two discussions if we need to include more zones.
>>>
>>> Please reply if you're interested so we can get something set up. Thanks!
>>>
>>> rb
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>>
>>>
>
> --
> Xinli Shang
>
Re: Sync to discuss secondary index proposal
Posted by Xinli shang <sh...@uber.com.INVALID>.
I had some earlier discussion with Miao on this. I am still interested in
it. My time zone is PST.
On Thu, Jan 28, 2021 at 2:50 PM Jack Ye <ye...@gmail.com> wrote:
> +1, looking forward to the discussion, please include me and Yan (
> yyanyyyy@gmail.com), also in PST.
> -Jack
>
> On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> CST Please :) But I don’t mind waking up early or staying up late as
>> required
>>
>> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>>
>> Hi everyone,
>>
>> The proposal that Miao wrote about secondary indexes has come up a lot
>> lately. I think it would be a good time to have a discussion about the
>> proposal and set some initial goals for what we want to do next. Since
>> there hasn't been much discussion on the dev list, I'll schedule a sync so
>> that everyone has a deadline to read the proposal and be ready with
>> questions. Then we can have a quick summary to start with and a productive
>> discussion.
>>
>> First, who is interested in attending? I think that Miao and I are in the
>> US in PST (UTC-8). I think Paula from IBM in Israel is interested. Anyone
>> else in a time zone that we should try to include? We can always have two
>> discussions if we need to include more zones.
>>
>> Please reply if you're interested so we can get something set up. Thanks!
>>
>> rb
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>>
>>
--
Xinli Shang
Re: Sync to discuss secondary index proposal
Posted by Jack Ye <ye...@gmail.com>.
+1, looking forward to the discussion, please include me and Yan (
yyanyyyy@gmail.com), also in PST.
-Jack
On Thu, Jan 28, 2021 at 2:16 PM Russell Spitzer <ru...@gmail.com>
wrote:
> CST Please :) But I don’t mind waking up early or staying up late as
> required
>
> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> Hi everyone,
>
> The proposal that Miao wrote about secondary indexes has come up a lot
> lately. I think it would be a good time to have a discussion about the
> proposal and set some initial goals for what we want to do next. Since
> there hasn't been much discussion on the dev list, I'll schedule a sync so
> that everyone has a deadline to read the proposal and be ready with
> questions. Then we can have a quick summary to start with and a productive
> discussion.
>
> First, who is interested in attending? I think that Miao and I are in the
> US in PST (UTC-8). I think Paula from IBM in Israel is interested. Anyone
> else in a time zone that we should try to include? We can always have two
> discussions if we need to include more zones.
>
> Please reply if you're interested so we can get something set up. Thanks!
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>
Re: Sync to discuss secondary index proposal
Posted by Russell Spitzer <ru...@gmail.com>.
CST Please :) But I don’t mind waking up early or staying up late as required
> On Jan 28, 2021, at 4:14 PM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> Hi everyone,
>
> The proposal that Miao wrote about secondary indexes has come up a lot lately. I think it would be a good time to have a discussion about the proposal and set some initial goals for what we want to do next. Since there hasn't been much discussion on the dev list, I'll schedule a sync so that everyone has a deadline to read the proposal and be ready with questions. Then we can have a quick summary to start with and a productive discussion.
>
> First, who is interested in attending? I think that Miao and I are in the US in PST (UTC-8). I think Paula from IBM in Israel is interested. Anyone else in a time zone that we should try to include? We can always have two discussions if we need to include more zones.
>
> Please reply if you're interested so we can get something set up. Thanks!
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Netflix