[DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Hi everyone,

At the last sync meeting, we brought up publishing a community roadmap and
brainstormed the many features and initiatives that the community is
working on. In this thread, I want to make sure that we have a good list of
what people are thinking about and I think we should try to categorize the
projects by size and general priority. When we reach a rough agreement,
I’ll write this up and post it on the ASF site along with links to some
projects in Github.

My rationale for attempting to prioritize projects is that if we try to do
too many things, it will be slower progress across everything rather than
getting a few important items done. I know that priorities don’t align very
cleanly in practice, but it is hopefully worth trying. To come up with a
priority, I’m trying to keep top priority items to a minimum by including
only one from each group (Spark, Flink, Python, etc.). The remaining items
are split between priority 2 and 3. Priority 3 is not urgent, including
things that can be plugged in (like other IO libraries), docs, etc.
Everything else is priority 2.

That something isn’t priority 1 doesn’t mean it isn’t important or
progressing, just that it isn’t the current focus. I think of it this way:
if someone has extra time to review something, what should be next? That’s
top priority.

Here’s my rough categorization. If you disagree, please speak up:

   - If you think that something should be top priority, what gets moved to
   priority 2?
   - Should the priority for a project in 2 or 3 change?
   - Is the S/M/L size of a project wrong?

Top priority, 1:

   - API: Iceberg 1.0 [medium]
   - Spark: Merge-on-read plans [large]
   - Maintenance: Delete file compaction [medium]
   -

   Flink: Upgrade to 1.13.2 (document compatibility) [medium]
   -

   Python: Pythonic refactor [medium]

Priority 2:

   - ORC: Support delete files stored as ORC [small]
   - Spark: DSv2 streaming improvements [small]
   - Flink: Inline file compaction [small]
   - Flink: Support UPSERT [small]
   - Views: Spec [medium]
   - Spec: Z-ordering / Space-filling curves [medium]
   - Spec: Snapshot tagging and branching [small]
   - Spec: Secondary indexes [large]
   - Spec v3: Encryption [large]
   -

   Spec v3: Relative paths [large]
   -

   Spec v3: Default field values [medium]

Priority 3:

   - Docs: versioned docs [medium]
   - IO: Support Aliyun OSS/DLF [medium]
   - IO: Support Dell ECS [medium]

External:

   - Trino: Bucketed joins [small]
   - Trino: Row-level delete support [medium]
   - Trino: Merge-on-read plans [medium]
   - Trino: Multi-catalog support [small]

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[OpenInx <op...@gmail.com>]

Update:

As the Dell EMC EcsFileIO has been merged into apache iceberg
official repo, so I think it's okay to get this project from roadmap closed
now: https://github.com/apache/iceberg/projects/22

Thanks.

On Wed, Nov 10, 2021 at 10:22 AM Zhao Chun <zh...@apache.org> wrote:

> Thanks Ryan.
> We will keep a close eye on what is happening in the iceberg community and
> seek help when necessary.
>
> Thanks,
> Zhao Chun
>
>
> Ryan Blue <bl...@tabular.io> 于2021年11月10日周三 上午8:54写道：
>
>> Thanks, Zhao. I think those are great ways to work together. Let us know
>> how we can help you make StarRocks successful with Iceberg as its data
>> format. We're always happy to help people understand how Iceberg works and
>> improve our docs on how to use it.
>>
>> Ryan
>>
>> On Mon, Nov 8, 2021 at 8:17 PM Zhao Chun <zh...@apache.org> wrote:
>>
>>> I feel that Ryan's response exemplifies the generosity of an Apache
>>> project creator,
>>> a quality that has touched and benefited us. We look forward to
>>> contributing
>>> further to the Apache project in the future.
>>> As for the need for an issue to track progress,I don't think so for now.
>>> At the moment the main development work is done in the StarRocks
>>> repository.
>>> As for further cooperation in the future, I think there are several
>>> aspects.
>>> 1. StarRocks will be trying to support Iceberg.
>>> I think this will help StarRocks to re-examine how it integrates with
>>> the lakehouse system
>>> and we will be happy to feed back to the Apache Iceberg community the
>>> issues and benefits
>>> we encounter during the integration process.
>>> This will also validate the versatility of the iceberg project to
>>> support more query engines.
>>> I think this project will benefit both projects.
>>> 2. In the future, we will share some of our best practices for iceberg
>>> and StarRocks integration in a blog or talk.
>>> If the Apache Iceberg project feels that these blogs or talks would be
>>> beneficial to the Apache iceberg community,
>>> please consider linking our subsequent blogs or talks to the apache
>>> iceberg website blog.
>>> The Iceberg community can, of course, not link if they feel it is
>>> inappropriate.
>>> 3. we expect to contribute to the Apache Iceberg community under the
>>> Apache License V2.
>>>
>>> Thanks,
>>> Zhao Chun
>>>
>>>
>>> Ryan Blue <bl...@tabular.io> 于2021年11月9日周二 上午3:05写道：
>>>
>>>> I think it is great to see another processing engine adding support for
>>>> Apache Iceberg, and I do look forward to collaborating with the StarRocks
>>>> community in the future.
>>>>
>>>> I'm not entirely sure what that collaboration would look like just yet
>>>> though. For most processing engines, it is people joining the Apache
>>>> Iceberg community. No matter what the license of the downstream project, we
>>>> always welcome more people contributing here!
>>>>
>>>> As for opening a project in our tracker, I'm not sure it makes sense to
>>>> do that just yet. As far as I know there aren't any issues to track there.
>>>> And would the StarRocks community find it helpful?
>>>>
>>>> On Mon, Nov 8, 2021 at 12:14 AM Zhao Chun <bu...@gmail.com> wrote:
>>>>
>>>>> Thanks to @OpenInx for mentioning StarRocks in the iceberg community.
>>>>>
>>>>> I'm from the StarRocks community.
>>>>>
>>>>> StarRocks is based on the Apache Doris project.
>>>>> It has been in development internally for almost two years and is
>>>>> currently used by hundreds of companies.
>>>>> It was just opened 2 months ago.
>>>>>
>>>>> Iceberg is a great project that makes huge datasets analysis more
>>>>> convenient.
>>>>> The StarRocks community is planning to support the iceberg engine.
>>>>> This will provide StarRocks users with the ability to analyze data in
>>>>> iceberg.
>>>>>
>>>>> Regarding the license, StarRocks' ELv2 will not affect our
>>>>> contribution to the iceberg community under the Apache License V2.
>>>>>
>>>>> We are also looking forward to receiving help from the iceberg
>>>>> community and will be contributing back to the iceberg community.
>>>>>
>>>>> Thanks,
>>>>> Zhao Chun
>>>>>
>>>>>
>>>>> Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：
>>>>>
>>>>>> +1 around concerns with the Elastic license.
>>>>>>
>>>>>> Also, more importantly, how important is integration with either of
>>>>>> these tools to the Iceberg community and contributors?
>>>>>>
>>>>>> The Elastic license makes a bit more sense for elasticsearch, as it
>>>>>> was an existing project for quite some time. I won’t reiterate the details
>>>>>> of that situation, but it’s odd to see a fork of a new, active project
>>>>>> using the Elastic license in my opinion.
>>>>>>
>>>>>> StarRocks admits that they’re at least 40% of code from the Apache
>>>>>> Doris project.
>>>>>>
>>>>>> That said, StarRocks claims to not require other dependencies. It
>>>>>> seems StarRocks supports query federation with a few tools so as not to
>>>>>> have to import the data and query those systems directly. So I’m not sure
>>>>>> what Iceberg support would look like beyond additional query federation.
>>>>>> What benefit does this provide?
>>>>>>
>>>>>> If we determined that integration with one of these tools was
>>>>>> something the community valued, could a connector be built to target the
>>>>>> Apache Doris project and then StarRocks could fork that code if they liked?
>>>>>>
>>>>>> - Kyle Bendickson
>>>>>> GitHub @kbendick
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>>>>>>
>>>>>>> +1, I have the same concern for the incompatible license.
>>>>>>>
>>>>>>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>>>>>>
>>>>>>>> A few additional observations about StarRocks...
>>>>>>>>
>>>>>>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>>>>>>> (Elastic License 2.0).
>>>>>>>> - It appears to be a hard fork of Apache Doris, a project still in
>>>>>>>> the incubator (and looks like it probably is destructive to the Doris
>>>>>>>> project)
>>>>>>>> - The project has only existed for ~2 months.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>>>>>>
>>>>>>>>> I think the guys from StarRocks community can provide more
>>>>>>>>> background and inputs.
>>>>>>>>>
>>>>>>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Update:
>>>>>>>>>>
>>>>>>>>>> StarRocks[1] is a next-gen sub-second MPP database for full
>>>>>>>>>> analysis scenarios, including multi-dimensional analytics, real-time
>>>>>>>>>> analytics and ad-hoc query.  Their team is planning to integrate iceberg
>>>>>>>>>> tables as StarRocks external tables in the next month [2], so that people
>>>>>>>>>> could connect the data lake and StarRocks warehouse in the same engine.
>>>>>>>>>> The excellent performance of StarRocks will also help accelerate
>>>>>>>>>> the analysis and access of the iceberg table, I think this is a great thing
>>>>>>>>>> for both the iceberg community and the StarRocks community.   I think we
>>>>>>>>>> can add an extra project about StarRocks integration work in the apache
>>>>>>>>>> iceberg roadmap [3] ?
>>>>>>>>>>
>>>>>>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>>>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>>>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I closed the upgrade project and marked the FLIP-27 project
>>>>>>>>>>> priority 1. Thanks for all the work to get this done!
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Update:
>>>>>>>>>>>>
>>>>>>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap
>>>>>>>>>>>> can be closed now, because all of the issues have been addressed.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I created a Roadmap section in
>>>>>>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to
>>>>>>>>>>>>> the planning boards that Jack created. I figured it makes sense if we link
>>>>>>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, Jack!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap
>>>>>>>>>>>>>> page as well that links to the projects that Jack just created.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems like we have reached some consensus around the
>>>>>>>>>>>>>>> projects listed here. I have created corresponding Github projects for
>>>>>>>>>>>>>>> each: https://github.com/apache/iceberg/projects
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Would it make sense to have a section on the website where
>>>>>>>>>>>>>>>> we collect all the links to the design docs/specs as that would be easier
>>>>>>>>>>>>>>>> to find than searching for things on the ML?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At the last sync meeting, we brought up publishing a
>>>>>>>>>>>>>>>>> community roadmap and brainstormed the many features and initiatives that
>>>>>>>>>>>>>>>>> the community is working on. In this thread, I want to make sure that we
>>>>>>>>>>>>>>>>> have a good list of what people are thinking about and I think we should
>>>>>>>>>>>>>>>>> try to categorize the projects by size and general priority. When we reach
>>>>>>>>>>>>>>>>> a rough agreement, I’ll write this up and post it on the ASF site along
>>>>>>>>>>>>>>>>> with links to some projects in Github.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My rationale for attempting to prioritize projects is that
>>>>>>>>>>>>>>>>> if we try to do too many things, it will be slower progress across
>>>>>>>>>>>>>>>>> everything rather than getting a few important items done. I know that
>>>>>>>>>>>>>>>>> priorities don’t align very cleanly in practice, but it is hopefully worth
>>>>>>>>>>>>>>>>> trying. To come up with a priority, I’m trying to keep top priority items
>>>>>>>>>>>>>>>>> to a minimum by including only one from each group (Spark, Flink, Python,
>>>>>>>>>>>>>>>>> etc.). The remaining items are split between priority 2 and 3. Priority 3
>>>>>>>>>>>>>>>>> is not urgent, including things that can be plugged in (like other IO
>>>>>>>>>>>>>>>>> libraries), docs, etc. Everything else is priority 2.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t
>>>>>>>>>>>>>>>>> important or progressing, just that it isn’t the current focus. I think of
>>>>>>>>>>>>>>>>> it this way: if someone has extra time to review something, what should be
>>>>>>>>>>>>>>>>> next? That’s top priority.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please
>>>>>>>>>>>>>>>>> speak up:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - If you think that something should be top priority,
>>>>>>>>>>>>>>>>>    what gets moved to priority 2?
>>>>>>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility)
>>>>>>>>>>>>>>>>>    [medium]
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> External:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Tabular
>>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[Zhao Chun <zh...@apache.org>]

Thanks Ryan.
We will keep a close eye on what is happening in the iceberg community and
seek help when necessary.

Thanks,
Zhao Chun


Ryan Blue <bl...@tabular.io> 于2021年11月10日周三 上午8:54写道：

> Thanks, Zhao. I think those are great ways to work together. Let us know
> how we can help you make StarRocks successful with Iceberg as its data
> format. We're always happy to help people understand how Iceberg works and
> improve our docs on how to use it.
>
> Ryan
>
> On Mon, Nov 8, 2021 at 8:17 PM Zhao Chun <zh...@apache.org> wrote:
>
>> I feel that Ryan's response exemplifies the generosity of an Apache
>> project creator,
>> a quality that has touched and benefited us. We look forward to
>> contributing
>> further to the Apache project in the future.
>> As for the need for an issue to track progress,I don't think so for now.
>> At the moment the main development work is done in the StarRocks
>> repository.
>> As for further cooperation in the future, I think there are several
>> aspects.
>> 1. StarRocks will be trying to support Iceberg.
>> I think this will help StarRocks to re-examine how it integrates with the
>> lakehouse system
>> and we will be happy to feed back to the Apache Iceberg community the
>> issues and benefits
>> we encounter during the integration process.
>> This will also validate the versatility of the iceberg project to support
>> more query engines.
>> I think this project will benefit both projects.
>> 2. In the future, we will share some of our best practices for iceberg
>> and StarRocks integration in a blog or talk.
>> If the Apache Iceberg project feels that these blogs or talks would be
>> beneficial to the Apache iceberg community,
>> please consider linking our subsequent blogs or talks to the apache
>> iceberg website blog.
>> The Iceberg community can, of course, not link if they feel it is
>> inappropriate.
>> 3. we expect to contribute to the Apache Iceberg community under the
>> Apache License V2.
>>
>> Thanks,
>> Zhao Chun
>>
>>
>> Ryan Blue <bl...@tabular.io> 于2021年11月9日周二 上午3:05写道：
>>
>>> I think it is great to see another processing engine adding support for
>>> Apache Iceberg, and I do look forward to collaborating with the StarRocks
>>> community in the future.
>>>
>>> I'm not entirely sure what that collaboration would look like just yet
>>> though. For most processing engines, it is people joining the Apache
>>> Iceberg community. No matter what the license of the downstream project, we
>>> always welcome more people contributing here!
>>>
>>> As for opening a project in our tracker, I'm not sure it makes sense to
>>> do that just yet. As far as I know there aren't any issues to track there.
>>> And would the StarRocks community find it helpful?
>>>
>>> On Mon, Nov 8, 2021 at 12:14 AM Zhao Chun <bu...@gmail.com> wrote:
>>>
>>>> Thanks to @OpenInx for mentioning StarRocks in the iceberg community.
>>>>
>>>> I'm from the StarRocks community.
>>>>
>>>> StarRocks is based on the Apache Doris project.
>>>> It has been in development internally for almost two years and is
>>>> currently used by hundreds of companies.
>>>> It was just opened 2 months ago.
>>>>
>>>> Iceberg is a great project that makes huge datasets analysis more
>>>> convenient.
>>>> The StarRocks community is planning to support the iceberg engine.
>>>> This will provide StarRocks users with the ability to analyze data in
>>>> iceberg.
>>>>
>>>> Regarding the license, StarRocks' ELv2 will not affect our contribution
>>>> to the iceberg community under the Apache License V2.
>>>>
>>>> We are also looking forward to receiving help from the iceberg
>>>> community and will be contributing back to the iceberg community.
>>>>
>>>> Thanks,
>>>> Zhao Chun
>>>>
>>>>
>>>> Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：
>>>>
>>>>> +1 around concerns with the Elastic license.
>>>>>
>>>>> Also, more importantly, how important is integration with either of
>>>>> these tools to the Iceberg community and contributors?
>>>>>
>>>>> The Elastic license makes a bit more sense for elasticsearch, as it
>>>>> was an existing project for quite some time. I won’t reiterate the details
>>>>> of that situation, but it’s odd to see a fork of a new, active project
>>>>> using the Elastic license in my opinion.
>>>>>
>>>>> StarRocks admits that they’re at least 40% of code from the Apache
>>>>> Doris project.
>>>>>
>>>>> That said, StarRocks claims to not require other dependencies. It
>>>>> seems StarRocks supports query federation with a few tools so as not to
>>>>> have to import the data and query those systems directly. So I’m not sure
>>>>> what Iceberg support would look like beyond additional query federation.
>>>>> What benefit does this provide?
>>>>>
>>>>> If we determined that integration with one of these tools was
>>>>> something the community valued, could a connector be built to target the
>>>>> Apache Doris project and then StarRocks could fork that code if they liked?
>>>>>
>>>>> - Kyle Bendickson
>>>>> GitHub @kbendick
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>>>>>
>>>>>> +1, I have the same concern for the incompatible license.
>>>>>>
>>>>>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>>>>>
>>>>>>> A few additional observations about StarRocks...
>>>>>>>
>>>>>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>>>>>> (Elastic License 2.0).
>>>>>>> - It appears to be a hard fork of Apache Doris, a project still in
>>>>>>> the incubator (and looks like it probably is destructive to the Doris
>>>>>>> project)
>>>>>>> - The project has only existed for ~2 months.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>>>>>
>>>>>>>> I think the guys from StarRocks community can provide more
>>>>>>>> background and inputs.
>>>>>>>>
>>>>>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Update:
>>>>>>>>>
>>>>>>>>> StarRocks[1] is a next-gen sub-second MPP database for full
>>>>>>>>> analysis scenarios, including multi-dimensional analytics, real-time
>>>>>>>>> analytics and ad-hoc query.  Their team is planning to integrate iceberg
>>>>>>>>> tables as StarRocks external tables in the next month [2], so that people
>>>>>>>>> could connect the data lake and StarRocks warehouse in the same engine.
>>>>>>>>> The excellent performance of StarRocks will also help accelerate
>>>>>>>>> the analysis and access of the iceberg table, I think this is a great thing
>>>>>>>>> for both the iceberg community and the StarRocks community.   I think we
>>>>>>>>> can add an extra project about StarRocks integration work in the apache
>>>>>>>>> iceberg roadmap [3] ?
>>>>>>>>>
>>>>>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>>>>>
>>>>>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>>>
>>>>>>>>>> I closed the upgrade project and marked the FLIP-27 project
>>>>>>>>>> priority 1. Thanks for all the work to get this done!
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Update:
>>>>>>>>>>>
>>>>>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap
>>>>>>>>>>> can be closed now, because all of the issues have been addressed.
>>>>>>>>>>>
>>>>>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I created a Roadmap section in
>>>>>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to
>>>>>>>>>>>> the planning boards that Jack created. I figured it makes sense if we link
>>>>>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, Jack!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap
>>>>>>>>>>>>> page as well that links to the projects that Jack just created.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> It seems like we have reached some consensus around the
>>>>>>>>>>>>>> projects listed here. I have created corresponding Github projects for
>>>>>>>>>>>>>> each: https://github.com/apache/iceberg/projects
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Would it make sense to have a section on the website where
>>>>>>>>>>>>>>> we collect all the links to the design docs/specs as that would be easier
>>>>>>>>>>>>>>> to find than searching for things on the ML?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At the last sync meeting, we brought up publishing a
>>>>>>>>>>>>>>>> community roadmap and brainstormed the many features and initiatives that
>>>>>>>>>>>>>>>> the community is working on. In this thread, I want to make sure that we
>>>>>>>>>>>>>>>> have a good list of what people are thinking about and I think we should
>>>>>>>>>>>>>>>> try to categorize the projects by size and general priority. When we reach
>>>>>>>>>>>>>>>> a rough agreement, I’ll write this up and post it on the ASF site along
>>>>>>>>>>>>>>>> with links to some projects in Github.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My rationale for attempting to prioritize projects is that
>>>>>>>>>>>>>>>> if we try to do too many things, it will be slower progress across
>>>>>>>>>>>>>>>> everything rather than getting a few important items done. I know that
>>>>>>>>>>>>>>>> priorities don’t align very cleanly in practice, but it is hopefully worth
>>>>>>>>>>>>>>>> trying. To come up with a priority, I’m trying to keep top priority items
>>>>>>>>>>>>>>>> to a minimum by including only one from each group (Spark, Flink, Python,
>>>>>>>>>>>>>>>> etc.). The remaining items are split between priority 2 and 3. Priority 3
>>>>>>>>>>>>>>>> is not urgent, including things that can be plugged in (like other IO
>>>>>>>>>>>>>>>> libraries), docs, etc. Everything else is priority 2.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t
>>>>>>>>>>>>>>>> important or progressing, just that it isn’t the current focus. I think of
>>>>>>>>>>>>>>>> it this way: if someone has extra time to review something, what should be
>>>>>>>>>>>>>>>> next? That’s top priority.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please
>>>>>>>>>>>>>>>> speak up:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - If you think that something should be top priority,
>>>>>>>>>>>>>>>>    what gets moved to priority 2?
>>>>>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility)
>>>>>>>>>>>>>>>>    [medium]
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> External:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Tabular
>>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Thanks, Zhao. I think those are great ways to work together. Let us know
how we can help you make StarRocks successful with Iceberg as its data
format. We're always happy to help people understand how Iceberg works and
improve our docs on how to use it.

Ryan

On Mon, Nov 8, 2021 at 8:17 PM Zhao Chun <zh...@apache.org> wrote:

> I feel that Ryan's response exemplifies the generosity of an Apache
> project creator,
> a quality that has touched and benefited us. We look forward to
> contributing
> further to the Apache project in the future.
> As for the need for an issue to track progress,I don't think so for now.
> At the moment the main development work is done in the StarRocks
> repository.
> As for further cooperation in the future, I think there are several
> aspects.
> 1. StarRocks will be trying to support Iceberg.
> I think this will help StarRocks to re-examine how it integrates with the
> lakehouse system
> and we will be happy to feed back to the Apache Iceberg community the
> issues and benefits
> we encounter during the integration process.
> This will also validate the versatility of the iceberg project to support
> more query engines.
> I think this project will benefit both projects.
> 2. In the future, we will share some of our best practices for iceberg and
> StarRocks integration in a blog or talk.
> If the Apache Iceberg project feels that these blogs or talks would be
> beneficial to the Apache iceberg community,
> please consider linking our subsequent blogs or talks to the apache
> iceberg website blog.
> The Iceberg community can, of course, not link if they feel it is
> inappropriate.
> 3. we expect to contribute to the Apache Iceberg community under the
> Apache License V2.
>
> Thanks,
> Zhao Chun
>
>
> Ryan Blue <bl...@tabular.io> 于2021年11月9日周二 上午3:05写道：
>
>> I think it is great to see another processing engine adding support for
>> Apache Iceberg, and I do look forward to collaborating with the StarRocks
>> community in the future.
>>
>> I'm not entirely sure what that collaboration would look like just yet
>> though. For most processing engines, it is people joining the Apache
>> Iceberg community. No matter what the license of the downstream project, we
>> always welcome more people contributing here!
>>
>> As for opening a project in our tracker, I'm not sure it makes sense to
>> do that just yet. As far as I know there aren't any issues to track there.
>> And would the StarRocks community find it helpful?
>>
>> On Mon, Nov 8, 2021 at 12:14 AM Zhao Chun <bu...@gmail.com> wrote:
>>
>>> Thanks to @OpenInx for mentioning StarRocks in the iceberg community.
>>>
>>> I'm from the StarRocks community.
>>>
>>> StarRocks is based on the Apache Doris project.
>>> It has been in development internally for almost two years and is
>>> currently used by hundreds of companies.
>>> It was just opened 2 months ago.
>>>
>>> Iceberg is a great project that makes huge datasets analysis more
>>> convenient.
>>> The StarRocks community is planning to support the iceberg engine.
>>> This will provide StarRocks users with the ability to analyze data in
>>> iceberg.
>>>
>>> Regarding the license, StarRocks' ELv2 will not affect our contribution
>>> to the iceberg community under the Apache License V2.
>>>
>>> We are also looking forward to receiving help from the iceberg community
>>> and will be contributing back to the iceberg community.
>>>
>>> Thanks,
>>> Zhao Chun
>>>
>>>
>>> Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：
>>>
>>>> +1 around concerns with the Elastic license.
>>>>
>>>> Also, more importantly, how important is integration with either of
>>>> these tools to the Iceberg community and contributors?
>>>>
>>>> The Elastic license makes a bit more sense for elasticsearch, as it was
>>>> an existing project for quite some time. I won’t reiterate the details of
>>>> that situation, but it’s odd to see a fork of a new, active project using
>>>> the Elastic license in my opinion.
>>>>
>>>> StarRocks admits that they’re at least 40% of code from the Apache
>>>> Doris project.
>>>>
>>>> That said, StarRocks claims to not require other dependencies. It seems
>>>> StarRocks supports query federation with a few tools so as not to have to
>>>> import the data and query those systems directly. So I’m not sure what
>>>> Iceberg support would look like beyond additional query federation. What
>>>> benefit does this provide?
>>>>
>>>> If we determined that integration with one of these tools was something
>>>> the community valued, could a connector be built to target the Apache Doris
>>>> project and then StarRocks could fork that code if they liked?
>>>>
>>>> - Kyle Bendickson
>>>> GitHub @kbendick
>>>>
>>>>
>>>>
>>>> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>>>>
>>>>> +1, I have the same concern for the incompatible license.
>>>>>
>>>>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>>>>
>>>>>> A few additional observations about StarRocks...
>>>>>>
>>>>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>>>>> (Elastic License 2.0).
>>>>>> - It appears to be a hard fork of Apache Doris, a project still in
>>>>>> the incubator (and looks like it probably is destructive to the Doris
>>>>>> project)
>>>>>> - The project has only existed for ~2 months.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>>>>
>>>>>>> I think the guys from StarRocks community can provide more
>>>>>>> background and inputs.
>>>>>>>
>>>>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Update:
>>>>>>>>
>>>>>>>> StarRocks[1] is a next-gen sub-second MPP database for full
>>>>>>>> analysis scenarios, including multi-dimensional analytics, real-time
>>>>>>>> analytics and ad-hoc query.  Their team is planning to integrate iceberg
>>>>>>>> tables as StarRocks external tables in the next month [2], so that people
>>>>>>>> could connect the data lake and StarRocks warehouse in the same engine.
>>>>>>>> The excellent performance of StarRocks will also help accelerate
>>>>>>>> the analysis and access of the iceberg table, I think this is a great thing
>>>>>>>> for both the iceberg community and the StarRocks community.   I think we
>>>>>>>> can add an extra project about StarRocks integration work in the apache
>>>>>>>> iceberg roadmap [3] ?
>>>>>>>>
>>>>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>>>>
>>>>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>>
>>>>>>>>> I closed the upgrade project and marked the FLIP-27 project
>>>>>>>>> priority 1. Thanks for all the work to get this done!
>>>>>>>>>
>>>>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Update:
>>>>>>>>>>
>>>>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can
>>>>>>>>>> be closed now, because all of the issues have been addressed.
>>>>>>>>>>
>>>>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I created a Roadmap section in
>>>>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Jack!
>>>>>>>>>>>>
>>>>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap
>>>>>>>>>>>> page as well that links to the projects that Jack just created.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> It seems like we have reached some consensus around the
>>>>>>>>>>>>> projects listed here. I have created corresponding Github projects for
>>>>>>>>>>>>> each: https://github.com/apache/iceberg/projects
>>>>>>>>>>>>>
>>>>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At the last sync meeting, we brought up publishing a
>>>>>>>>>>>>>>> community roadmap and brainstormed the many features and initiatives that
>>>>>>>>>>>>>>> the community is working on. In this thread, I want to make sure that we
>>>>>>>>>>>>>>> have a good list of what people are thinking about and I think we should
>>>>>>>>>>>>>>> try to categorize the projects by size and general priority. When we reach
>>>>>>>>>>>>>>> a rough agreement, I’ll write this up and post it on the ASF site along
>>>>>>>>>>>>>>> with links to some projects in Github.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My rationale for attempting to prioritize projects is that
>>>>>>>>>>>>>>> if we try to do too many things, it will be slower progress across
>>>>>>>>>>>>>>> everything rather than getting a few important items done. I know that
>>>>>>>>>>>>>>> priorities don’t align very cleanly in practice, but it is hopefully worth
>>>>>>>>>>>>>>> trying. To come up with a priority, I’m trying to keep top priority items
>>>>>>>>>>>>>>> to a minimum by including only one from each group (Spark, Flink, Python,
>>>>>>>>>>>>>>> etc.). The remaining items are split between priority 2 and 3. Priority 3
>>>>>>>>>>>>>>> is not urgent, including things that can be plugged in (like other IO
>>>>>>>>>>>>>>> libraries), docs, etc. Everything else is priority 2.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t
>>>>>>>>>>>>>>> important or progressing, just that it isn’t the current focus. I think of
>>>>>>>>>>>>>>> it this way: if someone has extra time to review something, what should be
>>>>>>>>>>>>>>> next? That’s top priority.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please
>>>>>>>>>>>>>>> speak up:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - If you think that something should be top priority,
>>>>>>>>>>>>>>>    what gets moved to priority 2?
>>>>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility)
>>>>>>>>>>>>>>>    [medium]
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> External:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> Tabular
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Tabular
>>>>>>>>>
>>>>>>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Zhao Chun <zh...@apache.org>]

I feel that Ryan's response exemplifies the generosity of an Apache project
creator,
a quality that has touched and benefited us. We look forward to
contributing
further to the Apache project in the future.
As for the need for an issue to track progress,I don't think so for now.
At the moment the main development work is done in the StarRocks
repository.
As for further cooperation in the future, I think there are several
aspects.
1. StarRocks will be trying to support Iceberg.
I think this will help StarRocks to re-examine how it integrates with the
lakehouse system
and we will be happy to feed back to the Apache Iceberg community the
issues and benefits
we encounter during the integration process.
This will also validate the versatility of the iceberg project to support
more query engines.
I think this project will benefit both projects.
2. In the future, we will share some of our best practices for iceberg and
StarRocks integration in a blog or talk.
If the Apache Iceberg project feels that these blogs or talks would be
beneficial to the Apache iceberg community,
please consider linking our subsequent blogs or talks to the apache iceberg
website blog.
The Iceberg community can, of course, not link if they feel it is
inappropriate.
3. we expect to contribute to the Apache Iceberg community under the Apache
License V2.

Thanks,
Zhao Chun


Ryan Blue <bl...@tabular.io> 于2021年11月9日周二 上午3:05写道：

> I think it is great to see another processing engine adding support for
> Apache Iceberg, and I do look forward to collaborating with the StarRocks
> community in the future.
>
> I'm not entirely sure what that collaboration would look like just yet
> though. For most processing engines, it is people joining the Apache
> Iceberg community. No matter what the license of the downstream project, we
> always welcome more people contributing here!
>
> As for opening a project in our tracker, I'm not sure it makes sense to do
> that just yet. As far as I know there aren't any issues to track there. And
> would the StarRocks community find it helpful?
>
> On Mon, Nov 8, 2021 at 12:14 AM Zhao Chun <bu...@gmail.com> wrote:
>
>> Thanks to @OpenInx for mentioning StarRocks in the iceberg community.
>>
>> I'm from the StarRocks community.
>>
>> StarRocks is based on the Apache Doris project.
>> It has been in development internally for almost two years and is
>> currently used by hundreds of companies.
>> It was just opened 2 months ago.
>>
>> Iceberg is a great project that makes huge datasets analysis more
>> convenient.
>> The StarRocks community is planning to support the iceberg engine.
>> This will provide StarRocks users with the ability to analyze data in
>> iceberg.
>>
>> Regarding the license, StarRocks' ELv2 will not affect our contribution
>> to the iceberg community under the Apache License V2.
>>
>> We are also looking forward to receiving help from the iceberg community
>> and will be contributing back to the iceberg community.
>>
>> Thanks,
>> Zhao Chun
>>
>>
>> Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：
>>
>>> +1 around concerns with the Elastic license.
>>>
>>> Also, more importantly, how important is integration with either of
>>> these tools to the Iceberg community and contributors?
>>>
>>> The Elastic license makes a bit more sense for elasticsearch, as it was
>>> an existing project for quite some time. I won’t reiterate the details of
>>> that situation, but it’s odd to see a fork of a new, active project using
>>> the Elastic license in my opinion.
>>>
>>> StarRocks admits that they’re at least 40% of code from the Apache Doris
>>> project.
>>>
>>> That said, StarRocks claims to not require other dependencies. It seems
>>> StarRocks supports query federation with a few tools so as not to have to
>>> import the data and query those systems directly. So I’m not sure what
>>> Iceberg support would look like beyond additional query federation. What
>>> benefit does this provide?
>>>
>>> If we determined that integration with one of these tools was something
>>> the community valued, could a connector be built to target the Apache Doris
>>> project and then StarRocks could fork that code if they liked?
>>>
>>> - Kyle Bendickson
>>> GitHub @kbendick
>>>
>>>
>>>
>>> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>>>
>>>> +1, I have the same concern for the incompatible license.
>>>>
>>>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>>>
>>>>> A few additional observations about StarRocks...
>>>>>
>>>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>>>> (Elastic License 2.0).
>>>>> - It appears to be a hard fork of Apache Doris, a project still in the
>>>>> incubator (and looks like it probably is destructive to the Doris project)
>>>>> - The project has only existed for ~2 months.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>>>
>>>>>> I think the guys from StarRocks community can provide more background
>>>>>> and inputs.
>>>>>>
>>>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> Update:
>>>>>>>
>>>>>>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>>>>>>> scenarios, including multi-dimensional analytics, real-time analytics and
>>>>>>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>>>>>>> StarRocks external tables in the next month [2], so that people could
>>>>>>> connect the data lake and StarRocks warehouse in the same engine.
>>>>>>> The excellent performance of StarRocks will also help accelerate the
>>>>>>> analysis and access of the iceberg table, I think this is a great thing for
>>>>>>> both the iceberg community and the StarRocks community.   I think we can
>>>>>>> add an extra project about StarRocks integration work in the apache iceberg
>>>>>>> roadmap [3] ?
>>>>>>>
>>>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>>>
>>>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>
>>>>>>>> I closed the upgrade project and marked the FLIP-27 project
>>>>>>>> priority 1. Thanks for all the work to get this done!
>>>>>>>>
>>>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Update:
>>>>>>>>>
>>>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can
>>>>>>>>> be closed now, because all of the issues have been addressed.
>>>>>>>>>
>>>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>>>
>>>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>
>>>>>>>>>> I created a Roadmap section in
>>>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, Jack!
>>>>>>>>>>>
>>>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap
>>>>>>>>>>> page as well that links to the projects that Jack just created.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> It seems like we have reached some consensus around the
>>>>>>>>>>>> projects listed here. I have created corresponding Github projects for
>>>>>>>>>>>> each: https://github.com/apache/iceberg/projects
>>>>>>>>>>>>
>>>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At the last sync meeting, we brought up publishing a
>>>>>>>>>>>>>> community roadmap and brainstormed the many features and initiatives that
>>>>>>>>>>>>>> the community is working on. In this thread, I want to make sure that we
>>>>>>>>>>>>>> have a good list of what people are thinking about and I think we should
>>>>>>>>>>>>>> try to categorize the projects by size and general priority. When we reach
>>>>>>>>>>>>>> a rough agreement, I’ll write this up and post it on the ASF site along
>>>>>>>>>>>>>> with links to some projects in Github.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My rationale for attempting to prioritize projects is that if
>>>>>>>>>>>>>> we try to do too many things, it will be slower progress across everything
>>>>>>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t
>>>>>>>>>>>>>> important or progressing, just that it isn’t the current focus. I think of
>>>>>>>>>>>>>> it this way: if someone has extra time to review something, what should be
>>>>>>>>>>>>>> next? That’s top priority.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please speak
>>>>>>>>>>>>>> up:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - If you think that something should be top priority,
>>>>>>>>>>>>>>    what gets moved to priority 2?
>>>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> External:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Tabular
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

I think it is great to see another processing engine adding support for
Apache Iceberg, and I do look forward to collaborating with the StarRocks
community in the future.

I'm not entirely sure what that collaboration would look like just yet
though. For most processing engines, it is people joining the Apache
Iceberg community. No matter what the license of the downstream project, we
always welcome more people contributing here!

As for opening a project in our tracker, I'm not sure it makes sense to do
that just yet. As far as I know there aren't any issues to track there. And
would the StarRocks community find it helpful?

On Mon, Nov 8, 2021 at 12:14 AM Zhao Chun <bu...@gmail.com> wrote:

> Thanks to @OpenInx for mentioning StarRocks in the iceberg community.
>
> I'm from the StarRocks community.
>
> StarRocks is based on the Apache Doris project.
> It has been in development internally for almost two years and is
> currently used by hundreds of companies.
> It was just opened 2 months ago.
>
> Iceberg is a great project that makes huge datasets analysis more
> convenient.
> The StarRocks community is planning to support the iceberg engine.
> This will provide StarRocks users with the ability to analyze data in
> iceberg.
>
> Regarding the license, StarRocks' ELv2 will not affect our contribution to
> the iceberg community under the Apache License V2.
>
> We are also looking forward to receiving help from the iceberg community
> and will be contributing back to the iceberg community.
>
> Thanks,
> Zhao Chun
>
>
> Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：
>
>> +1 around concerns with the Elastic license.
>>
>> Also, more importantly, how important is integration with either of these
>> tools to the Iceberg community and contributors?
>>
>> The Elastic license makes a bit more sense for elasticsearch, as it was
>> an existing project for quite some time. I won’t reiterate the details of
>> that situation, but it’s odd to see a fork of a new, active project using
>> the Elastic license in my opinion.
>>
>> StarRocks admits that they’re at least 40% of code from the Apache Doris
>> project.
>>
>> That said, StarRocks claims to not require other dependencies. It seems
>> StarRocks supports query federation with a few tools so as not to have to
>> import the data and query those systems directly. So I’m not sure what
>> Iceberg support would look like beyond additional query federation. What
>> benefit does this provide?
>>
>> If we determined that integration with one of these tools was something
>> the community valued, could a connector be built to target the Apache Doris
>> project and then StarRocks could fork that code if they liked?
>>
>> - Kyle Bendickson
>> GitHub @kbendick
>>
>>
>>
>> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>>
>>> +1, I have the same concern for the incompatible license.
>>>
>>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>>
>>>> A few additional observations about StarRocks...
>>>>
>>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>>> (Elastic License 2.0).
>>>> - It appears to be a hard fork of Apache Doris, a project still in the
>>>> incubator (and looks like it probably is destructive to the Doris project)
>>>> - The project has only existed for ~2 months.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>>
>>>>> I think the guys from StarRocks community can provide more background
>>>>> and inputs.
>>>>>
>>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> Update:
>>>>>>
>>>>>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>>>>>> scenarios, including multi-dimensional analytics, real-time analytics and
>>>>>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>>>>>> StarRocks external tables in the next month [2], so that people could
>>>>>> connect the data lake and StarRocks warehouse in the same engine.
>>>>>> The excellent performance of StarRocks will also help accelerate the
>>>>>> analysis and access of the iceberg table, I think this is a great thing for
>>>>>> both the iceberg community and the StarRocks community.   I think we can
>>>>>> add an extra project about StarRocks integration work in the apache iceberg
>>>>>> roadmap [3] ?
>>>>>>
>>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>>
>>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>
>>>>>>> I closed the upgrade project and marked the FLIP-27 project priority
>>>>>>> 1. Thanks for all the work to get this done!
>>>>>>>
>>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Update:
>>>>>>>>
>>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can
>>>>>>>> be closed now, because all of the issues have been addressed.
>>>>>>>>
>>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>>
>>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>
>>>>>>>>> I created a Roadmap section in
>>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>>
>>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, Jack!
>>>>>>>>>>
>>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap page
>>>>>>>>>> as well that links to the projects that Jack just created.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> It seems like we have reached some consensus around the projects
>>>>>>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>>>>>>> https://github.com/apache/iceberg/projects
>>>>>>>>>>>
>>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Jack Ye
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>>>>
>>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>>
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>>>>>>> to some projects in Github.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My rationale for attempting to prioritize projects is that if
>>>>>>>>>>>>> we try to do too many things, it will be slower progress across everything
>>>>>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t
>>>>>>>>>>>>> important or progressing, just that it isn’t the current focus. I think of
>>>>>>>>>>>>> it this way: if someone has extra time to review something, what should be
>>>>>>>>>>>>> next? That’s top priority.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please speak
>>>>>>>>>>>>> up:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - If you think that something should be top priority, what
>>>>>>>>>>>>>    gets moved to priority 2?
>>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>>
>>>>>>>>>>>>> External:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Tabular
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Tabular
>>>>>>>
>>>>>>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Zhao Chun <bu...@gmail.com>]

Thanks to @OpenInx for mentioning StarRocks in the iceberg community.

I'm from the StarRocks community.

StarRocks is based on the Apache Doris project.
It has been in development internally for almost two years and is currently
used by hundreds of companies.
It was just opened 2 months ago.

Iceberg is a great project that makes huge datasets analysis more
convenient.
The StarRocks community is planning to support the iceberg engine.
This will provide StarRocks users with the ability to analyze data in
iceberg.

Regarding the license, StarRocks' ELv2 will not affect our contribution to
the iceberg community under the Apache License V2.

We are also looking forward to receiving help from the iceberg community
and will be contributing back to the iceberg community.

Thanks,
Zhao Chun


Kyle Bendickson <ky...@tabular.io> 于2021年11月8日周一 下午2:53写道：

> +1 around concerns with the Elastic license.
>
> Also, more importantly, how important is integration with either of these
> tools to the Iceberg community and contributors?
>
> The Elastic license makes a bit more sense for elasticsearch, as it was an
> existing project for quite some time. I won’t reiterate the details of that
> situation, but it’s odd to see a fork of a new, active project using the
> Elastic license in my opinion.
>
> StarRocks admits that they’re at least 40% of code from the Apache Doris
> project.
>
> That said, StarRocks claims to not require other dependencies. It seems
> StarRocks supports query federation with a few tools so as not to have to
> import the data and query those systems directly. So I’m not sure what
> Iceberg support would look like beyond additional query federation. What
> benefit does this provide?
>
> If we determined that integration with one of these tools was something
> the community valued, could a connector be built to target the Apache Doris
> project and then StarRocks could fork that code if they liked?
>
> - Kyle Bendickson
> GitHub @kbendick
>
>
>
> On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:
>
>> +1, I have the same concern for the incompatible license.
>>
>> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>>
>>> A few additional observations about StarRocks...
>>>
>>> - As far as I can tell, StarRocks has an ASF incompatible license
>>> (Elastic License 2.0).
>>> - It appears to be a hard fork of Apache Doris, a project still in the
>>> incubator (and looks like it probably is destructive to the Doris project)
>>> - The project has only existed for ~2 months.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>>
>>>> I think the guys from StarRocks community can provide more background
>>>> and inputs.
>>>>
>>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Update:
>>>>>
>>>>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>>>>> scenarios, including multi-dimensional analytics, real-time analytics and
>>>>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>>>>> StarRocks external tables in the next month [2], so that people could
>>>>> connect the data lake and StarRocks warehouse in the same engine.
>>>>> The excellent performance of StarRocks will also help accelerate the
>>>>> analysis and access of the iceberg table, I think this is a great thing for
>>>>> both the iceberg community and the StarRocks community.   I think we can
>>>>> add an extra project about StarRocks integration work in the apache iceberg
>>>>> roadmap [3] ?
>>>>>
>>>>> [1].  https://github.com/StarRocks/starrocks
>>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>>> [3].  https://github.com/apache/iceberg/projects
>>>>>
>>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>
>>>>>> I closed the upgrade project and marked the FLIP-27 project priority
>>>>>> 1. Thanks for all the work to get this done!
>>>>>>
>>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> Update:
>>>>>>>
>>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>>>>>>> closed now, because all of the issues have been addressed.
>>>>>>>
>>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>>
>>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>>> eduard@dremio.com> wrote:
>>>>>>>
>>>>>>>> I created a Roadmap section in
>>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>>
>>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>>
>>>>>>>>> Thanks, Jack!
>>>>>>>>>
>>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap page
>>>>>>>>> as well that links to the projects that Jack just created.
>>>>>>>>>
>>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It seems like we have reached some consensus around the projects
>>>>>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>>>>>> https://github.com/apache/iceberg/projects
>>>>>>>>>>
>>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jack Ye
>>>>>>>>>>
>>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>>>
>>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>>> * link to the ML discussion
>>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>>>>>> to some projects in Github.
>>>>>>>>>>>>
>>>>>>>>>>>> My rationale for attempting to prioritize projects is that if
>>>>>>>>>>>> we try to do too many things, it will be slower progress across everything
>>>>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>>>>
>>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important
>>>>>>>>>>>> or progressing, just that it isn’t the current focus. I think of it this
>>>>>>>>>>>> way: if someone has extra time to review something, what should be next?
>>>>>>>>>>>> That’s top priority.
>>>>>>>>>>>>
>>>>>>>>>>>> Here’s my rough categorization. If you disagree, please speak
>>>>>>>>>>>> up:
>>>>>>>>>>>>
>>>>>>>>>>>>    - If you think that something should be top priority, what
>>>>>>>>>>>>    gets moved to priority 2?
>>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>>
>>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>>
>>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>>
>>>>>>>>>>>> Priority 2:
>>>>>>>>>>>>
>>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>>
>>>>>>>>>>>> Priority 3:
>>>>>>>>>>>>
>>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>>
>>>>>>>>>>>> External:
>>>>>>>>>>>>
>>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> Tabular
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Tabular
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Tabular
>>>>>>
>>>>>

Re: [DISCUSS] Iceberg roadmap

[Kyle Bendickson <ky...@tabular.io>]

+1 around concerns with the Elastic license.

Also, more importantly, how important is integration with either of these
tools to the Iceberg community and contributors?

The Elastic license makes a bit more sense for elasticsearch, as it was an
existing project for quite some time. I won’t reiterate the details of that
situation, but it’s odd to see a fork of a new, active project using the
Elastic license in my opinion.

StarRocks admits that they’re at least 40% of code from the Apache Doris
project.

That said, StarRocks claims to not require other dependencies. It seems
StarRocks supports query federation with a few tools so as not to have to
import the data and query those systems directly. So I’m not sure what
Iceberg support would look like beyond additional query federation. What
benefit does this provide?

If we determined that integration with one of these tools was something the
community valued, could a connector be built to target the Apache Doris
project and then StarRocks could fork that code if they liked?

- Kyle Bendickson
GitHub @kbendick



On Sun, Nov 7, 2021 at 9:24 PM Reo Lei <le...@gmail.com> wrote:

> +1, I have the same concern for the incompatible license.
>
> Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：
>
>> A few additional observations about StarRocks...
>>
>> - As far as I can tell, StarRocks has an ASF incompatible license
>> (Elastic License 2.0).
>> - It appears to be a hard fork of Apache Doris, a project still in the
>> incubator (and looks like it probably is destructive to the Doris project)
>> - The project has only existed for ~2 months.
>>
>>
>>
>>
>>
>> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Any thoughts for adding StarRocks integration to the roadmap ?
>>>
>>> I think the guys from StarRocks community can provide more background
>>> and inputs.
>>>
>>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Update:
>>>>
>>>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>>>> scenarios, including multi-dimensional analytics, real-time analytics and
>>>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>>>> StarRocks external tables in the next month [2], so that people could
>>>> connect the data lake and StarRocks warehouse in the same engine.
>>>> The excellent performance of StarRocks will also help accelerate the
>>>> analysis and access of the iceberg table, I think this is a great thing for
>>>> both the iceberg community and the StarRocks community.   I think we can
>>>> add an extra project about StarRocks integration work in the apache iceberg
>>>> roadmap [3] ?
>>>>
>>>> [1].  https://github.com/StarRocks/starrocks
>>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>>> [3].  https://github.com/apache/iceberg/projects
>>>>
>>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>
>>>>> I closed the upgrade project and marked the FLIP-27 project priority
>>>>> 1. Thanks for all the work to get this done!
>>>>>
>>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> Update:
>>>>>>
>>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>>>>>> closed now, because all of the issues have been addressed.
>>>>>>
>>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>>
>>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <
>>>>>> eduard@dremio.com> wrote:
>>>>>>
>>>>>>> I created a Roadmap section in
>>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>>
>>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>
>>>>>>>> Thanks, Jack!
>>>>>>>>
>>>>>>>> Eduard, I think that's a good idea. We should have a roadmap page
>>>>>>>> as well that links to the projects that Jack just created.
>>>>>>>>
>>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It seems like we have reached some consensus around the projects
>>>>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>>>>> https://github.com/apache/iceberg/projects
>>>>>>>>>
>>>>>>>>> Related design docs are also linked there.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Jack Ye
>>>>>>>>>
>>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>>
>>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>>
>>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>>> * link to the ML discussion
>>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>>>>> to some projects in Github.
>>>>>>>>>>>
>>>>>>>>>>> My rationale for attempting to prioritize projects is that if we
>>>>>>>>>>> try to do too many things, it will be slower progress across everything
>>>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>>>
>>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important
>>>>>>>>>>> or progressing, just that it isn’t the current focus. I think of it this
>>>>>>>>>>> way: if someone has extra time to review something, what should be next?
>>>>>>>>>>> That’s top priority.
>>>>>>>>>>>
>>>>>>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>>>>>>
>>>>>>>>>>>    - If you think that something should be top priority, what
>>>>>>>>>>>    gets moved to priority 2?
>>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>>
>>>>>>>>>>> Top priority, 1:
>>>>>>>>>>>
>>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>>
>>>>>>>>>>> Priority 2:
>>>>>>>>>>>
>>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>>
>>>>>>>>>>> Priority 3:
>>>>>>>>>>>
>>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>>
>>>>>>>>>>> External:
>>>>>>>>>>>
>>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Tabular
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Tabular
>>>>>
>>>>

Re: [DISCUSS] Iceberg roadmap

[Reo Lei <le...@gmail.com>]

+1, I have the same concern for the incompatible license.

Jacques Nadeau <ja...@gmail.com> 于2021年11月8日周一 上午11:48写道：

> A few additional observations about StarRocks...
>
> - As far as I can tell, StarRocks has an ASF incompatible license (Elastic
> License 2.0).
> - It appears to be a hard fork of Apache Doris, a project still in the
> incubator (and looks like it probably is destructive to the Doris project)
> - The project has only existed for ~2 months.
>
>
>
>
>
> On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:
>
>> Any thoughts for adding StarRocks integration to the roadmap ?
>>
>> I think the guys from StarRocks community can provide more background and
>> inputs.
>>
>> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Update:
>>>
>>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>>> scenarios, including multi-dimensional analytics, real-time analytics and
>>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>>> StarRocks external tables in the next month [2], so that people could
>>> connect the data lake and StarRocks warehouse in the same engine.
>>> The excellent performance of StarRocks will also help accelerate the
>>> analysis and access of the iceberg table, I think this is a great thing for
>>> both the iceberg community and the StarRocks community.   I think we can
>>> add an extra project about StarRocks integration work in the apache iceberg
>>> roadmap [3] ?
>>>
>>> [1].  https://github.com/StarRocks/starrocks
>>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>>> [3].  https://github.com/apache/iceberg/projects
>>>
>>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> I closed the upgrade project and marked the FLIP-27 project priority 1.
>>>> Thanks for all the work to get this done!
>>>>
>>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Update:
>>>>>
>>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>>>>> closed now, because all of the issues have been addressed.
>>>>>
>>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>>
>>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>>> wrote:
>>>>>
>>>>>> I created a Roadmap section in
>>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>>> available Design Docs directly on those Boards (as was already done),
>>>>>> because then the Design docs are closer to the set of related issues.
>>>>>>
>>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>
>>>>>>> Thanks, Jack!
>>>>>>>
>>>>>>> Eduard, I think that's a good idea. We should have a roadmap page as
>>>>>>> well that links to the projects that Jack just created.
>>>>>>>
>>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It seems like we have reached some consensus around the projects
>>>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>>>> https://github.com/apache/iceberg/projects
>>>>>>>>
>>>>>>>> Related design docs are also linked there.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jack Ye
>>>>>>>>
>>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>>> eduard@dremio.com> wrote:
>>>>>>>>
>>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>>> find than searching for things on the ML?
>>>>>>>>>
>>>>>>>>> I was thinking about something like for each component:
>>>>>>>>> * link to the ML discussion
>>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>>>> to some projects in Github.
>>>>>>>>>>
>>>>>>>>>> My rationale for attempting to prioritize projects is that if we
>>>>>>>>>> try to do too many things, it will be slower progress across everything
>>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>>
>>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important
>>>>>>>>>> or progressing, just that it isn’t the current focus. I think of it this
>>>>>>>>>> way: if someone has extra time to review something, what should be next?
>>>>>>>>>> That’s top priority.
>>>>>>>>>>
>>>>>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>>>>>
>>>>>>>>>>    - If you think that something should be top priority, what
>>>>>>>>>>    gets moved to priority 2?
>>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>>
>>>>>>>>>> Top priority, 1:
>>>>>>>>>>
>>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>>
>>>>>>>>>> Priority 2:
>>>>>>>>>>
>>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>>
>>>>>>>>>> Priority 3:
>>>>>>>>>>
>>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>>
>>>>>>>>>> External:
>>>>>>>>>>
>>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Tabular
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Tabular
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>

Re: [DISCUSS] Iceberg roadmap

[Jacques Nadeau <ja...@gmail.com>]

A few additional observations about StarRocks...

- As far as I can tell, StarRocks has an ASF incompatible license (Elastic
License 2.0).
- It appears to be a hard fork of Apache Doris, a project still in the
incubator (and looks like it probably is destructive to the Doris project)
- The project has only existed for ~2 months.





On Sun, Nov 7, 2021 at 7:34 PM OpenInx <op...@gmail.com> wrote:

> Any thoughts for adding StarRocks integration to the roadmap ?
>
> I think the guys from StarRocks community can provide more background and
> inputs.
>
> On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:
>
>> Update:
>>
>> StarRocks[1] is a next-gen sub-second MPP database for full analysis
>> scenarios, including multi-dimensional analytics, real-time analytics and
>> ad-hoc query.  Their team is planning to integrate iceberg tables as
>> StarRocks external tables in the next month [2], so that people could
>> connect the data lake and StarRocks warehouse in the same engine.
>> The excellent performance of StarRocks will also help accelerate the
>> analysis and access of the iceberg table, I think this is a great thing for
>> both the iceberg community and the StarRocks community.   I think we can
>> add an extra project about StarRocks integration work in the apache iceberg
>> roadmap [3] ?
>>
>> [1].  https://github.com/StarRocks/starrocks
>> [2].  https://github.com/StarRocks/starrocks/issues/1030
>> [3].  https://github.com/apache/iceberg/projects
>>
>> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> I closed the upgrade project and marked the FLIP-27 project priority 1.
>>> Thanks for all the work to get this done!
>>>
>>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Update:
>>>>
>>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>>>> closed now, because all of the issues have been addressed.
>>>>
>>>> [1]. https://github.com/apache/iceberg/projects/12
>>>>
>>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>> wrote:
>>>>
>>>>> I created a Roadmap section in
>>>>>  https://github.com/apache/iceberg/pull/3163
>>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>>> planning boards that Jack created. I figured it makes sense if we link
>>>>> available Design Docs directly on those Boards (as was already done),
>>>>> because then the Design docs are closer to the set of related issues.
>>>>>
>>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>
>>>>>> Thanks, Jack!
>>>>>>
>>>>>> Eduard, I think that's a good idea. We should have a roadmap page as
>>>>>> well that links to the projects that Jack just created.
>>>>>>
>>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>>>>>>
>>>>>>> It seems like we have reached some consensus around the projects
>>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>>> https://github.com/apache/iceberg/projects
>>>>>>>
>>>>>>> Related design docs are also linked there.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jack Ye
>>>>>>>
>>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>>> eduard@dremio.com> wrote:
>>>>>>>
>>>>>>>> Would it make sense to have a section on the website where we
>>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>>> find than searching for things on the ML?
>>>>>>>>
>>>>>>>> I was thinking about something like for each component:
>>>>>>>> * link to the ML discussion
>>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>>> to some projects in Github.
>>>>>>>>>
>>>>>>>>> My rationale for attempting to prioritize projects is that if we
>>>>>>>>> try to do too many things, it will be slower progress across everything
>>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>>
>>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>>>>>> if someone has extra time to review something, what should be next? That’s
>>>>>>>>> top priority.
>>>>>>>>>
>>>>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>>>>
>>>>>>>>>    - If you think that something should be top priority, what
>>>>>>>>>    gets moved to priority 2?
>>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>>
>>>>>>>>> Top priority, 1:
>>>>>>>>>
>>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>>
>>>>>>>>> Priority 2:
>>>>>>>>>
>>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>>    - Views: Spec [medium]
>>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>>
>>>>>>>>> Priority 3:
>>>>>>>>>
>>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>>
>>>>>>>>> External:
>>>>>>>>>
>>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Tabular
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Tabular
>>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>

Re: [DISCUSS] Iceberg roadmap

[OpenInx <op...@gmail.com>]

Any thoughts for adding StarRocks integration to the roadmap ?

I think the guys from StarRocks community can provide more background and
inputs.

On Thu, Nov 4, 2021 at 5:59 PM OpenInx <op...@gmail.com> wrote:

> Update:
>
> StarRocks[1] is a next-gen sub-second MPP database for full analysis
> scenarios, including multi-dimensional analytics, real-time analytics and
> ad-hoc query.  Their team is planning to integrate iceberg tables as
> StarRocks external tables in the next month [2], so that people could
> connect the data lake and StarRocks warehouse in the same engine.
> The excellent performance of StarRocks will also help accelerate the
> analysis and access of the iceberg table, I think this is a great thing for
> both the iceberg community and the StarRocks community.   I think we can
> add an extra project about StarRocks integration work in the apache iceberg
> roadmap [3] ?
>
> [1].  https://github.com/StarRocks/starrocks
> [2].  https://github.com/StarRocks/starrocks/issues/1030
> [3].  https://github.com/apache/iceberg/projects
>
> On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> I closed the upgrade project and marked the FLIP-27 project priority 1.
>> Thanks for all the work to get this done!
>>
>> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Update:
>>>
>>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>>> closed now, because all of the issues have been addressed.
>>>
>>> [1]. https://github.com/apache/iceberg/projects/12
>>>
>>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
>>> wrote:
>>>
>>>> I created a Roadmap section in
>>>>  https://github.com/apache/iceberg/pull/3163
>>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>>> planning boards that Jack created. I figured it makes sense if we link
>>>> available Design Docs directly on those Boards (as was already done),
>>>> because then the Design docs are closer to the set of related issues.
>>>>
>>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>
>>>>> Thanks, Jack!
>>>>>
>>>>> Eduard, I think that's a good idea. We should have a roadmap page as
>>>>> well that links to the projects that Jack just created.
>>>>>
>>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>>>>>
>>>>>> It seems like we have reached some consensus around the projects
>>>>>> listed here. I have created corresponding Github projects for each:
>>>>>> https://github.com/apache/iceberg/projects
>>>>>>
>>>>>> Related design docs are also linked there.
>>>>>>
>>>>>> Best,
>>>>>> Jack Ye
>>>>>>
>>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>>> eduard@dremio.com> wrote:
>>>>>>
>>>>>>> Would it make sense to have a section on the website where we
>>>>>>> collect all the links to the design docs/specs as that would be easier to
>>>>>>> find than searching for things on the ML?
>>>>>>>
>>>>>>> I was thinking about something like for each component:
>>>>>>> * link to the ML discussion
>>>>>>> * link to the actual Spec/Design Doc
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>>> to some projects in Github.
>>>>>>>>
>>>>>>>> My rationale for attempting to prioritize projects is that if we
>>>>>>>> try to do too many things, it will be slower progress across everything
>>>>>>>> rather than getting a few important items done. I know that priorities
>>>>>>>> don’t align very cleanly in practice, but it is hopefully worth trying. To
>>>>>>>> come up with a priority, I’m trying to keep top priority items to a minimum
>>>>>>>> by including only one from each group (Spark, Flink, Python, etc.). The
>>>>>>>> remaining items are split between priority 2 and 3. Priority 3 is not
>>>>>>>> urgent, including things that can be plugged in (like other IO libraries),
>>>>>>>> docs, etc. Everything else is priority 2.
>>>>>>>>
>>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>>>>> if someone has extra time to review something, what should be next? That’s
>>>>>>>> top priority.
>>>>>>>>
>>>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>>>
>>>>>>>>    - If you think that something should be top priority, what gets
>>>>>>>>    moved to priority 2?
>>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>>
>>>>>>>> Top priority, 1:
>>>>>>>>
>>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>>
>>>>>>>> Priority 2:
>>>>>>>>
>>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>>    - Views: Spec [medium]
>>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>>
>>>>>>>> Priority 3:
>>>>>>>>
>>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>>
>>>>>>>> External:
>>>>>>>>
>>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Tabular
>>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[OpenInx <op...@gmail.com>]

Update:

StarRocks[1] is a next-gen sub-second MPP database for full analysis
scenarios, including multi-dimensional analytics, real-time analytics and
ad-hoc query.  Their team is planning to integrate iceberg tables as
StarRocks external tables in the next month [2], so that people could
connect the data lake and StarRocks warehouse in the same engine.
The excellent performance of StarRocks will also help accelerate the
analysis and access of the iceberg table, I think this is a great thing for
both the iceberg community and the StarRocks community.   I think we can
add an extra project about StarRocks integration work in the apache iceberg
roadmap [3] ?

[1].  https://github.com/StarRocks/starrocks
[2].  https://github.com/StarRocks/starrocks/issues/1030
[3].  https://github.com/apache/iceberg/projects

On Mon, Nov 1, 2021 at 11:52 PM Ryan Blue <bl...@tabular.io> wrote:

> I closed the upgrade project and marked the FLIP-27 project priority 1.
> Thanks for all the work to get this done!
>
> On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:
>
>> Update:
>>
>> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
>> closed now, because all of the issues have been addressed.
>>
>> [1]. https://github.com/apache/iceberg/projects/12
>>
>> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
>> wrote:
>>
>>> I created a Roadmap section in
>>>  https://github.com/apache/iceberg/pull/3163
>>> <https://github.com/apache/iceberg/pull/3163> that links to the
>>> planning boards that Jack created. I figured it makes sense if we link
>>> available Design Docs directly on those Boards (as was already done),
>>> because then the Design docs are closer to the set of related issues.
>>>
>>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> Thanks, Jack!
>>>>
>>>> Eduard, I think that's a good idea. We should have a roadmap page as
>>>> well that links to the projects that Jack just created.
>>>>
>>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>>> It seems like we have reached some consensus around the projects
>>>>> listed here. I have created corresponding Github projects for each:
>>>>> https://github.com/apache/iceberg/projects
>>>>>
>>>>> Related design docs are also linked there.
>>>>>
>>>>> Best,
>>>>> Jack Ye
>>>>>
>>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <
>>>>> eduard@dremio.com> wrote:
>>>>>
>>>>>> Would it make sense to have a section on the website where we collect
>>>>>> all the links to the design docs/specs as that would be easier to find than
>>>>>> searching for things on the ML?
>>>>>>
>>>>>> I was thinking about something like for each component:
>>>>>> * link to the ML discussion
>>>>>> * link to the actual Spec/Design Doc
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>>> good list of what people are thinking about and I think we should try to
>>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>>> to some projects in Github.
>>>>>>>
>>>>>>> My rationale for attempting to prioritize projects is that if we try
>>>>>>> to do too many things, it will be slower progress across everything rather
>>>>>>> than getting a few important items done. I know that priorities don’t align
>>>>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>>>>> Everything else is priority 2.
>>>>>>>
>>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>>>> if someone has extra time to review something, what should be next? That’s
>>>>>>> top priority.
>>>>>>>
>>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>>
>>>>>>>    - If you think that something should be top priority, what gets
>>>>>>>    moved to priority 2?
>>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>>
>>>>>>> Top priority, 1:
>>>>>>>
>>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>>    -
>>>>>>>
>>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>>    -
>>>>>>>
>>>>>>>    Python: Pythonic refactor [medium]
>>>>>>>
>>>>>>> Priority 2:
>>>>>>>
>>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>>    - Flink: Inline file compaction [small]
>>>>>>>    - Flink: Support UPSERT [small]
>>>>>>>    - Views: Spec [medium]
>>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>>    - Spec: Secondary indexes [large]
>>>>>>>    - Spec v3: Encryption [large]
>>>>>>>    -
>>>>>>>
>>>>>>>    Spec v3: Relative paths [large]
>>>>>>>    -
>>>>>>>
>>>>>>>    Spec v3: Default field values [medium]
>>>>>>>
>>>>>>> Priority 3:
>>>>>>>
>>>>>>>    - Docs: versioned docs [medium]
>>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>>
>>>>>>> External:
>>>>>>>
>>>>>>>    - Trino: Bucketed joins [small]
>>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Tabular
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

I closed the upgrade project and marked the FLIP-27 project priority 1.
Thanks for all the work to get this done!

On Sun, Oct 31, 2021 at 8:10 PM OpenInx <op...@gmail.com> wrote:

> Update:
>
> I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be
> closed now, because all of the issues have been addressed.
>
> [1]. https://github.com/apache/iceberg/projects/12
>
> On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
>> I created a Roadmap section in
>>  https://github.com/apache/iceberg/pull/3163
>> <https://github.com/apache/iceberg/pull/3163> that links to the planning
>> boards that Jack created. I figured it makes sense if we link available
>> Design Docs directly on those Boards (as was already done), because then
>> the Design docs are closer to the set of related issues.
>>
>> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Thanks, Jack!
>>>
>>> Eduard, I think that's a good idea. We should have a roadmap page as
>>> well that links to the projects that Jack just created.
>>>
>>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> It seems like we have reached some consensus around the projects listed
>>>> here. I have created corresponding Github projects for each:
>>>> https://github.com/apache/iceberg/projects
>>>>
>>>> Related design docs are also linked there.
>>>>
>>>> Best,
>>>> Jack Ye
>>>>
>>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>> wrote:
>>>>
>>>>> Would it make sense to have a section on the website where we collect
>>>>> all the links to the design docs/specs as that would be easier to find than
>>>>> searching for things on the ML?
>>>>>
>>>>> I was thinking about something like for each component:
>>>>> * link to the ML discussion
>>>>> * link to the actual Spec/Design Doc
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>> good list of what people are thinking about and I think we should try to
>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>> to some projects in Github.
>>>>>>
>>>>>> My rationale for attempting to prioritize projects is that if we try
>>>>>> to do too many things, it will be slower progress across everything rather
>>>>>> than getting a few important items done. I know that priorities don’t align
>>>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>>>> Everything else is priority 2.
>>>>>>
>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>>> if someone has extra time to review something, what should be next? That’s
>>>>>> top priority.
>>>>>>
>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>
>>>>>>    - If you think that something should be top priority, what gets
>>>>>>    moved to priority 2?
>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>
>>>>>> Top priority, 1:
>>>>>>
>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>    -
>>>>>>
>>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>    -
>>>>>>
>>>>>>    Python: Pythonic refactor [medium]
>>>>>>
>>>>>> Priority 2:
>>>>>>
>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>    - Flink: Inline file compaction [small]
>>>>>>    - Flink: Support UPSERT [small]
>>>>>>    - Views: Spec [medium]
>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>    - Spec: Secondary indexes [large]
>>>>>>    - Spec v3: Encryption [large]
>>>>>>    -
>>>>>>
>>>>>>    Spec v3: Relative paths [large]
>>>>>>    -
>>>>>>
>>>>>>    Spec v3: Default field values [medium]
>>>>>>
>>>>>> Priority 3:
>>>>>>
>>>>>>    - Docs: versioned docs [medium]
>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>
>>>>>> External:
>>>>>>
>>>>>>    - Trino: Bucketed joins [small]
>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Tabular
>>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[OpenInx <op...@gmail.com>]

Update:

I think the project  [Flink: Upgrade to 1.13.2][1] in RoadMap can be closed
now, because all of the issues have been addressed.

[1]. https://github.com/apache/iceberg/projects/12

On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner <ed...@dremio.com>
wrote:

> I created a Roadmap section in https://github.com/apache/iceberg/pull/3163
> <https://github.com/apache/iceberg/pull/3163> that links to the planning
> boards that Jack created. I figured it makes sense if we link available
> Design Docs directly on those Boards (as was already done), because then
> the Design docs are closer to the set of related issues.
>
> On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Thanks, Jack!
>>
>> Eduard, I think that's a good idea. We should have a roadmap page as well
>> that links to the projects that Jack just created.
>>
>> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>>
>>> It seems like we have reached some consensus around the projects listed
>>> here. I have created corresponding Github projects for each:
>>> https://github.com/apache/iceberg/projects
>>>
>>> Related design docs are also linked there.
>>>
>>> Best,
>>> Jack Ye
>>>
>>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <ed...@dremio.com>
>>> wrote:
>>>
>>>> Would it make sense to have a section on the website where we collect
>>>> all the links to the design docs/specs as that would be easier to find than
>>>> searching for things on the ML?
>>>>
>>>> I was thinking about something like for each component:
>>>> * link to the ML discussion
>>>> * link to the actual Spec/Design Doc
>>>>
>>>> Thoughts?
>>>>
>>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> At the last sync meeting, we brought up publishing a community roadmap
>>>>> and brainstormed the many features and initiatives that the community is
>>>>> working on. In this thread, I want to make sure that we have a good list of
>>>>> what people are thinking about and I think we should try to categorize the
>>>>> projects by size and general priority. When we reach a rough agreement,
>>>>> I’ll write this up and post it on the ASF site along with links to some
>>>>> projects in Github.
>>>>>
>>>>> My rationale for attempting to prioritize projects is that if we try
>>>>> to do too many things, it will be slower progress across everything rather
>>>>> than getting a few important items done. I know that priorities don’t align
>>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>>> Everything else is priority 2.
>>>>>
>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>> if someone has extra time to review something, what should be next? That’s
>>>>> top priority.
>>>>>
>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>
>>>>>    - If you think that something should be top priority, what gets
>>>>>    moved to priority 2?
>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>    - Is the S/M/L size of a project wrong?
>>>>>
>>>>> Top priority, 1:
>>>>>
>>>>>    - API: Iceberg 1.0 [medium]
>>>>>    - Spark: Merge-on-read plans [large]
>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>    -
>>>>>
>>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>    -
>>>>>
>>>>>    Python: Pythonic refactor [medium]
>>>>>
>>>>> Priority 2:
>>>>>
>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>    - Flink: Inline file compaction [small]
>>>>>    - Flink: Support UPSERT [small]
>>>>>    - Views: Spec [medium]
>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>    - Spec: Secondary indexes [large]
>>>>>    - Spec v3: Encryption [large]
>>>>>    -
>>>>>
>>>>>    Spec v3: Relative paths [large]
>>>>>    -
>>>>>
>>>>>    Spec v3: Default field values [medium]
>>>>>
>>>>> Priority 3:
>>>>>
>>>>>    - Docs: versioned docs [medium]
>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>    - IO: Support Dell ECS [medium]
>>>>>
>>>>> External:
>>>>>
>>>>>    - Trino: Bucketed joins [small]
>>>>>    - Trino: Row-level delete support [medium]
>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>    - Trino: Multi-catalog support [small]
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Tabular
>>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[Eduard Tudenhoefner <ed...@dremio.com>]

I created a Roadmap section in https://github.com/apache/iceberg/pull/3163
<https://github.com/apache/iceberg/pull/3163> that links to the planning
boards that Jack created. I figured it makes sense if we link available
Design Docs directly on those Boards (as was already done), because then
the Design docs are closer to the set of related issues.

On Mon, Sep 20, 2021 at 10:02 PM Ryan Blue <bl...@tabular.io> wrote:

> Thanks, Jack!
>
> Eduard, I think that's a good idea. We should have a roadmap page as well
> that links to the projects that Jack just created.
>
> On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:
>
>> It seems like we have reached some consensus around the projects listed
>> here. I have created corresponding Github projects for each:
>> https://github.com/apache/iceberg/projects
>>
>> Related design docs are also linked there.
>>
>> Best,
>> Jack Ye
>>
>> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <ed...@dremio.com>
>> wrote:
>>
>>> Would it make sense to have a section on the website where we collect
>>> all the links to the design docs/specs as that would be easier to find than
>>> searching for things on the ML?
>>>
>>> I was thinking about something like for each component:
>>> * link to the ML discussion
>>> * link to the actual Spec/Design Doc
>>>
>>> Thoughts?
>>>
>>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> At the last sync meeting, we brought up publishing a community roadmap
>>>> and brainstormed the many features and initiatives that the community is
>>>> working on. In this thread, I want to make sure that we have a good list of
>>>> what people are thinking about and I think we should try to categorize the
>>>> projects by size and general priority. When we reach a rough agreement,
>>>> I’ll write this up and post it on the ASF site along with links to some
>>>> projects in Github.
>>>>
>>>> My rationale for attempting to prioritize projects is that if we try to
>>>> do too many things, it will be slower progress across everything rather
>>>> than getting a few important items done. I know that priorities don’t align
>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>> Everything else is priority 2.
>>>>
>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>> if someone has extra time to review something, what should be next? That’s
>>>> top priority.
>>>>
>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>
>>>>    - If you think that something should be top priority, what gets
>>>>    moved to priority 2?
>>>>    - Should the priority for a project in 2 or 3 change?
>>>>    - Is the S/M/L size of a project wrong?
>>>>
>>>> Top priority, 1:
>>>>
>>>>    - API: Iceberg 1.0 [medium]
>>>>    - Spark: Merge-on-read plans [large]
>>>>    - Maintenance: Delete file compaction [medium]
>>>>    -
>>>>
>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>    -
>>>>
>>>>    Python: Pythonic refactor [medium]
>>>>
>>>> Priority 2:
>>>>
>>>>    - ORC: Support delete files stored as ORC [small]
>>>>    - Spark: DSv2 streaming improvements [small]
>>>>    - Flink: Inline file compaction [small]
>>>>    - Flink: Support UPSERT [small]
>>>>    - Views: Spec [medium]
>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>    - Spec: Snapshot tagging and branching [small]
>>>>    - Spec: Secondary indexes [large]
>>>>    - Spec v3: Encryption [large]
>>>>    -
>>>>
>>>>    Spec v3: Relative paths [large]
>>>>    -
>>>>
>>>>    Spec v3: Default field values [medium]
>>>>
>>>> Priority 3:
>>>>
>>>>    - Docs: versioned docs [medium]
>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>    - IO: Support Dell ECS [medium]
>>>>
>>>> External:
>>>>
>>>>    - Trino: Bucketed joins [small]
>>>>    - Trino: Row-level delete support [medium]
>>>>    - Trino: Merge-on-read plans [medium]
>>>>    - Trino: Multi-catalog support [small]
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Thanks, Jack!

Eduard, I think that's a good idea. We should have a roadmap page as well
that links to the projects that Jack just created.

On Mon, Sep 20, 2021 at 12:57 PM Jack Ye <ye...@gmail.com> wrote:

> It seems like we have reached some consensus around the projects listed
> here. I have created corresponding Github projects for each:
> https://github.com/apache/iceberg/projects
>
> Related design docs are also linked there.
>
> Best,
> Jack Ye
>
> On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
>> Would it make sense to have a section on the website where we collect all
>> the links to the design docs/specs as that would be easier to find than
>> searching for things on the ML?
>>
>> I was thinking about something like for each component:
>> * link to the ML discussion
>> * link to the actual Spec/Design Doc
>>
>> Thoughts?
>>
>> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Hi everyone,
>>>
>>> At the last sync meeting, we brought up publishing a community roadmap
>>> and brainstormed the many features and initiatives that the community is
>>> working on. In this thread, I want to make sure that we have a good list of
>>> what people are thinking about and I think we should try to categorize the
>>> projects by size and general priority. When we reach a rough agreement,
>>> I’ll write this up and post it on the ASF site along with links to some
>>> projects in Github.
>>>
>>> My rationale for attempting to prioritize projects is that if we try to
>>> do too many things, it will be slower progress across everything rather
>>> than getting a few important items done. I know that priorities don’t align
>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>> a priority, I’m trying to keep top priority items to a minimum by including
>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>> things that can be plugged in (like other IO libraries), docs, etc.
>>> Everything else is priority 2.
>>>
>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>> progressing, just that it isn’t the current focus. I think of it this way:
>>> if someone has extra time to review something, what should be next? That’s
>>> top priority.
>>>
>>> Here’s my rough categorization. If you disagree, please speak up:
>>>
>>>    - If you think that something should be top priority, what gets
>>>    moved to priority 2?
>>>    - Should the priority for a project in 2 or 3 change?
>>>    - Is the S/M/L size of a project wrong?
>>>
>>> Top priority, 1:
>>>
>>>    - API: Iceberg 1.0 [medium]
>>>    - Spark: Merge-on-read plans [large]
>>>    - Maintenance: Delete file compaction [medium]
>>>    -
>>>
>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>    -
>>>
>>>    Python: Pythonic refactor [medium]
>>>
>>> Priority 2:
>>>
>>>    - ORC: Support delete files stored as ORC [small]
>>>    - Spark: DSv2 streaming improvements [small]
>>>    - Flink: Inline file compaction [small]
>>>    - Flink: Support UPSERT [small]
>>>    - Views: Spec [medium]
>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>    - Spec: Snapshot tagging and branching [small]
>>>    - Spec: Secondary indexes [large]
>>>    - Spec v3: Encryption [large]
>>>    -
>>>
>>>    Spec v3: Relative paths [large]
>>>    -
>>>
>>>    Spec v3: Default field values [medium]
>>>
>>> Priority 3:
>>>
>>>    - Docs: versioned docs [medium]
>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>    - IO: Support Dell ECS [medium]
>>>
>>> External:
>>>
>>>    - Trino: Bucketed joins [small]
>>>    - Trino: Row-level delete support [medium]
>>>    - Trino: Merge-on-read plans [medium]
>>>    - Trino: Multi-catalog support [small]
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Jack Ye <ye...@gmail.com>]

It seems like we have reached some consensus around the projects listed
here. I have created corresponding Github projects for each:
https://github.com/apache/iceberg/projects

Related design docs are also linked there.

Best,
Jack Ye

On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner <ed...@dremio.com>
wrote:

> Would it make sense to have a section on the website where we collect all
> the links to the design docs/specs as that would be easier to find than
> searching for things on the ML?
>
> I was thinking about something like for each component:
> * link to the ML discussion
> * link to the actual Spec/Design Doc
>
> Thoughts?
>
> On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Hi everyone,
>>
>> At the last sync meeting, we brought up publishing a community roadmap
>> and brainstormed the many features and initiatives that the community is
>> working on. In this thread, I want to make sure that we have a good list of
>> what people are thinking about and I think we should try to categorize the
>> projects by size and general priority. When we reach a rough agreement,
>> I’ll write this up and post it on the ASF site along with links to some
>> projects in Github.
>>
>> My rationale for attempting to prioritize projects is that if we try to
>> do too many things, it will be slower progress across everything rather
>> than getting a few important items done. I know that priorities don’t align
>> very cleanly in practice, but it is hopefully worth trying. To come up with
>> a priority, I’m trying to keep top priority items to a minimum by including
>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>> are split between priority 2 and 3. Priority 3 is not urgent, including
>> things that can be plugged in (like other IO libraries), docs, etc.
>> Everything else is priority 2.
>>
>> That something isn’t priority 1 doesn’t mean it isn’t important or
>> progressing, just that it isn’t the current focus. I think of it this way:
>> if someone has extra time to review something, what should be next? That’s
>> top priority.
>>
>> Here’s my rough categorization. If you disagree, please speak up:
>>
>>    - If you think that something should be top priority, what gets moved
>>    to priority 2?
>>    - Should the priority for a project in 2 or 3 change?
>>    - Is the S/M/L size of a project wrong?
>>
>> Top priority, 1:
>>
>>    - API: Iceberg 1.0 [medium]
>>    - Spark: Merge-on-read plans [large]
>>    - Maintenance: Delete file compaction [medium]
>>    -
>>
>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>    -
>>
>>    Python: Pythonic refactor [medium]
>>
>> Priority 2:
>>
>>    - ORC: Support delete files stored as ORC [small]
>>    - Spark: DSv2 streaming improvements [small]
>>    - Flink: Inline file compaction [small]
>>    - Flink: Support UPSERT [small]
>>    - Views: Spec [medium]
>>    - Spec: Z-ordering / Space-filling curves [medium]
>>    - Spec: Snapshot tagging and branching [small]
>>    - Spec: Secondary indexes [large]
>>    - Spec v3: Encryption [large]
>>    -
>>
>>    Spec v3: Relative paths [large]
>>    -
>>
>>    Spec v3: Default field values [medium]
>>
>> Priority 3:
>>
>>    - Docs: versioned docs [medium]
>>    - IO: Support Aliyun OSS/DLF [medium]
>>    - IO: Support Dell ECS [medium]
>>
>> External:
>>
>>    - Trino: Bucketed joins [small]
>>    - Trino: Row-level delete support [medium]
>>    - Trino: Merge-on-read plans [medium]
>>    - Trino: Multi-catalog support [small]
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[Eduard Tudenhoefner <ed...@dremio.com>]

Would it make sense to have a section on the website where we collect all
the links to the design docs/specs as that would be easier to find than
searching for things on the ML?

I was thinking about something like for each component:
* link to the ML discussion
* link to the actual Spec/Design Doc

Thoughts?

On Fri, Sep 10, 2021 at 11:38 PM Ryan Blue <bl...@tabular.io> wrote:

> Hi everyone,
>
> At the last sync meeting, we brought up publishing a community roadmap and
> brainstormed the many features and initiatives that the community is
> working on. In this thread, I want to make sure that we have a good list of
> what people are thinking about and I think we should try to categorize the
> projects by size and general priority. When we reach a rough agreement,
> I’ll write this up and post it on the ASF site along with links to some
> projects in Github.
>
> My rationale for attempting to prioritize projects is that if we try to do
> too many things, it will be slower progress across everything rather than
> getting a few important items done. I know that priorities don’t align very
> cleanly in practice, but it is hopefully worth trying. To come up with a
> priority, I’m trying to keep top priority items to a minimum by including
> only one from each group (Spark, Flink, Python, etc.). The remaining items
> are split between priority 2 and 3. Priority 3 is not urgent, including
> things that can be plugged in (like other IO libraries), docs, etc.
> Everything else is priority 2.
>
> That something isn’t priority 1 doesn’t mean it isn’t important or
> progressing, just that it isn’t the current focus. I think of it this way:
> if someone has extra time to review something, what should be next? That’s
> top priority.
>
> Here’s my rough categorization. If you disagree, please speak up:
>
>    - If you think that something should be top priority, what gets moved
>    to priority 2?
>    - Should the priority for a project in 2 or 3 change?
>    - Is the S/M/L size of a project wrong?
>
> Top priority, 1:
>
>    - API: Iceberg 1.0 [medium]
>    - Spark: Merge-on-read plans [large]
>    - Maintenance: Delete file compaction [medium]
>    -
>
>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>    -
>
>    Python: Pythonic refactor [medium]
>
> Priority 2:
>
>    - ORC: Support delete files stored as ORC [small]
>    - Spark: DSv2 streaming improvements [small]
>    - Flink: Inline file compaction [small]
>    - Flink: Support UPSERT [small]
>    - Views: Spec [medium]
>    - Spec: Z-ordering / Space-filling curves [medium]
>    - Spec: Snapshot tagging and branching [small]
>    - Spec: Secondary indexes [large]
>    - Spec v3: Encryption [large]
>    -
>
>    Spec v3: Relative paths [large]
>    -
>
>    Spec v3: Default field values [medium]
>
> Priority 3:
>
>    - Docs: versioned docs [medium]
>    - IO: Support Aliyun OSS/DLF [medium]
>    - IO: Support Dell ECS [medium]
>
> External:
>
>    - Trino: Bucketed joins [small]
>    - Trino: Row-level delete support [medium]
>    - Trino: Merge-on-read plans [medium]
>    - Trino: Multi-catalog support [small]
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[OpenInx <op...@gmail.com>]

Thanks Steven & Kyle.

Yes,  the flip-27 source and flink 1.13.2 are orthogonal because the
flink's flip-27 API  was successfully introduced in flink 1.12 release (
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface).
The WIP flip-27 iceberg source proposed from Steven is also created on top
of flink 1.12.  There are two different things.  The flip27 source has
great value when people want to replace their kafka with iceberg tables to
accomplish historical data backfill & bootstrap , I believe the guys from
netflix has clearly explained the value & best practise in this talk (
https://www.youtube.com/watch?v=rtz3p_iijP8&feature=youtu.be&ab_channel=NetflixData)
.  So I agree with Steven to put the flip-27 source into the priority 2 in
our apache iceberg roadmap.

On Sat, Sep 18, 2021 at 4:41 AM Kyle Bendickson
<kb...@apple.com.invalid> wrote:

> This list looks overall pretty good to me. +1
>
> For Flink 1.13 upgrade, I suggest we consider starting another thread for
> it. There are some open PRs, but they have outstanding questions.
> Specifically, dropping support for Flink 1.12 or not. I think we can
> upgrade without dropping support for Flink 1.12, but we wouldn’t get some
> of the proposed benefits of 1.13 (though that can be a follow up task).
>
> I’m not presently involved in the Flink Community enough to say with
> certainty, but I believe the FLIP-27 (Using the new source interface) and
> the Flink 1.13.2 upgrade are orthogonal to each other and can both progress
> independently. But I would defer to Steven or anybody else who works with
> Flink much more often than I do currently.
>
> - Kyle Bendickson
>
> On Sep 15, 2021, at 4:06 PM, Ryan Blue <bl...@tabular.io> wrote:
>
> That sounds great, thanks for taking that on Jack!
>
> On Wed, Sep 15, 2021 at 3:51 PM Jack Ye <ye...@gmail.com> wrote:
>
>> For external Trino and PrestoDB tasks, I am thinking about creating one
>> Github project for Trino and another one for PrestoDB to manage all tasks
>> under them, adding links of issues and PRs in the other communities to
>> track progress. This is mostly to improve visibility so that people who are
>> interested can see what is going on in those 2 places.
>>
>> -Jack Ye
>>
>> On Wed, Sep 15, 2021 at 2:14 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Gidon, I think that the v3 part of encryption is actually documenting
>>> how it works and adding it to the spec. Right now we have hooks for
>>> building some encryption around it, but almost no requirements in the spec
>>> for how to use it across implementations. This is fine while we're working
>>> on defining encryption, but we eventually want to update the spec.
>>>
>>> Jack, I'm happy to add the external PrestoDB items to the roadmap. I'm
>>> just not quite sure what to do here since we aren't tracking them in the
>>> Iceberg community ourselves. I listed those as external so that we can
>>> publish links to where those are tracked in other communities. We can add
>>> as many of these as we want.
>>>
>>> Anton, I agree. The goal here is to identify the top priority items to
>>> help direct review effort. We want everything to continue progressing, but
>>> I think it's good to identify where we as a community want to focus review
>>> time.
>>>
>>> Sounds like one area of uncertainty is FLIP-27 vs Flink 1.13.2. Can
>>> someone summarize the status of Flink and what we need? I don't think I
>>> understand it well enough to suggest which one takes priority.
>>>
>>> Ryan
>>>
>>> On Mon, Sep 13, 2021 at 7:54 PM Anton Okolnychyi <
>>> aokolnychyi@apple.com.invalid> wrote:
>>>
>>>> The discussed roadmap makes sense to me. I think it is important to
>>>> agree on what we should do first as the review pool is limited. There are
>>>> more and more large items that are half done or half discussed. I think we
>>>> better focus on finishing them quickly and then move to something else as
>>>> opposed to making very minor progress on a number of issues.
>>>>
>>>> To be clear, it is not like other things are not important or we should
>>>> stop their development. It is more about making sure certain high-priority
>>>> features for most folks in the community get enough attention.
>>>>
>>>> - Anton
>>>>
>>>> On 13 Sep 2021, at 12:19, Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>> I'd like to also propose adding the following in the external section:
>>>> 1. the PrestoDB equivalent for each item listed for Trino. I am not
>>>> sure what's the best way to track them, but I feel it's better to list and
>>>> track them separately. I have talked with related people currently
>>>> maintaining the PrestoDB Iceberg connector (mostly in Twitter), and they
>>>> would like to take a different route from Trino to fully remove Hive
>>>> dependencies in the connector. This means the 2 connectors will likely
>>>> diverge in implementation in the near future.
>>>> 2. adding a medium item for Trino and PrestoDB Avro support
>>>> 3. adding a small item for Trino and PrestoDB full system table support
>>>> (the system table schema in them are diverging from core, and missing a few
>>>> latest system tables)
>>>>
>>>> For the items listed with "Spec" and "Spec v3", what are the key
>>>> differences? I thought we are treating any new spec changes after the
>>>> format v2 vote as v3.
>>>>
>>>> Best,
>>>> Jack Ye
>>>>
>>>> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ryan,
>>>>>
>>>>> I just wonder if the encryption should be a Spec v3 category. We have
>>>>> the key_metadata fields in both data_file and manifest_file structs, which
>>>>> might be sufficient for a reasonable basic encryption support.
>>>>> But I certainly agree this is an L-sized project.
>>>>>
>>>>> Cheers, Gidon
>>>>>
>>>>>
>>>>> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> At the last sync meeting, we brought up publishing a community
>>>>>> roadmap and brainstormed the many features and initiatives that the
>>>>>> community is working on. In this thread, I want to make sure that we have a
>>>>>> good list of what people are thinking about and I think we should try to
>>>>>> categorize the projects by size and general priority. When we reach a rough
>>>>>> agreement, I’ll write this up and post it on the ASF site along with links
>>>>>> to some projects in Github.
>>>>>>
>>>>>> My rationale for attempting to prioritize projects is that if we try
>>>>>> to do too many things, it will be slower progress across everything rather
>>>>>> than getting a few important items done. I know that priorities don’t align
>>>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>>>> Everything else is priority 2.
>>>>>>
>>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>>> if someone has extra time to review something, what should be next? That’s
>>>>>> top priority.
>>>>>>
>>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>>
>>>>>>    - If you think that something should be top priority, what gets
>>>>>>    moved to priority 2?
>>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>>    - Is the S/M/L size of a project wrong?
>>>>>>
>>>>>> Top priority, 1:
>>>>>>
>>>>>>    - API: Iceberg 1.0 [medium]
>>>>>>    - Spark: Merge-on-read plans [large]
>>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>>    - Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>>    - Python: Pythonic refactor [medium]
>>>>>>
>>>>>> Priority 2:
>>>>>>
>>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>>    - Flink: Inline file compaction [small]
>>>>>>    - Flink: Support UPSERT [small]
>>>>>>    - Views: Spec [medium]
>>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>>    - Spec: Secondary indexes [large]
>>>>>>    - Spec v3: Encryption [large]
>>>>>>    - Spec v3: Relative paths [large]
>>>>>>    - Spec v3: Default field values [medium]
>>>>>>
>>>>>> Priority 3:
>>>>>>
>>>>>>    - Docs: versioned docs [medium]
>>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>>    - IO: Support Dell ECS [medium]
>>>>>>
>>>>>> External:
>>>>>>
>>>>>>    - Trino: Bucketed joins [small]
>>>>>>    - Trino: Row-level delete support [medium]
>>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>>    - Trino: Multi-catalog support [small]
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Tabular
>>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>
> --
> Ryan Blue
> Tabular
>
>
>

Re: [DISCUSS] Iceberg roadmap

[Kyle Bendickson <kb...@apple.com.INVALID>]

This list looks overall pretty good to me. +1

For Flink 1.13 upgrade, I suggest we consider starting another thread for it. There are some open PRs, but they have outstanding questions. Specifically, dropping support for Flink 1.12 or not. I think we can upgrade without dropping support for Flink 1.12, but we wouldn’t get some of the proposed benefits of 1.13 (though that can be a follow up task).

I’m not presently involved in the Flink Community enough to say with certainty, but I believe the FLIP-27 (Using the new source interface) and the Flink 1.13.2 upgrade are orthogonal to each other and can both progress independently. But I would defer to Steven or anybody else who works with Flink much more often than I do currently.

- Kyle Bendickson

> On Sep 15, 2021, at 4:06 PM, Ryan Blue <bl...@tabular.io> wrote:
> 
> That sounds great, thanks for taking that on Jack!
> 
> On Wed, Sep 15, 2021 at 3:51 PM Jack Ye <yezhaoqin@gmail.com <ma...@gmail.com>> wrote:
> For external Trino and PrestoDB tasks, I am thinking about creating one Github project for Trino and another one for PrestoDB to manage all tasks under them, adding links of issues and PRs in the other communities to track progress. This is mostly to improve visibility so that people who are interested can see what is going on in those 2 places.
> 
> -Jack Ye
> 
> On Wed, Sep 15, 2021 at 2:14 PM Ryan Blue <blue@tabular.io <ma...@tabular.io>> wrote:
> Gidon, I think that the v3 part of encryption is actually documenting how it works and adding it to the spec. Right now we have hooks for building some encryption around it, but almost no requirements in the spec for how to use it across implementations. This is fine while we're working on defining encryption, but we eventually want to update the spec.
> 
> Jack, I'm happy to add the external PrestoDB items to the roadmap. I'm just not quite sure what to do here since we aren't tracking them in the Iceberg community ourselves. I listed those as external so that we can publish links to where those are tracked in other communities. We can add as many of these as we want.
> 
> Anton, I agree. The goal here is to identify the top priority items to help direct review effort. We want everything to continue progressing, but I think it's good to identify where we as a community want to focus review time.
> 
> Sounds like one area of uncertainty is FLIP-27 vs Flink 1.13.2. Can someone summarize the status of Flink and what we need? I don't think I understand it well enough to suggest which one takes priority.
> 
> Ryan
> 
> On Mon, Sep 13, 2021 at 7:54 PM Anton Okolnychyi <ao...@apple.com.invalid> wrote:
> The discussed roadmap makes sense to me. I think it is important to agree on what we should do first as the review pool is limited. There are more and more large items that are half done or half discussed. I think we better focus on finishing them quickly and then move to something else as opposed to making very minor progress on a number of issues.
> 
> To be clear, it is not like other things are not important or we should stop their development. It is more about making sure certain high-priority features for most folks in the community get enough attention.
> 
> - Anton
> 
>> On 13 Sep 2021, at 12:19, Jack Ye <yezhaoqin@gmail.com <ma...@gmail.com>> wrote:
>> 
>> I'd like to also propose adding the following in the external section:
>> 1. the PrestoDB equivalent for each item listed for Trino. I am not sure what's the best way to track them, but I feel it's better to list and track them separately. I have talked with related people currently maintaining the PrestoDB Iceberg connector (mostly in Twitter), and they would like to take a different route from Trino to fully remove Hive dependencies in the connector. This means the 2 connectors will likely diverge in implementation in the near future.
>> 2. adding a medium item for Trino and PrestoDB Avro support
>> 3. adding a small item for Trino and PrestoDB full system table support (the system table schema in them are diverging from core, and missing a few latest system tables)
>> 
>> For the items listed with "Spec" and "Spec v3", what are the key differences? I thought we are treating any new spec changes after the format v2 vote as v3.
>> 
>> Best,
>> Jack Ye
>> 
>> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg5070@gmail.com <ma...@gmail.com>> wrote:
>> Hi Ryan,
>> 
>> I just wonder if the encryption should be a Spec v3 category. We have the key_metadata fields in both data_file and manifest_file structs, which might be sufficient for a reasonable basic encryption support.
>> But I certainly agree this is an L-sized project.
>> 
>> Cheers, Gidon
>> 
>> 
>> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <blue@tabular.io <ma...@tabular.io>> wrote:
>> Hi everyone,
>> 
>> At the last sync meeting, we brought up publishing a community roadmap and brainstormed the many features and initiatives that the community is working on. In this thread, I want to make sure that we have a good list of what people are thinking about and I think we should try to categorize the projects by size and general priority. When we reach a rough agreement, I’ll write this up and post it on the ASF site along with links to some projects in Github.
>> 
>> My rationale for attempting to prioritize projects is that if we try to do too many things, it will be slower progress across everything rather than getting a few important items done. I know that priorities don’t align very cleanly in practice, but it is hopefully worth trying. To come up with a priority, I’m trying to keep top priority items to a minimum by including only one from each group (Spark, Flink, Python, etc.). The remaining items are split between priority 2 and 3. Priority 3 is not urgent, including things that can be plugged in (like other IO libraries), docs, etc. Everything else is priority 2.
>> 
>> That something isn’t priority 1 doesn’t mean it isn’t important or progressing, just that it isn’t the current focus. I think of it this way: if someone has extra time to review something, what should be next? That’s top priority.
>> 
>> Here’s my rough categorization. If you disagree, please speak up:
>> 
>> If you think that something should be top priority, what gets moved to priority 2?
>> Should the priority for a project in 2 or 3 change?
>> Is the S/M/L size of a project wrong?
>> Top priority, 1:
>> 
>> API: Iceberg 1.0 [medium]
>> Spark: Merge-on-read plans [large]
>> Maintenance: Delete file compaction [medium]
>> Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>> Python: Pythonic refactor [medium]
>> Priority 2:
>> 
>> ORC: Support delete files stored as ORC [small]
>> Spark: DSv2 streaming improvements [small]
>> Flink: Inline file compaction [small]
>> Flink: Support UPSERT [small]
>> Views: Spec [medium]
>> Spec: Z-ordering / Space-filling curves [medium]
>> Spec: Snapshot tagging and branching [small]
>> Spec: Secondary indexes [large]
>> Spec v3: Encryption [large]
>> Spec v3: Relative paths [large]
>> Spec v3: Default field values [medium]
>> Priority 3:
>> 
>> Docs: versioned docs [medium]
>> IO: Support Aliyun OSS/DLF [medium]
>> IO: Support Dell ECS [medium]
>> External:
>> 
>> Trino: Bucketed joins [small]
>> Trino: Row-level delete support [medium]
>> Trino: Merge-on-read plans [medium]
>> Trino: Multi-catalog support [small]
>> -- 
>> Ryan Blue
>> Tabular
> 
> 
> 
> -- 
> Ryan Blue
> Tabular
> 
> 
> -- 
> Ryan Blue
> Tabular

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

That sounds great, thanks for taking that on Jack!

On Wed, Sep 15, 2021 at 3:51 PM Jack Ye <ye...@gmail.com> wrote:

> For external Trino and PrestoDB tasks, I am thinking about creating one
> Github project for Trino and another one for PrestoDB to manage all tasks
> under them, adding links of issues and PRs in the other communities to
> track progress. This is mostly to improve visibility so that people who are
> interested can see what is going on in those 2 places.
>
> -Jack Ye
>
> On Wed, Sep 15, 2021 at 2:14 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Gidon, I think that the v3 part of encryption is actually documenting how
>> it works and adding it to the spec. Right now we have hooks for building
>> some encryption around it, but almost no requirements in the spec for how
>> to use it across implementations. This is fine while we're working on
>> defining encryption, but we eventually want to update the spec.
>>
>> Jack, I'm happy to add the external PrestoDB items to the roadmap. I'm
>> just not quite sure what to do here since we aren't tracking them in the
>> Iceberg community ourselves. I listed those as external so that we can
>> publish links to where those are tracked in other communities. We can add
>> as many of these as we want.
>>
>> Anton, I agree. The goal here is to identify the top priority items to
>> help direct review effort. We want everything to continue progressing, but
>> I think it's good to identify where we as a community want to focus review
>> time.
>>
>> Sounds like one area of uncertainty is FLIP-27 vs Flink 1.13.2. Can
>> someone summarize the status of Flink and what we need? I don't think I
>> understand it well enough to suggest which one takes priority.
>>
>> Ryan
>>
>> On Mon, Sep 13, 2021 at 7:54 PM Anton Okolnychyi
>> <ao...@apple.com.invalid> wrote:
>>
>>> The discussed roadmap makes sense to me. I think it is important to
>>> agree on what we should do first as the review pool is limited. There are
>>> more and more large items that are half done or half discussed. I think we
>>> better focus on finishing them quickly and then move to something else as
>>> opposed to making very minor progress on a number of issues.
>>>
>>> To be clear, it is not like other things are not important or we should
>>> stop their development. It is more about making sure certain high-priority
>>> features for most folks in the community get enough attention.
>>>
>>> - Anton
>>>
>>> On 13 Sep 2021, at 12:19, Jack Ye <ye...@gmail.com> wrote:
>>>
>>> I'd like to also propose adding the following in the external section:
>>> 1. the PrestoDB equivalent for each item listed for Trino. I am not sure
>>> what's the best way to track them, but I feel it's better to list and track
>>> them separately. I have talked with related people currently maintaining
>>> the PrestoDB Iceberg connector (mostly in Twitter), and they would like to
>>> take a different route from Trino to fully remove Hive dependencies in the
>>> connector. This means the 2 connectors will likely diverge in
>>> implementation in the near future.
>>> 2. adding a medium item for Trino and PrestoDB Avro support
>>> 3. adding a small item for Trino and PrestoDB full system table support
>>> (the system table schema in them are diverging from core, and missing a few
>>> latest system tables)
>>>
>>> For the items listed with "Spec" and "Spec v3", what are the key
>>> differences? I thought we are treating any new spec changes after the
>>> format v2 vote as v3.
>>>
>>> Best,
>>> Jack Ye
>>>
>>> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ryan,
>>>>
>>>> I just wonder if the encryption should be a Spec v3 category. We have
>>>> the key_metadata fields in both data_file and manifest_file structs, which
>>>> might be sufficient for a reasonable basic encryption support.
>>>> But I certainly agree this is an L-sized project.
>>>>
>>>> Cheers, Gidon
>>>>
>>>>
>>>> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> At the last sync meeting, we brought up publishing a community roadmap
>>>>> and brainstormed the many features and initiatives that the community is
>>>>> working on. In this thread, I want to make sure that we have a good list of
>>>>> what people are thinking about and I think we should try to categorize the
>>>>> projects by size and general priority. When we reach a rough agreement,
>>>>> I’ll write this up and post it on the ASF site along with links to some
>>>>> projects in Github.
>>>>>
>>>>> My rationale for attempting to prioritize projects is that if we try
>>>>> to do too many things, it will be slower progress across everything rather
>>>>> than getting a few important items done. I know that priorities don’t align
>>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>>> Everything else is priority 2.
>>>>>
>>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>>> if someone has extra time to review something, what should be next? That’s
>>>>> top priority.
>>>>>
>>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>>
>>>>>    - If you think that something should be top priority, what gets
>>>>>    moved to priority 2?
>>>>>    - Should the priority for a project in 2 or 3 change?
>>>>>    - Is the S/M/L size of a project wrong?
>>>>>
>>>>> Top priority, 1:
>>>>>
>>>>>    - API: Iceberg 1.0 [medium]
>>>>>    - Spark: Merge-on-read plans [large]
>>>>>    - Maintenance: Delete file compaction [medium]
>>>>>    - Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>>    - Python: Pythonic refactor [medium]
>>>>>
>>>>> Priority 2:
>>>>>
>>>>>    - ORC: Support delete files stored as ORC [small]
>>>>>    - Spark: DSv2 streaming improvements [small]
>>>>>    - Flink: Inline file compaction [small]
>>>>>    - Flink: Support UPSERT [small]
>>>>>    - Views: Spec [medium]
>>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>>    - Spec: Snapshot tagging and branching [small]
>>>>>    - Spec: Secondary indexes [large]
>>>>>    - Spec v3: Encryption [large]
>>>>>    - Spec v3: Relative paths [large]
>>>>>    - Spec v3: Default field values [medium]
>>>>>
>>>>> Priority 3:
>>>>>
>>>>>    - Docs: versioned docs [medium]
>>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>>    - IO: Support Dell ECS [medium]
>>>>>
>>>>> External:
>>>>>
>>>>>    - Trino: Bucketed joins [small]
>>>>>    - Trino: Row-level delete support [medium]
>>>>>    - Trino: Merge-on-read plans [medium]
>>>>>    - Trino: Multi-catalog support [small]
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Tabular
>>>>>
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Jack Ye <ye...@gmail.com>]

For external Trino and PrestoDB tasks, I am thinking about creating one
Github project for Trino and another one for PrestoDB to manage all tasks
under them, adding links of issues and PRs in the other communities to
track progress. This is mostly to improve visibility so that people who are
interested can see what is going on in those 2 places.

-Jack Ye

On Wed, Sep 15, 2021 at 2:14 PM Ryan Blue <bl...@tabular.io> wrote:

> Gidon, I think that the v3 part of encryption is actually documenting how
> it works and adding it to the spec. Right now we have hooks for building
> some encryption around it, but almost no requirements in the spec for how
> to use it across implementations. This is fine while we're working on
> defining encryption, but we eventually want to update the spec.
>
> Jack, I'm happy to add the external PrestoDB items to the roadmap. I'm
> just not quite sure what to do here since we aren't tracking them in the
> Iceberg community ourselves. I listed those as external so that we can
> publish links to where those are tracked in other communities. We can add
> as many of these as we want.
>
> Anton, I agree. The goal here is to identify the top priority items to
> help direct review effort. We want everything to continue progressing, but
> I think it's good to identify where we as a community want to focus review
> time.
>
> Sounds like one area of uncertainty is FLIP-27 vs Flink 1.13.2. Can
> someone summarize the status of Flink and what we need? I don't think I
> understand it well enough to suggest which one takes priority.
>
> Ryan
>
> On Mon, Sep 13, 2021 at 7:54 PM Anton Okolnychyi
> <ao...@apple.com.invalid> wrote:
>
>> The discussed roadmap makes sense to me. I think it is important to agree
>> on what we should do first as the review pool is limited. There are more
>> and more large items that are half done or half discussed. I think we
>> better focus on finishing them quickly and then move to something else as
>> opposed to making very minor progress on a number of issues.
>>
>> To be clear, it is not like other things are not important or we should
>> stop their development. It is more about making sure certain high-priority
>> features for most folks in the community get enough attention.
>>
>> - Anton
>>
>> On 13 Sep 2021, at 12:19, Jack Ye <ye...@gmail.com> wrote:
>>
>> I'd like to also propose adding the following in the external section:
>> 1. the PrestoDB equivalent for each item listed for Trino. I am not sure
>> what's the best way to track them, but I feel it's better to list and track
>> them separately. I have talked with related people currently maintaining
>> the PrestoDB Iceberg connector (mostly in Twitter), and they would like to
>> take a different route from Trino to fully remove Hive dependencies in the
>> connector. This means the 2 connectors will likely diverge in
>> implementation in the near future.
>> 2. adding a medium item for Trino and PrestoDB Avro support
>> 3. adding a small item for Trino and PrestoDB full system table support
>> (the system table schema in them are diverging from core, and missing a few
>> latest system tables)
>>
>> For the items listed with "Spec" and "Spec v3", what are the key
>> differences? I thought we are treating any new spec changes after the
>> format v2 vote as v3.
>>
>> Best,
>> Jack Ye
>>
>> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg...@gmail.com>
>> wrote:
>>
>>> Hi Ryan,
>>>
>>> I just wonder if the encryption should be a Spec v3 category. We have
>>> the key_metadata fields in both data_file and manifest_file structs, which
>>> might be sufficient for a reasonable basic encryption support.
>>> But I certainly agree this is an L-sized project.
>>>
>>> Cheers, Gidon
>>>
>>>
>>> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> At the last sync meeting, we brought up publishing a community roadmap
>>>> and brainstormed the many features and initiatives that the community is
>>>> working on. In this thread, I want to make sure that we have a good list of
>>>> what people are thinking about and I think we should try to categorize the
>>>> projects by size and general priority. When we reach a rough agreement,
>>>> I’ll write this up and post it on the ASF site along with links to some
>>>> projects in Github.
>>>>
>>>> My rationale for attempting to prioritize projects is that if we try to
>>>> do too many things, it will be slower progress across everything rather
>>>> than getting a few important items done. I know that priorities don’t align
>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>> Everything else is priority 2.
>>>>
>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>> if someone has extra time to review something, what should be next? That’s
>>>> top priority.
>>>>
>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>
>>>>    - If you think that something should be top priority, what gets
>>>>    moved to priority 2?
>>>>    - Should the priority for a project in 2 or 3 change?
>>>>    - Is the S/M/L size of a project wrong?
>>>>
>>>> Top priority, 1:
>>>>
>>>>    - API: Iceberg 1.0 [medium]
>>>>    - Spark: Merge-on-read plans [large]
>>>>    - Maintenance: Delete file compaction [medium]
>>>>    - Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>    - Python: Pythonic refactor [medium]
>>>>
>>>> Priority 2:
>>>>
>>>>    - ORC: Support delete files stored as ORC [small]
>>>>    - Spark: DSv2 streaming improvements [small]
>>>>    - Flink: Inline file compaction [small]
>>>>    - Flink: Support UPSERT [small]
>>>>    - Views: Spec [medium]
>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>    - Spec: Snapshot tagging and branching [small]
>>>>    - Spec: Secondary indexes [large]
>>>>    - Spec v3: Encryption [large]
>>>>    - Spec v3: Relative paths [large]
>>>>    - Spec v3: Default field values [medium]
>>>>
>>>> Priority 3:
>>>>
>>>>    - Docs: versioned docs [medium]
>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>    - IO: Support Dell ECS [medium]
>>>>
>>>> External:
>>>>
>>>>    - Trino: Bucketed joins [small]
>>>>    - Trino: Row-level delete support [medium]
>>>>    - Trino: Merge-on-read plans [medium]
>>>>    - Trino: Multi-catalog support [small]
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Gidon, I think that the v3 part of encryption is actually documenting how
it works and adding it to the spec. Right now we have hooks for building
some encryption around it, but almost no requirements in the spec for how
to use it across implementations. This is fine while we're working on
defining encryption, but we eventually want to update the spec.

Jack, I'm happy to add the external PrestoDB items to the roadmap. I'm just
not quite sure what to do here since we aren't tracking them in the Iceberg
community ourselves. I listed those as external so that we can publish
links to where those are tracked in other communities. We can add as many
of these as we want.

Anton, I agree. The goal here is to identify the top priority items to help
direct review effort. We want everything to continue progressing, but I
think it's good to identify where we as a community want to focus review
time.

Sounds like one area of uncertainty is FLIP-27 vs Flink 1.13.2. Can someone
summarize the status of Flink and what we need? I don't think I understand
it well enough to suggest which one takes priority.

Ryan

On Mon, Sep 13, 2021 at 7:54 PM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:

> The discussed roadmap makes sense to me. I think it is important to agree
> on what we should do first as the review pool is limited. There are more
> and more large items that are half done or half discussed. I think we
> better focus on finishing them quickly and then move to something else as
> opposed to making very minor progress on a number of issues.
>
> To be clear, it is not like other things are not important or we should
> stop their development. It is more about making sure certain high-priority
> features for most folks in the community get enough attention.
>
> - Anton
>
> On 13 Sep 2021, at 12:19, Jack Ye <ye...@gmail.com> wrote:
>
> I'd like to also propose adding the following in the external section:
> 1. the PrestoDB equivalent for each item listed for Trino. I am not sure
> what's the best way to track them, but I feel it's better to list and track
> them separately. I have talked with related people currently maintaining
> the PrestoDB Iceberg connector (mostly in Twitter), and they would like to
> take a different route from Trino to fully remove Hive dependencies in the
> connector. This means the 2 connectors will likely diverge in
> implementation in the near future.
> 2. adding a medium item for Trino and PrestoDB Avro support
> 3. adding a small item for Trino and PrestoDB full system table support
> (the system table schema in them are diverging from core, and missing a few
> latest system tables)
>
> For the items listed with "Spec" and "Spec v3", what are the key
> differences? I thought we are treating any new spec changes after the
> format v2 vote as v3.
>
> Best,
> Jack Ye
>
> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg...@gmail.com> wrote:
>
>> Hi Ryan,
>>
>> I just wonder if the encryption should be a Spec v3 category. We have the
>> key_metadata fields in both data_file and manifest_file structs, which
>> might be sufficient for a reasonable basic encryption support.
>> But I certainly agree this is an L-sized project.
>>
>> Cheers, Gidon
>>
>>
>> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Hi everyone,
>>>
>>> At the last sync meeting, we brought up publishing a community roadmap
>>> and brainstormed the many features and initiatives that the community is
>>> working on. In this thread, I want to make sure that we have a good list of
>>> what people are thinking about and I think we should try to categorize the
>>> projects by size and general priority. When we reach a rough agreement,
>>> I’ll write this up and post it on the ASF site along with links to some
>>> projects in Github.
>>>
>>> My rationale for attempting to prioritize projects is that if we try to
>>> do too many things, it will be slower progress across everything rather
>>> than getting a few important items done. I know that priorities don’t align
>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>> a priority, I’m trying to keep top priority items to a minimum by including
>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>> things that can be plugged in (like other IO libraries), docs, etc.
>>> Everything else is priority 2.
>>>
>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>> progressing, just that it isn’t the current focus. I think of it this way:
>>> if someone has extra time to review something, what should be next? That’s
>>> top priority.
>>>
>>> Here’s my rough categorization. If you disagree, please speak up:
>>>
>>>    - If you think that something should be top priority, what gets
>>>    moved to priority 2?
>>>    - Should the priority for a project in 2 or 3 change?
>>>    - Is the S/M/L size of a project wrong?
>>>
>>> Top priority, 1:
>>>
>>>    - API: Iceberg 1.0 [medium]
>>>    - Spark: Merge-on-read plans [large]
>>>    - Maintenance: Delete file compaction [medium]
>>>    - Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>    - Python: Pythonic refactor [medium]
>>>
>>> Priority 2:
>>>
>>>    - ORC: Support delete files stored as ORC [small]
>>>    - Spark: DSv2 streaming improvements [small]
>>>    - Flink: Inline file compaction [small]
>>>    - Flink: Support UPSERT [small]
>>>    - Views: Spec [medium]
>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>    - Spec: Snapshot tagging and branching [small]
>>>    - Spec: Secondary indexes [large]
>>>    - Spec v3: Encryption [large]
>>>    - Spec v3: Relative paths [large]
>>>    - Spec v3: Default field values [medium]
>>>
>>> Priority 3:
>>>
>>>    - Docs: versioned docs [medium]
>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>    - IO: Support Dell ECS [medium]
>>>
>>> External:
>>>
>>>    - Trino: Bucketed joins [small]
>>>    - Trino: Row-level delete support [medium]
>>>    - Trino: Merge-on-read plans [medium]
>>>    - Trino: Multi-catalog support [small]
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Anton Okolnychyi <ao...@apple.com.INVALID>]

The discussed roadmap makes sense to me. I think it is important to agree on what we should do first as the review pool is limited. There are more and more large items that are half done or half discussed. I think we better focus on finishing them quickly and then move to something else as opposed to making very minor progress on a number of issues.

To be clear, it is not like other things are not important or we should stop their development. It is more about making sure certain high-priority features for most folks in the community get enough attention.

- Anton

> On 13 Sep 2021, at 12:19, Jack Ye <ye...@gmail.com> wrote:
> 
> I'd like to also propose adding the following in the external section:
> 1. the PrestoDB equivalent for each item listed for Trino. I am not sure what's the best way to track them, but I feel it's better to list and track them separately. I have talked with related people currently maintaining the PrestoDB Iceberg connector (mostly in Twitter), and they would like to take a different route from Trino to fully remove Hive dependencies in the connector. This means the 2 connectors will likely diverge in implementation in the near future.
> 2. adding a medium item for Trino and PrestoDB Avro support
> 3. adding a small item for Trino and PrestoDB full system table support (the system table schema in them are diverging from core, and missing a few latest system tables)
> 
> For the items listed with "Spec" and "Spec v3", what are the key differences? I thought we are treating any new spec changes after the format v2 vote as v3.
> 
> Best,
> Jack Ye
> 
> On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg5070@gmail.com <ma...@gmail.com>> wrote:
> Hi Ryan,
> 
> I just wonder if the encryption should be a Spec v3 category. We have the key_metadata fields in both data_file and manifest_file structs, which might be sufficient for a reasonable basic encryption support.
> But I certainly agree this is an L-sized project.
> 
> Cheers, Gidon
> 
> 
> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <blue@tabular.io <ma...@tabular.io>> wrote:
> Hi everyone,
> 
> At the last sync meeting, we brought up publishing a community roadmap and brainstormed the many features and initiatives that the community is working on. In this thread, I want to make sure that we have a good list of what people are thinking about and I think we should try to categorize the projects by size and general priority. When we reach a rough agreement, I’ll write this up and post it on the ASF site along with links to some projects in Github.
> 
> My rationale for attempting to prioritize projects is that if we try to do too many things, it will be slower progress across everything rather than getting a few important items done. I know that priorities don’t align very cleanly in practice, but it is hopefully worth trying. To come up with a priority, I’m trying to keep top priority items to a minimum by including only one from each group (Spark, Flink, Python, etc.). The remaining items are split between priority 2 and 3. Priority 3 is not urgent, including things that can be plugged in (like other IO libraries), docs, etc. Everything else is priority 2.
> 
> That something isn’t priority 1 doesn’t mean it isn’t important or progressing, just that it isn’t the current focus. I think of it this way: if someone has extra time to review something, what should be next? That’s top priority.
> 
> Here’s my rough categorization. If you disagree, please speak up:
> 
> If you think that something should be top priority, what gets moved to priority 2?
> Should the priority for a project in 2 or 3 change?
> Is the S/M/L size of a project wrong?
> Top priority, 1:
> 
> API: Iceberg 1.0 [medium]
> Spark: Merge-on-read plans [large]
> Maintenance: Delete file compaction [medium]
> Flink: Upgrade to 1.13.2 (document compatibility) [medium]
> Python: Pythonic refactor [medium]
> Priority 2:
> 
> ORC: Support delete files stored as ORC [small]
> Spark: DSv2 streaming improvements [small]
> Flink: Inline file compaction [small]
> Flink: Support UPSERT [small]
> Views: Spec [medium]
> Spec: Z-ordering / Space-filling curves [medium]
> Spec: Snapshot tagging and branching [small]
> Spec: Secondary indexes [large]
> Spec v3: Encryption [large]
> Spec v3: Relative paths [large]
> Spec v3: Default field values [medium]
> Priority 3:
> 
> Docs: versioned docs [medium]
> IO: Support Aliyun OSS/DLF [medium]
> IO: Support Dell ECS [medium]
> External:
> 
> Trino: Bucketed joins [small]
> Trino: Row-level delete support [medium]
> Trino: Merge-on-read plans [medium]
> Trino: Multi-catalog support [small]
> -- 
> Ryan Blue
> Tabular

Re: [DISCUSS] Iceberg roadmap

[Jack Ye <ye...@gmail.com>]

I'd like to also propose adding the following in the external section:
1. the PrestoDB equivalent for each item listed for Trino. I am not sure
what's the best way to track them, but I feel it's better to list and track
them separately. I have talked with related people currently maintaining
the PrestoDB Iceberg connector (mostly in Twitter), and they would like to
take a different route from Trino to fully remove Hive dependencies in the
connector. This means the 2 connectors will likely diverge in
implementation in the near future.
2. adding a medium item for Trino and PrestoDB Avro support
3. adding a small item for Trino and PrestoDB full system table support
(the system table schema in them are diverging from core, and missing a few
latest system tables)

For the items listed with "Spec" and "Spec v3", what are the key
differences? I thought we are treating any new spec changes after the
format v2 vote as v3.

Best,
Jack Ye

On Mon, Sep 13, 2021 at 7:13 AM Gidon Gershinsky <gg...@gmail.com> wrote:

> Hi Ryan,
>
> I just wonder if the encryption should be a Spec v3 category. We have the
> key_metadata fields in both data_file and manifest_file structs, which
> might be sufficient for a reasonable basic encryption support.
> But I certainly agree this is an L-sized project.
>
> Cheers, Gidon
>
>
> On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:
>
>> Hi everyone,
>>
>> At the last sync meeting, we brought up publishing a community roadmap
>> and brainstormed the many features and initiatives that the community is
>> working on. In this thread, I want to make sure that we have a good list of
>> what people are thinking about and I think we should try to categorize the
>> projects by size and general priority. When we reach a rough agreement,
>> I’ll write this up and post it on the ASF site along with links to some
>> projects in Github.
>>
>> My rationale for attempting to prioritize projects is that if we try to
>> do too many things, it will be slower progress across everything rather
>> than getting a few important items done. I know that priorities don’t align
>> very cleanly in practice, but it is hopefully worth trying. To come up with
>> a priority, I’m trying to keep top priority items to a minimum by including
>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>> are split between priority 2 and 3. Priority 3 is not urgent, including
>> things that can be plugged in (like other IO libraries), docs, etc.
>> Everything else is priority 2.
>>
>> That something isn’t priority 1 doesn’t mean it isn’t important or
>> progressing, just that it isn’t the current focus. I think of it this way:
>> if someone has extra time to review something, what should be next? That’s
>> top priority.
>>
>> Here’s my rough categorization. If you disagree, please speak up:
>>
>>    - If you think that something should be top priority, what gets moved
>>    to priority 2?
>>    - Should the priority for a project in 2 or 3 change?
>>    - Is the S/M/L size of a project wrong?
>>
>> Top priority, 1:
>>
>>    - API: Iceberg 1.0 [medium]
>>    - Spark: Merge-on-read plans [large]
>>    - Maintenance: Delete file compaction [medium]
>>    -
>>
>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>    -
>>
>>    Python: Pythonic refactor [medium]
>>
>> Priority 2:
>>
>>    - ORC: Support delete files stored as ORC [small]
>>    - Spark: DSv2 streaming improvements [small]
>>    - Flink: Inline file compaction [small]
>>    - Flink: Support UPSERT [small]
>>    - Views: Spec [medium]
>>    - Spec: Z-ordering / Space-filling curves [medium]
>>    - Spec: Snapshot tagging and branching [small]
>>    - Spec: Secondary indexes [large]
>>    - Spec v3: Encryption [large]
>>    -
>>
>>    Spec v3: Relative paths [large]
>>    -
>>
>>    Spec v3: Default field values [medium]
>>
>> Priority 3:
>>
>>    - Docs: versioned docs [medium]
>>    - IO: Support Aliyun OSS/DLF [medium]
>>    - IO: Support Dell ECS [medium]
>>
>> External:
>>
>>    - Trino: Bucketed joins [small]
>>    - Trino: Row-level delete support [medium]
>>    - Trino: Merge-on-read plans [medium]
>>    - Trino: Multi-catalog support [small]
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[Gidon Gershinsky <gg...@gmail.com>]

Hi Ryan,

I just wonder if the encryption should be a Spec v3 category. We have the
key_metadata fields in both data_file and manifest_file structs, which
might be sufficient for a reasonable basic encryption support.
But I certainly agree this is an L-sized project.

Cheers, Gidon


On Sat, Sep 11, 2021 at 12:38 AM Ryan Blue <bl...@tabular.io> wrote:

> Hi everyone,
>
> At the last sync meeting, we brought up publishing a community roadmap and
> brainstormed the many features and initiatives that the community is
> working on. In this thread, I want to make sure that we have a good list of
> what people are thinking about and I think we should try to categorize the
> projects by size and general priority. When we reach a rough agreement,
> I’ll write this up and post it on the ASF site along with links to some
> projects in Github.
>
> My rationale for attempting to prioritize projects is that if we try to do
> too many things, it will be slower progress across everything rather than
> getting a few important items done. I know that priorities don’t align very
> cleanly in practice, but it is hopefully worth trying. To come up with a
> priority, I’m trying to keep top priority items to a minimum by including
> only one from each group (Spark, Flink, Python, etc.). The remaining items
> are split between priority 2 and 3. Priority 3 is not urgent, including
> things that can be plugged in (like other IO libraries), docs, etc.
> Everything else is priority 2.
>
> That something isn’t priority 1 doesn’t mean it isn’t important or
> progressing, just that it isn’t the current focus. I think of it this way:
> if someone has extra time to review something, what should be next? That’s
> top priority.
>
> Here’s my rough categorization. If you disagree, please speak up:
>
>    - If you think that something should be top priority, what gets moved
>    to priority 2?
>    - Should the priority for a project in 2 or 3 change?
>    - Is the S/M/L size of a project wrong?
>
> Top priority, 1:
>
>    - API: Iceberg 1.0 [medium]
>    - Spark: Merge-on-read plans [large]
>    - Maintenance: Delete file compaction [medium]
>    -
>
>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>    -
>
>    Python: Pythonic refactor [medium]
>
> Priority 2:
>
>    - ORC: Support delete files stored as ORC [small]
>    - Spark: DSv2 streaming improvements [small]
>    - Flink: Inline file compaction [small]
>    - Flink: Support UPSERT [small]
>    - Views: Spec [medium]
>    - Spec: Z-ordering / Space-filling curves [medium]
>    - Spec: Snapshot tagging and branching [small]
>    - Spec: Secondary indexes [large]
>    - Spec v3: Encryption [large]
>    -
>
>    Spec v3: Relative paths [large]
>    -
>
>    Spec v3: Default field values [medium]
>
> Priority 3:
>
>    - Docs: versioned docs [medium]
>    - IO: Support Aliyun OSS/DLF [medium]
>    - IO: Support Dell ECS [medium]
>
> External:
>
>    - Trino: Bucketed joins [small]
>    - Trino: Row-level delete support [medium]
>    - Trino: Merge-on-read plans [medium]
>    - Trino: Multi-catalog support [small]
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Steven, I thought that this was part of upgrading Flink to 1.13.2. Is that
correct?

On Fri, Sep 10, 2021 at 4:40 PM Steven Wu <st...@gmail.com> wrote:

> I would like to add a item
>
> Priority 2:
> Flink: FLIP-27 based Iceberg source [large]
>
> On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Hi everyone,
>>
>> At the last sync meeting, we brought up publishing a community roadmap
>> and brainstormed the many features and initiatives that the community is
>> working on. In this thread, I want to make sure that we have a good list of
>> what people are thinking about and I think we should try to categorize the
>> projects by size and general priority. When we reach a rough agreement,
>> I’ll write this up and post it on the ASF site along with links to some
>> projects in Github.
>>
>> My rationale for attempting to prioritize projects is that if we try to
>> do too many things, it will be slower progress across everything rather
>> than getting a few important items done. I know that priorities don’t align
>> very cleanly in practice, but it is hopefully worth trying. To come up with
>> a priority, I’m trying to keep top priority items to a minimum by including
>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>> are split between priority 2 and 3. Priority 3 is not urgent, including
>> things that can be plugged in (like other IO libraries), docs, etc.
>> Everything else is priority 2.
>>
>> That something isn’t priority 1 doesn’t mean it isn’t important or
>> progressing, just that it isn’t the current focus. I think of it this way:
>> if someone has extra time to review something, what should be next? That’s
>> top priority.
>>
>> Here’s my rough categorization. If you disagree, please speak up:
>>
>>    - If you think that something should be top priority, what gets moved
>>    to priority 2?
>>    - Should the priority for a project in 2 or 3 change?
>>    - Is the S/M/L size of a project wrong?
>>
>> Top priority, 1:
>>
>>    - API: Iceberg 1.0 [medium]
>>    - Spark: Merge-on-read plans [large]
>>    - Maintenance: Delete file compaction [medium]
>>    -
>>
>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>    -
>>
>>    Python: Pythonic refactor [medium]
>>
>> Priority 2:
>>
>>    - ORC: Support delete files stored as ORC [small]
>>    - Spark: DSv2 streaming improvements [small]
>>    - Flink: Inline file compaction [small]
>>    - Flink: Support UPSERT [small]
>>    - Views: Spec [medium]
>>    - Spec: Z-ordering / Space-filling curves [medium]
>>    - Spec: Snapshot tagging and branching [small]
>>    - Spec: Secondary indexes [large]
>>    - Spec v3: Encryption [large]
>>    -
>>
>>    Spec v3: Relative paths [large]
>>    -
>>
>>    Spec v3: Default field values [medium]
>>
>> Priority 3:
>>
>>    - Docs: versioned docs [medium]
>>    - IO: Support Aliyun OSS/DLF [medium]
>>    - IO: Support Dell ECS [medium]
>>
>> External:
>>
>>    - Trino: Bucketed joins [small]
>>    - Trino: Row-level delete support [medium]
>>    - Trino: Merge-on-read plans [medium]
>>    - Trino: Multi-catalog support [small]
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Steven Wu <st...@gmail.com>]

Ryan, FLIP-27 Flink source is not part of the Flink 1.13.2 upgrade,
although Flink 1.13.2 is preferred for some bug fix.

On Sun, Sep 12, 2021 at 9:04 AM Ryan Blue <bl...@tabular.io> wrote:

> Szehon, I'm not sure that I'd consider either of those for the roadmap.
> While they are important to work on, they seem more like individual PRs
> than high-level projects that might span PRs. This is definitely a good
> thing to consider, though. Should we go more granular than the list that
> we've compiled?
>
> On Fri, Sep 10, 2021 at 5:07 PM Szehon Ho <sz...@gmail.com> wrote:
>
>> Hi
>>
>> I also missed the last sync, and wanted to add two things if possible.
>>
>> Thanks,
>> Szehon
>>
>> Priority 2:
>>
>>    - Core: Predicate pushdown for remaining Metadata tables [medium]
>>    - Core/Spark: Support serializable isolation for ReplacePartitions /
>>    Insert Overwrite [medium]
>>
>>
>> On Fri, Sep 10, 2021 at 4:40 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> I would like to add a item
>>>
>>> Priority 2:
>>> Flink: FLIP-27 based Iceberg source [large]
>>>
>>> On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> At the last sync meeting, we brought up publishing a community roadmap
>>>> and brainstormed the many features and initiatives that the community is
>>>> working on. In this thread, I want to make sure that we have a good list of
>>>> what people are thinking about and I think we should try to categorize the
>>>> projects by size and general priority. When we reach a rough agreement,
>>>> I’ll write this up and post it on the ASF site along with links to some
>>>> projects in Github.
>>>>
>>>> My rationale for attempting to prioritize projects is that if we try to
>>>> do too many things, it will be slower progress across everything rather
>>>> than getting a few important items done. I know that priorities don’t align
>>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>>> a priority, I’m trying to keep top priority items to a minimum by including
>>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>>> things that can be plugged in (like other IO libraries), docs, etc.
>>>> Everything else is priority 2.
>>>>
>>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>>> progressing, just that it isn’t the current focus. I think of it this way:
>>>> if someone has extra time to review something, what should be next? That’s
>>>> top priority.
>>>>
>>>> Here’s my rough categorization. If you disagree, please speak up:
>>>>
>>>>    - If you think that something should be top priority, what gets
>>>>    moved to priority 2?
>>>>    - Should the priority for a project in 2 or 3 change?
>>>>    - Is the S/M/L size of a project wrong?
>>>>
>>>> Top priority, 1:
>>>>
>>>>    - API: Iceberg 1.0 [medium]
>>>>    - Spark: Merge-on-read plans [large]
>>>>    - Maintenance: Delete file compaction [medium]
>>>>    -
>>>>
>>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>>    -
>>>>
>>>>    Python: Pythonic refactor [medium]
>>>>
>>>> Priority 2:
>>>>
>>>>    - ORC: Support delete files stored as ORC [small]
>>>>    - Spark: DSv2 streaming improvements [small]
>>>>    - Flink: Inline file compaction [small]
>>>>    - Flink: Support UPSERT [small]
>>>>    - Views: Spec [medium]
>>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>>    - Spec: Snapshot tagging and branching [small]
>>>>    - Spec: Secondary indexes [large]
>>>>    - Spec v3: Encryption [large]
>>>>    -
>>>>
>>>>    Spec v3: Relative paths [large]
>>>>    -
>>>>
>>>>    Spec v3: Default field values [medium]
>>>>
>>>> Priority 3:
>>>>
>>>>    - Docs: versioned docs [medium]
>>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>>    - IO: Support Dell ECS [medium]
>>>>
>>>> External:
>>>>
>>>>    - Trino: Bucketed joins [small]
>>>>    - Trino: Row-level delete support [medium]
>>>>    - Trino: Merge-on-read plans [medium]
>>>>    - Trino: Multi-catalog support [small]
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>
> --
> Ryan Blue
> Tabular
>

Re: [DISCUSS] Iceberg roadmap

[Ryan Blue <bl...@tabular.io>]

Szehon, I'm not sure that I'd consider either of those for the roadmap.
While they are important to work on, they seem more like individual PRs
than high-level projects that might span PRs. This is definitely a good
thing to consider, though. Should we go more granular than the list that
we've compiled?

On Fri, Sep 10, 2021 at 5:07 PM Szehon Ho <sz...@gmail.com> wrote:

> Hi
>
> I also missed the last sync, and wanted to add two things if possible.
>
> Thanks,
> Szehon
>
> Priority 2:
>
>    - Core: Predicate pushdown for remaining Metadata tables [medium]
>    - Core/Spark: Support serializable isolation for ReplacePartitions /
>    Insert Overwrite [medium]
>
>
> On Fri, Sep 10, 2021 at 4:40 PM Steven Wu <st...@gmail.com> wrote:
>
>> I would like to add a item
>>
>> Priority 2:
>> Flink: FLIP-27 based Iceberg source [large]
>>
>> On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Hi everyone,
>>>
>>> At the last sync meeting, we brought up publishing a community roadmap
>>> and brainstormed the many features and initiatives that the community is
>>> working on. In this thread, I want to make sure that we have a good list of
>>> what people are thinking about and I think we should try to categorize the
>>> projects by size and general priority. When we reach a rough agreement,
>>> I’ll write this up and post it on the ASF site along with links to some
>>> projects in Github.
>>>
>>> My rationale for attempting to prioritize projects is that if we try to
>>> do too many things, it will be slower progress across everything rather
>>> than getting a few important items done. I know that priorities don’t align
>>> very cleanly in practice, but it is hopefully worth trying. To come up with
>>> a priority, I’m trying to keep top priority items to a minimum by including
>>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>>> are split between priority 2 and 3. Priority 3 is not urgent, including
>>> things that can be plugged in (like other IO libraries), docs, etc.
>>> Everything else is priority 2.
>>>
>>> That something isn’t priority 1 doesn’t mean it isn’t important or
>>> progressing, just that it isn’t the current focus. I think of it this way:
>>> if someone has extra time to review something, what should be next? That’s
>>> top priority.
>>>
>>> Here’s my rough categorization. If you disagree, please speak up:
>>>
>>>    - If you think that something should be top priority, what gets
>>>    moved to priority 2?
>>>    - Should the priority for a project in 2 or 3 change?
>>>    - Is the S/M/L size of a project wrong?
>>>
>>> Top priority, 1:
>>>
>>>    - API: Iceberg 1.0 [medium]
>>>    - Spark: Merge-on-read plans [large]
>>>    - Maintenance: Delete file compaction [medium]
>>>    -
>>>
>>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>>    -
>>>
>>>    Python: Pythonic refactor [medium]
>>>
>>> Priority 2:
>>>
>>>    - ORC: Support delete files stored as ORC [small]
>>>    - Spark: DSv2 streaming improvements [small]
>>>    - Flink: Inline file compaction [small]
>>>    - Flink: Support UPSERT [small]
>>>    - Views: Spec [medium]
>>>    - Spec: Z-ordering / Space-filling curves [medium]
>>>    - Spec: Snapshot tagging and branching [small]
>>>    - Spec: Secondary indexes [large]
>>>    - Spec v3: Encryption [large]
>>>    -
>>>
>>>    Spec v3: Relative paths [large]
>>>    -
>>>
>>>    Spec v3: Default field values [medium]
>>>
>>> Priority 3:
>>>
>>>    - Docs: versioned docs [medium]
>>>    - IO: Support Aliyun OSS/DLF [medium]
>>>    - IO: Support Dell ECS [medium]
>>>
>>> External:
>>>
>>>    - Trino: Bucketed joins [small]
>>>    - Trino: Row-level delete support [medium]
>>>    - Trino: Merge-on-read plans [medium]
>>>    - Trino: Multi-catalog support [small]
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Iceberg roadmap

[Szehon Ho <sz...@gmail.com>]

Hi

I also missed the last sync, and wanted to add two things if possible.

Thanks,
Szehon

Priority 2:

   - Core: Predicate pushdown for remaining Metadata tables [medium]
   - Core/Spark: Support serializable isolation for ReplacePartitions /
   Insert Overwrite [medium]


On Fri, Sep 10, 2021 at 4:40 PM Steven Wu <st...@gmail.com> wrote:

> I would like to add a item
>
> Priority 2:
> Flink: FLIP-27 based Iceberg source [large]
>
> On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Hi everyone,
>>
>> At the last sync meeting, we brought up publishing a community roadmap
>> and brainstormed the many features and initiatives that the community is
>> working on. In this thread, I want to make sure that we have a good list of
>> what people are thinking about and I think we should try to categorize the
>> projects by size and general priority. When we reach a rough agreement,
>> I’ll write this up and post it on the ASF site along with links to some
>> projects in Github.
>>
>> My rationale for attempting to prioritize projects is that if we try to
>> do too many things, it will be slower progress across everything rather
>> than getting a few important items done. I know that priorities don’t align
>> very cleanly in practice, but it is hopefully worth trying. To come up with
>> a priority, I’m trying to keep top priority items to a minimum by including
>> only one from each group (Spark, Flink, Python, etc.). The remaining items
>> are split between priority 2 and 3. Priority 3 is not urgent, including
>> things that can be plugged in (like other IO libraries), docs, etc.
>> Everything else is priority 2.
>>
>> That something isn’t priority 1 doesn’t mean it isn’t important or
>> progressing, just that it isn’t the current focus. I think of it this way:
>> if someone has extra time to review something, what should be next? That’s
>> top priority.
>>
>> Here’s my rough categorization. If you disagree, please speak up:
>>
>>    - If you think that something should be top priority, what gets moved
>>    to priority 2?
>>    - Should the priority for a project in 2 or 3 change?
>>    - Is the S/M/L size of a project wrong?
>>
>> Top priority, 1:
>>
>>    - API: Iceberg 1.0 [medium]
>>    - Spark: Merge-on-read plans [large]
>>    - Maintenance: Delete file compaction [medium]
>>    -
>>
>>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>>    -
>>
>>    Python: Pythonic refactor [medium]
>>
>> Priority 2:
>>
>>    - ORC: Support delete files stored as ORC [small]
>>    - Spark: DSv2 streaming improvements [small]
>>    - Flink: Inline file compaction [small]
>>    - Flink: Support UPSERT [small]
>>    - Views: Spec [medium]
>>    - Spec: Z-ordering / Space-filling curves [medium]
>>    - Spec: Snapshot tagging and branching [small]
>>    - Spec: Secondary indexes [large]
>>    - Spec v3: Encryption [large]
>>    -
>>
>>    Spec v3: Relative paths [large]
>>    -
>>
>>    Spec v3: Default field values [medium]
>>
>> Priority 3:
>>
>>    - Docs: versioned docs [medium]
>>    - IO: Support Aliyun OSS/DLF [medium]
>>    - IO: Support Dell ECS [medium]
>>
>> External:
>>
>>    - Trino: Bucketed joins [small]
>>    - Trino: Row-level delete support [medium]
>>    - Trino: Merge-on-read plans [medium]
>>    - Trino: Multi-catalog support [small]
>>
>> --
>> Ryan Blue
>> Tabular
>>
>

Re: [DISCUSS] Iceberg roadmap

[Steven Wu <st...@gmail.com>]

I would like to add a item

Priority 2:
Flink: FLIP-27 based Iceberg source [large]

On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue <bl...@tabular.io> wrote:

> Hi everyone,
>
> At the last sync meeting, we brought up publishing a community roadmap and
> brainstormed the many features and initiatives that the community is
> working on. In this thread, I want to make sure that we have a good list of
> what people are thinking about and I think we should try to categorize the
> projects by size and general priority. When we reach a rough agreement,
> I’ll write this up and post it on the ASF site along with links to some
> projects in Github.
>
> My rationale for attempting to prioritize projects is that if we try to do
> too many things, it will be slower progress across everything rather than
> getting a few important items done. I know that priorities don’t align very
> cleanly in practice, but it is hopefully worth trying. To come up with a
> priority, I’m trying to keep top priority items to a minimum by including
> only one from each group (Spark, Flink, Python, etc.). The remaining items
> are split between priority 2 and 3. Priority 3 is not urgent, including
> things that can be plugged in (like other IO libraries), docs, etc.
> Everything else is priority 2.
>
> That something isn’t priority 1 doesn’t mean it isn’t important or
> progressing, just that it isn’t the current focus. I think of it this way:
> if someone has extra time to review something, what should be next? That’s
> top priority.
>
> Here’s my rough categorization. If you disagree, please speak up:
>
>    - If you think that something should be top priority, what gets moved
>    to priority 2?
>    - Should the priority for a project in 2 or 3 change?
>    - Is the S/M/L size of a project wrong?
>
> Top priority, 1:
>
>    - API: Iceberg 1.0 [medium]
>    - Spark: Merge-on-read plans [large]
>    - Maintenance: Delete file compaction [medium]
>    -
>
>    Flink: Upgrade to 1.13.2 (document compatibility) [medium]
>    -
>
>    Python: Pythonic refactor [medium]
>
> Priority 2:
>
>    - ORC: Support delete files stored as ORC [small]
>    - Spark: DSv2 streaming improvements [small]
>    - Flink: Inline file compaction [small]
>    - Flink: Support UPSERT [small]
>    - Views: Spec [medium]
>    - Spec: Z-ordering / Space-filling curves [medium]
>    - Spec: Snapshot tagging and branching [small]
>    - Spec: Secondary indexes [large]
>    - Spec v3: Encryption [large]
>    -
>
>    Spec v3: Relative paths [large]
>    -
>
>    Spec v3: Default field values [medium]
>
> Priority 3:
>
>    - Docs: versioned docs [medium]
>    - IO: Support Aliyun OSS/DLF [medium]
>    - IO: Support Dell ECS [medium]
>
> External:
>
>    - Trino: Bucketed joins [small]
>    - Trino: Row-level delete support [medium]
>    - Trino: Merge-on-read plans [medium]
>    - Trino: Multi-catalog support [small]
>
> --
> Ryan Blue
> Tabular
>