[DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi folks,

We would like to propose a new project to the ASF incubator: Polaris.

Polaris is a catalog for data lakes. It provides new levels of choice,
flexibility and control over data, with full enterprise security and
Apache Iceberg interoperability across a multitude of engines and
infrastructure. Polaris builds on standards such as those created by
Apache Iceberg, providing the following benefits for the ecosystem:
* Multi-engine interoperability over a single copy of data,
eliminating the need for moving and copying data across different
engines and catalogs.
* An interoperable security model providing a unified authorization
layer independent from the engines processing analytical tables.
* For multi-catalog scenarios, a unified catalog level view of data
across multiple catalogs via catalog notification integrations.
* The ability to host Polaris Catalog on the infrastructure of your choice.

Here is the proposal:
https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal

Comments and feedback are welcome.

Thanks!
Regards
JB

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Wangda Tan <wa...@apache.org>]

5 mentors are awesome! Excited to see this kind of support. 

Cheers, 
Wangda

On 2024/08/01 13:21:09 Jean-Baptiste Onofré wrote:
> Hi Wangda,
> 
> Thanks for your message and your proposal. Now, we already have 5
> mentors on the proposal. For good execution/mentoring, I think it's
> largely enough. I will let know you if one mentor wants to be
> replaced.
> 
> Thanks again!
> 
> Regards
> JB
> 
> On Thu, Aug 1, 2024 at 2:43 AM Wangda Tan <wa...@apache.org> wrote:
> >
> > It is a super interesting project, I'm +1 on Polaris joining the Apache incubator. I'd be willing to offer guidance or be a mentor if needed.
> >
> > Best,
> > Wangda Tan
> >
> > On 2024/07/30 17:33:57 Jean-Baptiste Onofré wrote:
> > > Hi folks,
> > >
> > > We would like to propose a new project to the ASF incubator: Polaris.
> > >
> > > Polaris is a catalog for data lakes. It provides new levels of choice,
> > > flexibility and control over data, with full enterprise security and
> > > Apache Iceberg interoperability across a multitude of engines and
> > > infrastructure. Polaris builds on standards such as those created by
> > > Apache Iceberg, providing the following benefits for the ecosystem:
> > > * Multi-engine interoperability over a single copy of data,
> > > eliminating the need for moving and copying data across different
> > > engines and catalogs.
> > > * An interoperable security model providing a unified authorization
> > > layer independent from the engines processing analytical tables.
> > > * For multi-catalog scenarios, a unified catalog level view of data
> > > across multiple catalogs via catalog notification integrations.
> > > * The ability to host Polaris Catalog on the infrastructure of your choice.
> > >
> > > Here is the proposal:
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > >
> > > Comments and feedback are welcome.
> > >
> > > Thanks!
> > > Regards
> > > JB
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Wangda,

Thanks for your message and your proposal. Now, we already have 5
mentors on the proposal. For good execution/mentoring, I think it's
largely enough. I will let know you if one mentor wants to be
replaced.

Thanks again!

Regards
JB

On Thu, Aug 1, 2024 at 2:43 AM Wangda Tan <wa...@apache.org> wrote:
>
> It is a super interesting project, I'm +1 on Polaris joining the Apache incubator. I'd be willing to offer guidance or be a mentor if needed.
>
> Best,
> Wangda Tan
>
> On 2024/07/30 17:33:57 Jean-Baptiste Onofré wrote:
> > Hi folks,
> >
> > We would like to propose a new project to the ASF incubator: Polaris.
> >
> > Polaris is a catalog for data lakes. It provides new levels of choice,
> > flexibility and control over data, with full enterprise security and
> > Apache Iceberg interoperability across a multitude of engines and
> > infrastructure. Polaris builds on standards such as those created by
> > Apache Iceberg, providing the following benefits for the ecosystem:
> > * Multi-engine interoperability over a single copy of data,
> > eliminating the need for moving and copying data across different
> > engines and catalogs.
> > * An interoperable security model providing a unified authorization
> > layer independent from the engines processing analytical tables.
> > * For multi-catalog scenarios, a unified catalog level view of data
> > across multiple catalogs via catalog notification integrations.
> > * The ability to host Polaris Catalog on the infrastructure of your choice.
> >
> > Here is the proposal:
> > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >
> > Comments and feedback are welcome.
> >
> > Thanks!
> > Regards
> > JB
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Wangda Tan <wa...@apache.org>]

It is a super interesting project, I'm +1 on Polaris joining the Apache incubator. I'd be willing to offer guidance or be a mentor if needed.

Best,
Wangda Tan

On 2024/07/30 17:33:57 Jean-Baptiste Onofré wrote:
> Hi folks,
> 
> We would like to propose a new project to the ASF incubator: Polaris.
> 
> Polaris is a catalog for data lakes. It provides new levels of choice,
> flexibility and control over data, with full enterprise security and
> Apache Iceberg interoperability across a multitude of engines and
> infrastructure. Polaris builds on standards such as those created by
> Apache Iceberg, providing the following benefits for the ecosystem:
> * Multi-engine interoperability over a single copy of data,
> eliminating the need for moving and copying data across different
> engines and catalogs.
> * An interoperable security model providing a unified authorization
> layer independent from the engines processing analytical tables.
> * For multi-catalog scenarios, a unified catalog level view of data
> across multiple catalogs via catalog notification integrations.
> * The ability to host Polaris Catalog on the infrastructure of your choice.
> 
> Here is the proposal:
> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> 
> Comments and feedback are welcome.
> 
> Thanks!
> Regards
> JB
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[ShaoFeng Shi <sh...@apache.org>]

Glad to see Polaris to join Apache incubator, together with Iceberg and
other ecosystems.

As a mentor of Gravitino project, I think the two projects have different
visions and can work together: Gravitino is an open data catalog for
various data sources including data lake formats, and Polaris is an Iceberg
REST catalog (at least today) with enterprise-level security control over
different cloud storages. They server different purposes and can work
together.

Good luck to Polaris!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC member,
Apache Incubator PMC member,
Email: shaofengshi@apache.org






Nicholas Jiang <ni...@apache.org> 于2024年8月1日周四 11:41写道：

> Hi JB,
>
>
>
>
> Thanks for driving the proposal of Polaris. I have some questions about
> the PolarisProposal as follows:
>
>
>
>
> 1. Is Polaris positioning oriented towards a unified catalog of data lakes
> and computing engines, which has mentioned as "Polaris is a catalog for
> data lakes" and "Polaris is connected with Apache Iceberg, Apache Hadoop,
> Apache Spark, Apache Flink, and potentially Apache Hudi, Apache XTable, …"?
>
>
>
>
> 2. If Polaris is only a catalog for data lakes, why not directly provide a
> catalog wrapper and implementaion of Gravitino catalog in Gravitino project?
>
>
>
>
> Regards,
>
> Nicholas Jiang
>
>
> At 2024-07-31 01:33:57, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
> >Hi folks,
> >
> >We would like to propose a new project to the ASF incubator: Polaris.
> >
> >Polaris is a catalog for data lakes. It provides new levels of choice,
> >flexibility and control over data, with full enterprise security and
> >Apache Iceberg interoperability across a multitude of engines and
> >infrastructure. Polaris builds on standards such as those created by
> >Apache Iceberg, providing the following benefits for the ecosystem:
> >* Multi-engine interoperability over a single copy of data,
> >eliminating the need for moving and copying data across different
> >engines and catalogs.
> >* An interoperable security model providing a unified authorization
> >layer independent from the engines processing analytical tables.
> >* For multi-catalog scenarios, a unified catalog level view of data
> >across multiple catalogs via catalog notification integrations.
> >* The ability to host Polaris Catalog on the infrastructure of your
> choice.
> >
> >Here is the proposal:
> >https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >
> >Comments and feedback are welcome.
> >
> >Thanks!
> >Regards
> >JB
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >For additional commands, e-mail: general-help@incubator.apache.org
>

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Nicholas,

I think Gravitino and Polaris are complementary by addressing
different use cases.

Gravitino is focusing on metadata lake with metadata features
(metadata management, metadata governance, geo-distribution, AI asset
management).

Polaris is focusing on Apache Iceberg REST Catalog implementation,
with associated features (RBAC, auth, etc).

So, I see extensions from one to another (in both ways) but not the
same use cases.

Regards
JB

On Thu, Aug 1, 2024 at 5:41 AM Nicholas Jiang <ni...@apache.org> wrote:
>
> Hi JB,
>
>
>
>
> Thanks for driving the proposal of Polaris. I have some questions about the PolarisProposal as follows:
>
>
>
>
> 1. Is Polaris positioning oriented towards a unified catalog of data lakes and computing engines, which has mentioned as "Polaris is a catalog for data lakes" and "Polaris is connected with Apache Iceberg, Apache Hadoop, Apache Spark, Apache Flink, and potentially Apache Hudi, Apache XTable, …"?
>
>
>
>
> 2. If Polaris is only a catalog for data lakes, why not directly provide a catalog wrapper and implementaion of Gravitino catalog in Gravitino project?
>
>
>
>
> Regards,
>
> Nicholas Jiang
>
>
> At 2024-07-31 01:33:57, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
> >Hi folks,
> >
> >We would like to propose a new project to the ASF incubator: Polaris.
> >
> >Polaris is a catalog for data lakes. It provides new levels of choice,
> >flexibility and control over data, with full enterprise security and
> >Apache Iceberg interoperability across a multitude of engines and
> >infrastructure. Polaris builds on standards such as those created by
> >Apache Iceberg, providing the following benefits for the ecosystem:
> >* Multi-engine interoperability over a single copy of data,
> >eliminating the need for moving and copying data across different
> >engines and catalogs.
> >* An interoperable security model providing a unified authorization
> >layer independent from the engines processing analytical tables.
> >* For multi-catalog scenarios, a unified catalog level view of data
> >across multiple catalogs via catalog notification integrations.
> >* The ability to host Polaris Catalog on the infrastructure of your choice.
> >
> >Here is the proposal:
> >https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >
> >Comments and feedback are welcome.
> >
> >Thanks!
> >Regards
> >JB
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >For additional commands, e-mail: general-help@incubator.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re:[DISCUSS] PolarisProposal to the incubator

[Nicholas Jiang <ni...@apache.org>]

Hi JB,




Thanks for driving the proposal of Polaris. I have some questions about the PolarisProposal as follows:




1. Is Polaris positioning oriented towards a unified catalog of data lakes and computing engines, which has mentioned as "Polaris is a catalog for data lakes" and "Polaris is connected with Apache Iceberg, Apache Hadoop, Apache Spark, Apache Flink, and potentially Apache Hudi, Apache XTable, …"?




2. If Polaris is only a catalog for data lakes, why not directly provide a catalog wrapper and implementaion of Gravitino catalog in Gravitino project?




Regards,

Nicholas Jiang


At 2024-07-31 01:33:57, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
>Hi folks,
>
>We would like to propose a new project to the ASF incubator: Polaris.
>
>Polaris is a catalog for data lakes. It provides new levels of choice,
>flexibility and control over data, with full enterprise security and
>Apache Iceberg interoperability across a multitude of engines and
>infrastructure. Polaris builds on standards such as those created by
>Apache Iceberg, providing the following benefits for the ecosystem:
>* Multi-engine interoperability over a single copy of data,
>eliminating the need for moving and copying data across different
>engines and catalogs.
>* An interoperable security model providing a unified authorization
>layer independent from the engines processing analytical tables.
>* For multi-catalog scenarios, a unified catalog level view of data
>across multiple catalogs via catalog notification integrations.
>* The ability to host Polaris Catalog on the infrastructure of your choice.
>
>Here is the proposal:
>https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
>
>Comments and feedback are welcome.
>
>Thanks!
>Regards
>JB
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi David,

I guess Justin meant "a lot of the proposed PPMC members".

Regards
JB

On Wed, Jul 31, 2024 at 6:11 PM David Jencks <da...@gmail.com> wrote:
>
> "a lot of the proposed PMCs”
>
> There are a lot of projects mentioned in this thread, and I think this is genuinely ambiguous. Do you mean PMCs or PMC members?
>
> David Jencks
>
> > On Jul 30, 2024, at 8:47 PM, Justin Mclean <ju...@classsoftware.com> wrote:
> >
> > Hi,
> >
> > In general, projects joining the incubator need two things:
> > - A codebase
> > - a community around it
> >
> > While this isn't a hard requirement, building either while in incubation can be difficult.
> >
> > The first issue I see is it’s not clear to me that there is a community here.
> >
> > The split between PPMC / committers is also unusual.
> >
> > It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed?
> >
> > Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
> > anoopj - 1 commit (fixing a basic typo)
> > ajantha-bhat - 2 commits (minor doc changes)
> >
> > From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.
> >
> > Kind Regards,
> > Justin
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[David Jencks <da...@gmail.com>]

"a lot of the proposed PMCs”

There are a lot of projects mentioned in this thread, and I think this is genuinely ambiguous. Do you mean PMCs or PMC members?

David Jencks

> On Jul 30, 2024, at 8:47 PM, Justin Mclean <ju...@classsoftware.com> wrote:
> 
> Hi,
> 
> In general, projects joining the incubator need two things: 
> - A codebase
> - a community around it
> 
> While this isn't a hard requirement, building either while in incubation can be difficult.
> 
> The first issue I see is it’s not clear to me that there is a community here.
> 
> The split between PPMC / committers is also unusual.
> 
> It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed? 
> 
> Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
> anoopj - 1 commit (fixing a basic typo)
> ajantha-bhat - 2 commits (minor doc changes)
> 
> From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.
> 
> Kind Regards,
> Justin
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jack Ye <ye...@gmail.com>]

Just to provide some additional data points to JB's point about
contributions that are not code, me (and other PPMC members) have been
involved in the Polaris community for quite some time now, and we have been
brainstorming ideas and driving important designs to lay a good foundation
for the project to grow. Some of the proposals we put up in Apache Iceberg
like the one for securable objects [1] are direct outcomes of those
discussions with the Polaris community, and were proposed to work with its
brand new role based access control model. We would like to continue to
work on both Iceberg and Polaris sides to provide a production-ready
reference implementation of the Iceberg catalog, and keep driving
innovations in the whole data lake/lakehouse/warehouse catalog space
through this mechanism.

-Jack

[1]
https://docs.google.com/document/d/1KmIDbPuN6IYF0nWs9ostXIB9F4b8iH3zZO0hjgs1lm4/edit


On Wed, Jul 31, 2024 at 5:26 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi
>
> Project Nessie (https://projectnessie.org/) is a catalog for data
> lakes, also implementing the Apache Iceberg REST spec.
> Nessie has some unique features like git-inspired data version
> control, multi-table transaction, catalog level versioning, etc.
> In order to build a larger community and provide a great experience to
> users, Nessie community decided to contribute Nessie features in
> Polaris.
>
> Regards
> JB
>
> On Wed, Jul 31, 2024 at 2:18 PM roryqi <ro...@apache.org> wrote:
> >
> > Just curious.  What's the relationship between Nessie and Polaris?
> >
> > Jean-Baptiste Onofré <jb...@nanthrax.net> 于2024年7月31日周三 20:09写道：
> >
> > > Hi Justin,
> > >
> > > Thanks for your email. Much appreciated!
> > >
> > > I agree with the requirements to join the incubator, I share the same
> > > vision.
> > >
> > > The codebase is there, public, under the Apache license. So, I think
> > > we are good here :)
> > >
> > > About the community, it's a small one, but present, vibrant, and eager
> > > to grow. We have a lot of enthusiasm around this proposal (and also
> > > Polaris open source). I think it would be a great podling.
> > >
> > > The split between PPMC / ocmmitters is unusual because of the
> > > community build goal. The team had a lot of feedback and interest from
> > > different parties, including some Apache projects.
> > > When I prepared the proposal with the team (that included people
> > > listed in the PPMC), we wanted to involve anyone helping the project
> > > to grow. It's not necessary code contributions, it's everything to:
> > > 1. Help in the design (for instance Jack Ye is helping on the Apache
> > > Iceberg REST impl vision)
> > > 2. Help to grow the community by including other communities (for
> > > instance Anoop proposed to bridge with Apache XTable, John with Apache
> > > Kafka, Robert with Project Nessie)
> > > 3. Mentor committers and other PPMC (that's my role mostly :)) in
> > > addition of the regular mentor. For instance, Tyler could be a great
> > > mentor thanks to his experience on Apache Beam.
> > >
> > > So about the commits:
> > > - Polaris started on a private repo at Snowflake including code that
> > > it was not possible to push open source (Snowflake specific). So, we
> > > had to change the history to remove these commits and have a clean
> > > open source codebase
> > > - You are right, not all PPMC members have committed on the repo, same
> > > about the committers. The reason they are there is that either:
> > >    1. Their contributions are not code
> > >    2. Their contributions are not yet there, but they plan to
> contribute
> > > I think it happens pretty often to have people listed in the initial
> > > committers without commit (for instance, I think on Gravitino, Ashish
> > > Singh is on the initial committer list without a commit afair).
> > >
> > > I understand your comment, and it's important to me.
> > > If it's a concern for you, I'm happy to rework the initial committer
> > > list with the team (also removing PPMC list to use the default layout)
> > > and grow the PPMC/committers when in the incubator.
> > >
> > > Thoughts ?
> > >
> > > Regards
> > > JB
> > >
> > > On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <
> justin@classsoftware.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > In general, projects joining the incubator need two things:
> > > > - A codebase
> > > > - a community around it
> > > >
> > > > While this isn't a hard requirement, building either while in
> incubation
> > > can be difficult.
> > > >
> > > > The first issue I see is it’s not clear to me that there is a
> community
> > > here.
> > > >
> > > > The split between PPMC / committers is also unusual.
> > > >
> > > > It also seems there’s a disconnect between the people who have
> > > contributed to the project and those on the initial PMC/committer
> list. I
> > > understand history has been lost. Is there any reason why history
> wasn’t
> > > preserved? But given it comes from a single vendor, I would assume
> that the
> > > commits in the new repo would then give an accurate representation of
> who
> > > is involved from outside that vendor since the plan/announcement to
> make it
> > > open source. However looking at the commits, 80%+ of the people
> involved
> > > come from that single vendor which and most of the proposed
> PPMC/committers
> > > have no activity in the project. How have these people contributed?
> > > >
> > > > Looking closer, other than jbonofre or snazy, I can see these commits
> > > from non-Snowflake people, and that is it.
> > > > anoopj - 1 commit (fixing a basic typo)
> > > > ajantha-bhat - 2 commits (minor doc changes)
> > > >
> > > > From the information provided, it seems that a lot of the proposed
> PMCs
> > > have not been involved with the project at all. Mentors are there to
> help a
> > > project, adding extra existing PMC members to the PPMC to just help
> out is
> > > also rather unusual. So both the PPMC/committer split and the
> composition
> > > seem problematic to me.
> > > >
> > > > Kind Regards,
> > > > Justin
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi

Project Nessie (https://projectnessie.org/) is a catalog for data
lakes, also implementing the Apache Iceberg REST spec.
Nessie has some unique features like git-inspired data version
control, multi-table transaction, catalog level versioning, etc.
In order to build a larger community and provide a great experience to
users, Nessie community decided to contribute Nessie features in
Polaris.

Regards
JB

On Wed, Jul 31, 2024 at 2:18 PM roryqi <ro...@apache.org> wrote:
>
> Just curious.  What's the relationship between Nessie and Polaris?
>
> Jean-Baptiste Onofré <jb...@nanthrax.net> 于2024年7月31日周三 20:09写道：
>
> > Hi Justin,
> >
> > Thanks for your email. Much appreciated!
> >
> > I agree with the requirements to join the incubator, I share the same
> > vision.
> >
> > The codebase is there, public, under the Apache license. So, I think
> > we are good here :)
> >
> > About the community, it's a small one, but present, vibrant, and eager
> > to grow. We have a lot of enthusiasm around this proposal (and also
> > Polaris open source). I think it would be a great podling.
> >
> > The split between PPMC / ocmmitters is unusual because of the
> > community build goal. The team had a lot of feedback and interest from
> > different parties, including some Apache projects.
> > When I prepared the proposal with the team (that included people
> > listed in the PPMC), we wanted to involve anyone helping the project
> > to grow. It's not necessary code contributions, it's everything to:
> > 1. Help in the design (for instance Jack Ye is helping on the Apache
> > Iceberg REST impl vision)
> > 2. Help to grow the community by including other communities (for
> > instance Anoop proposed to bridge with Apache XTable, John with Apache
> > Kafka, Robert with Project Nessie)
> > 3. Mentor committers and other PPMC (that's my role mostly :)) in
> > addition of the regular mentor. For instance, Tyler could be a great
> > mentor thanks to his experience on Apache Beam.
> >
> > So about the commits:
> > - Polaris started on a private repo at Snowflake including code that
> > it was not possible to push open source (Snowflake specific). So, we
> > had to change the history to remove these commits and have a clean
> > open source codebase
> > - You are right, not all PPMC members have committed on the repo, same
> > about the committers. The reason they are there is that either:
> >    1. Their contributions are not code
> >    2. Their contributions are not yet there, but they plan to contribute
> > I think it happens pretty often to have people listed in the initial
> > committers without commit (for instance, I think on Gravitino, Ashish
> > Singh is on the initial committer list without a commit afair).
> >
> > I understand your comment, and it's important to me.
> > If it's a concern for you, I'm happy to rework the initial committer
> > list with the team (also removing PPMC list to use the default layout)
> > and grow the PPMC/committers when in the incubator.
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <ju...@classsoftware.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > In general, projects joining the incubator need two things:
> > > - A codebase
> > > - a community around it
> > >
> > > While this isn't a hard requirement, building either while in incubation
> > can be difficult.
> > >
> > > The first issue I see is it’s not clear to me that there is a community
> > here.
> > >
> > > The split between PPMC / committers is also unusual.
> > >
> > > It also seems there’s a disconnect between the people who have
> > contributed to the project and those on the initial PMC/committer list. I
> > understand history has been lost. Is there any reason why history wasn’t
> > preserved? But given it comes from a single vendor, I would assume that the
> > commits in the new repo would then give an accurate representation of who
> > is involved from outside that vendor since the plan/announcement to make it
> > open source. However looking at the commits, 80%+ of the people involved
> > come from that single vendor which and most of the proposed PPMC/committers
> > have no activity in the project. How have these people contributed?
> > >
> > > Looking closer, other than jbonofre or snazy, I can see these commits
> > from non-Snowflake people, and that is it.
> > > anoopj - 1 commit (fixing a basic typo)
> > > ajantha-bhat - 2 commits (minor doc changes)
> > >
> > > From the information provided, it seems that a lot of the proposed PMCs
> > have not been involved with the project at all. Mentors are there to help a
> > project, adding extra existing PMC members to the PPMC to just help out is
> > also rather unusual. So both the PPMC/committer split and the composition
> > seem problematic to me.
> > >
> > > Kind Regards,
> > > Justin
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[roryqi <ro...@apache.org>]

Just curious.  What's the relationship between Nessie and Polaris?

Jean-Baptiste Onofré <jb...@nanthrax.net> 于2024年7月31日周三 20:09写道：

> Hi Justin,
>
> Thanks for your email. Much appreciated!
>
> I agree with the requirements to join the incubator, I share the same
> vision.
>
> The codebase is there, public, under the Apache license. So, I think
> we are good here :)
>
> About the community, it's a small one, but present, vibrant, and eager
> to grow. We have a lot of enthusiasm around this proposal (and also
> Polaris open source). I think it would be a great podling.
>
> The split between PPMC / ocmmitters is unusual because of the
> community build goal. The team had a lot of feedback and interest from
> different parties, including some Apache projects.
> When I prepared the proposal with the team (that included people
> listed in the PPMC), we wanted to involve anyone helping the project
> to grow. It's not necessary code contributions, it's everything to:
> 1. Help in the design (for instance Jack Ye is helping on the Apache
> Iceberg REST impl vision)
> 2. Help to grow the community by including other communities (for
> instance Anoop proposed to bridge with Apache XTable, John with Apache
> Kafka, Robert with Project Nessie)
> 3. Mentor committers and other PPMC (that's my role mostly :)) in
> addition of the regular mentor. For instance, Tyler could be a great
> mentor thanks to his experience on Apache Beam.
>
> So about the commits:
> - Polaris started on a private repo at Snowflake including code that
> it was not possible to push open source (Snowflake specific). So, we
> had to change the history to remove these commits and have a clean
> open source codebase
> - You are right, not all PPMC members have committed on the repo, same
> about the committers. The reason they are there is that either:
>    1. Their contributions are not code
>    2. Their contributions are not yet there, but they plan to contribute
> I think it happens pretty often to have people listed in the initial
> committers without commit (for instance, I think on Gravitino, Ashish
> Singh is on the initial committer list without a commit afair).
>
> I understand your comment, and it's important to me.
> If it's a concern for you, I'm happy to rework the initial committer
> list with the team (also removing PPMC list to use the default layout)
> and grow the PPMC/committers when in the incubator.
>
> Thoughts ?
>
> Regards
> JB
>
> On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <ju...@classsoftware.com>
> wrote:
> >
> > Hi,
> >
> > In general, projects joining the incubator need two things:
> > - A codebase
> > - a community around it
> >
> > While this isn't a hard requirement, building either while in incubation
> can be difficult.
> >
> > The first issue I see is it’s not clear to me that there is a community
> here.
> >
> > The split between PPMC / committers is also unusual.
> >
> > It also seems there’s a disconnect between the people who have
> contributed to the project and those on the initial PMC/committer list. I
> understand history has been lost. Is there any reason why history wasn’t
> preserved? But given it comes from a single vendor, I would assume that the
> commits in the new repo would then give an accurate representation of who
> is involved from outside that vendor since the plan/announcement to make it
> open source. However looking at the commits, 80%+ of the people involved
> come from that single vendor which and most of the proposed PPMC/committers
> have no activity in the project. How have these people contributed?
> >
> > Looking closer, other than jbonofre or snazy, I can see these commits
> from non-Snowflake people, and that is it.
> > anoopj - 1 commit (fixing a basic typo)
> > ajantha-bhat - 2 commits (minor doc changes)
> >
> > From the information provided, it seems that a lot of the proposed PMCs
> have not been involved with the project at all. Mentors are there to help a
> project, adding extra existing PMC members to the PPMC to just help out is
> also rather unusual. So both the PPMC/committer split and the composition
> seem problematic to me.
> >
> > Kind Regards,
> > Justin
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [DISCUSS] PolarisProposal to the incubator

[Justin Mclean <ju...@classsoftware.com>]

Hi,

I notice a number of people offering to be mentors. Experience has shown that three or four mentors seem to be best. More than that, and you can end up with a situation where everyone thinks everyone else is doing the job, so no one does it. If a mentor is not active during incubation they can be replaced.

Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi François,

To my knowledge, there's no minimum number of initial committers.
Some projects have a very small initial committers list (I remember
OpenDAL proposal with 4 initial committers), some have longer list
(Beam proposal contained 18 initial committers + interested
contributors).

I think the Polaris proposal is comparable to the Beam proposal. The
wording in Beam proposal was different (initial committers +
interested contributors instead of PPMC + committers), but the
intention is similar.

We do recommend 3 mentors though.

Regards
JB


On Wed, Jul 31, 2024 at 3:04 PM Francois Papon
<fr...@openobject.fr> wrote:
>
> Hi,
>
> I don't see the problem with people that contribute to a project without
> writing source code, as I know we already have these kind of people in
> ASF projects.
>
> Just for my information, do we have some numbers about the min
> contributors on a project and initial committers to be accepted in the
> incubator?
>
> May be it can help new project and people who wants to make a proposal
> to have an idea if their proposition could be accepted or not.
>
> regards,
>
> François
>
> On 31/07/2024 14:07, Jean-Baptiste Onofré wrote:
> > Hi Justin,
> >
> > Thanks for your email. Much appreciated!
> >
> > I agree with the requirements to join the incubator, I share the same vision.
> >
> > The codebase is there, public, under the Apache license. So, I think
> > we are good here :)
> >
> > About the community, it's a small one, but present, vibrant, and eager
> > to grow. We have a lot of enthusiasm around this proposal (and also
> > Polaris open source). I think it would be a great podling.
> >
> > The split between PPMC / ocmmitters is unusual because of the
> > community build goal. The team had a lot of feedback and interest from
> > different parties, including some Apache projects.
> > When I prepared the proposal with the team (that included people
> > listed in the PPMC), we wanted to involve anyone helping the project
> > to grow. It's not necessary code contributions, it's everything to:
> > 1. Help in the design (for instance Jack Ye is helping on the Apache
> > Iceberg REST impl vision)
> > 2. Help to grow the community by including other communities (for
> > instance Anoop proposed to bridge with Apache XTable, John with Apache
> > Kafka, Robert with Project Nessie)
> > 3. Mentor committers and other PPMC (that's my role mostly :)) in
> > addition of the regular mentor. For instance, Tyler could be a great
> > mentor thanks to his experience on Apache Beam.
> >
> > So about the commits:
> > - Polaris started on a private repo at Snowflake including code that
> > it was not possible to push open source (Snowflake specific). So, we
> > had to change the history to remove these commits and have a clean
> > open source codebase
> > - You are right, not all PPMC members have committed on the repo, same
> > about the committers. The reason they are there is that either:
> >     1. Their contributions are not code
> >     2. Their contributions are not yet there, but they plan to contribute
> > I think it happens pretty often to have people listed in the initial
> > committers without commit (for instance, I think on Gravitino, Ashish
> > Singh is on the initial committer list without a commit afair).
> >
> > I understand your comment, and it's important to me.
> > If it's a concern for you, I'm happy to rework the initial committer
> > list with the team (also removing PPMC list to use the default layout)
> > and grow the PPMC/committers when in the incubator.
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <ju...@classsoftware.com> wrote:
> >> Hi,
> >>
> >> In general, projects joining the incubator need two things:
> >> - A codebase
> >> - a community around it
> >>
> >> While this isn't a hard requirement, building either while in incubation can be difficult.
> >>
> >> The first issue I see is it’s not clear to me that there is a community here.
> >>
> >> The split between PPMC / committers is also unusual.
> >>
> >> It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed?
> >>
> >> Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
> >> anoopj - 1 commit (fixing a basic typo)
> >> ajantha-bhat - 2 commits (minor doc changes)
> >>
> >>  From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.
> >>
> >> Kind Regards,
> >> Justin
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Francois Papon <fr...@openobject.fr>]

Hi,

I don't see the problem with people that contribute to a project without 
writing source code, as I know we already have these kind of people in 
ASF projects.

Just for my information, do we have some numbers about the min 
contributors on a project and initial committers to be accepted in the 
incubator?

May be it can help new project and people who wants to make a proposal 
to have an idea if their proposition could be accepted or not.

regards,

François

On 31/07/2024 14:07, Jean-Baptiste Onofré wrote:
> Hi Justin,
>
> Thanks for your email. Much appreciated!
>
> I agree with the requirements to join the incubator, I share the same vision.
>
> The codebase is there, public, under the Apache license. So, I think
> we are good here :)
>
> About the community, it's a small one, but present, vibrant, and eager
> to grow. We have a lot of enthusiasm around this proposal (and also
> Polaris open source). I think it would be a great podling.
>
> The split between PPMC / ocmmitters is unusual because of the
> community build goal. The team had a lot of feedback and interest from
> different parties, including some Apache projects.
> When I prepared the proposal with the team (that included people
> listed in the PPMC), we wanted to involve anyone helping the project
> to grow. It's not necessary code contributions, it's everything to:
> 1. Help in the design (for instance Jack Ye is helping on the Apache
> Iceberg REST impl vision)
> 2. Help to grow the community by including other communities (for
> instance Anoop proposed to bridge with Apache XTable, John with Apache
> Kafka, Robert with Project Nessie)
> 3. Mentor committers and other PPMC (that's my role mostly :)) in
> addition of the regular mentor. For instance, Tyler could be a great
> mentor thanks to his experience on Apache Beam.
>
> So about the commits:
> - Polaris started on a private repo at Snowflake including code that
> it was not possible to push open source (Snowflake specific). So, we
> had to change the history to remove these commits and have a clean
> open source codebase
> - You are right, not all PPMC members have committed on the repo, same
> about the committers. The reason they are there is that either:
>     1. Their contributions are not code
>     2. Their contributions are not yet there, but they plan to contribute
> I think it happens pretty often to have people listed in the initial
> committers without commit (for instance, I think on Gravitino, Ashish
> Singh is on the initial committer list without a commit afair).
>
> I understand your comment, and it's important to me.
> If it's a concern for you, I'm happy to rework the initial committer
> list with the team (also removing PPMC list to use the default layout)
> and grow the PPMC/committers when in the incubator.
>
> Thoughts ?
>
> Regards
> JB
>
> On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <ju...@classsoftware.com> wrote:
>> Hi,
>>
>> In general, projects joining the incubator need two things:
>> - A codebase
>> - a community around it
>>
>> While this isn't a hard requirement, building either while in incubation can be difficult.
>>
>> The first issue I see is it’s not clear to me that there is a community here.
>>
>> The split between PPMC / committers is also unusual.
>>
>> It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed?
>>
>> Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
>> anoopj - 1 commit (fixing a basic typo)
>> ajantha-bhat - 2 commits (minor doc changes)
>>
>>  From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.
>>
>> Kind Regards,
>> Justin
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Dave Fisher <wa...@apache.org>]

> On Jul 31, 2024, at 6:56 PM, Justin Mclean <ju...@classsoftware.com> wrote:
> 
> Hi,
> 
>> I understand your comment, and it's important to me.
>> If it's a concern for you, I'm happy to rework the initial committer
>> list with the team (also removing PPMC list to use the default layout)
>> and grow the PPMC/committers when in the incubator.
> 
> I think that would be in the best interests of the project. That way, you would have a larger pool of people to add during graduation when they do make contributions and recognise merit. The final composition of the PMC can be decided on graduation.

I’m happy with JBO’s explanation about the PPMC/Initial Committers split. I think that their plan is fine. Incubation is all about proving the growth and the Polaris community will or will not grow from their their chosen starting point.

They have put thought into their plans and I say let’s VOTE already!

Best,
Dave


> 
> Kind Regards,
> Justin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> I understand your comment, and it's important to me.
> If it's a concern for you, I'm happy to rework the initial committer
> list with the team (also removing PPMC list to use the default layout)
> and grow the PPMC/committers when in the incubator.

I think that would be in the best interests of the project. That way, you would have a larger pool of people to add during graduation when they do make contributions and recognise merit. The final composition of the PMC can be decided on graduation.

Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Justin,

Thanks for your email. Much appreciated!

I agree with the requirements to join the incubator, I share the same vision.

The codebase is there, public, under the Apache license. So, I think
we are good here :)

About the community, it's a small one, but present, vibrant, and eager
to grow. We have a lot of enthusiasm around this proposal (and also
Polaris open source). I think it would be a great podling.

The split between PPMC / ocmmitters is unusual because of the
community build goal. The team had a lot of feedback and interest from
different parties, including some Apache projects.
When I prepared the proposal with the team (that included people
listed in the PPMC), we wanted to involve anyone helping the project
to grow. It's not necessary code contributions, it's everything to:
1. Help in the design (for instance Jack Ye is helping on the Apache
Iceberg REST impl vision)
2. Help to grow the community by including other communities (for
instance Anoop proposed to bridge with Apache XTable, John with Apache
Kafka, Robert with Project Nessie)
3. Mentor committers and other PPMC (that's my role mostly :)) in
addition of the regular mentor. For instance, Tyler could be a great
mentor thanks to his experience on Apache Beam.

So about the commits:
- Polaris started on a private repo at Snowflake including code that
it was not possible to push open source (Snowflake specific). So, we
had to change the history to remove these commits and have a clean
open source codebase
- You are right, not all PPMC members have committed on the repo, same
about the committers. The reason they are there is that either:
   1. Their contributions are not code
   2. Their contributions are not yet there, but they plan to contribute
I think it happens pretty often to have people listed in the initial
committers without commit (for instance, I think on Gravitino, Ashish
Singh is on the initial committer list without a commit afair).

I understand your comment, and it's important to me.
If it's a concern for you, I'm happy to rework the initial committer
list with the team (also removing PPMC list to use the default layout)
and grow the PPMC/committers when in the incubator.

Thoughts ?

Regards
JB

On Wed, Jul 31, 2024 at 5:47 AM Justin Mclean <ju...@classsoftware.com> wrote:
>
> Hi,
>
> In general, projects joining the incubator need two things:
> - A codebase
> - a community around it
>
> While this isn't a hard requirement, building either while in incubation can be difficult.
>
> The first issue I see is it’s not clear to me that there is a community here.
>
> The split between PPMC / committers is also unusual.
>
> It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed?
>
> Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
> anoopj - 1 commit (fixing a basic typo)
> ajantha-bhat - 2 commits (minor doc changes)
>
> From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.
>
> Kind Regards,
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Justin Mclean <ju...@classsoftware.com>]

Hi,

In general, projects joining the incubator need two things: 
- A codebase
- a community around it

While this isn't a hard requirement, building either while in incubation can be difficult.

The first issue I see is it’s not clear to me that there is a community here.

The split between PPMC / committers is also unusual.

It also seems there’s a disconnect between the people who have contributed to the project and those on the initial PMC/committer list. I understand history has been lost. Is there any reason why history wasn’t preserved? But given it comes from a single vendor, I would assume that the commits in the new repo would then give an accurate representation of who is involved from outside that vendor since the plan/announcement to make it open source. However looking at the commits, 80%+ of the people involved come from that single vendor which and most of the proposed PPMC/committers have no activity in the project. How have these people contributed? 

Looking closer, other than jbonofre or snazy, I can see these commits from non-Snowflake people, and that is it.
anoopj - 1 commit (fixing a basic typo)
ajantha-bhat - 2 commits (minor doc changes)

From the information provided, it seems that a lot of the proposed PMCs have not been involved with the project at all. Mentors are there to help a project, adding extra existing PMC members to the PPMC to just help out is also rather unusual. So both the PPMC/committer split and the composition seem problematic to me.

Kind Regards,
Justin


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Rich Bowen <rb...@rcbowen.com>]

> On Jul 31, 2024, at 4:02 PM, Tyler Akidau <ta...@apache.org> wrote:
> 
> From project overlap perspective, I just want to echo Jack’s take on things: Polaris for now is fully focused on Iceberg, taking a depth first approach, with the goal of implementing the entire Iceberg REST API spec and helping push forward the state of the art in the Iceberg ecosystem for features like governance that are highly important for all of our collective user bases. It’s absolutely adjacent to Gravitino, but as others have said, it feels to me that they are heading in somewhat different directions overall. I also think there’s lots of empty space in the open source catalog ecosystem in general at this point, with plenty of room for both of these efforts to beneficially exist in parallel. And we are absolutely open to discussing collaborations, with Gravitino, Amoro, or anyone else; JB has highlighted the importance of this from the very beginning of our Polaris conversations, and I completely agree.

It’s worth noting that the ASF has never objected to having multiple projects “competing” in the same general technology space. We don’t pick winners, and we don’t object to overlapping projects. Those questions of how it is like/different from other projects are not disqualifying.

Re: [DISCUSS] PolarisProposal to the incubator

[Ryan Blue <bl...@databricks.com.INVALID>]

Thanks, JB! I can add myself.

On Wed, Jul 31, 2024 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Thanks for your input Ryan ! Much appreciated.
>
> We would love to have you as mentor ! Do you want me to update the proposal
> wiki page or you do ?
>
> Thanks again
>
> Regards
> JB
>
> Le mer. 31 juil. 2024 à 22:28, Ryan Blue <bl...@databricks.com.invalid> a
> écrit :
>
> > I'm glad to see this proposal because people have been talking about
> > working on an implementation of the Iceberg REST catalog spec for a long
> > time. I don't think that it is a good idea to put an implementation in
> the
> > Iceberg project itself, so it is great to see a project that intends to
> > build one to meet that demand.
> >
> > I'd like to volunteer to help out and mentor the project. I have a lot of
> > context on the REST catalog spec from contributing to the design and
> client
> > implementation, and I've helped both Parquet and Iceberg through
> incubation
> > (which is why I talk about maintaining LICENSE and NOTICE so much).
> >
> > My take on the PPMC/committer list is that this seems like a reasonable
> > choice. I'm also not worried that the project won't be able to attract a
> > community given the size of the initial list.
> >
> > Ryan
> >
> > On Wed, Jul 31, 2024 at 1:02 PM Tyler Akidau <ta...@apache.org> wrote:
> >
> > > Hey folks,
> > >
> > > I wanted to give a little bit of additional context beyond what JB and
> > > Jack have said so far in other threads. Everyone’s observations about
> the
> > > level of community code contributions, the committer/PPMC list setup,
> > > adjacency to other projects, etc. are spot on. The code has been pretty
> > > much entirely delivered by Snowflake at this point. And the
> > PPMC/committer
> > > division in the proposal is atypical, but as Jack and JB called out,
> it’s
> > > reflective of the collaborative community building that’s been
> happening
> > > over the last two months; more on this below.
> > >
> > > From a code contribution perspective, we’ve largely been blocked on
> > > getting a shareable repo up and running, which I admit took longer than
> > > we’d hoped. That was primarily due to Snowflake internal logistics,
> which
> > > as with any large company, is what it is at times. Now that we have
> that
> > in
> > > place, I expect to see more material code contributions rolling in over
> > the
> > > coming weeks. We’ve been having early discussions with the Dremio folks
> > > about how Nessie features like catalog level versioning can be
> integrated
> > > into Polaris, and once we align on a concrete design, Robert and JB and
> > > crew will be diving in more deeply. Similarly, we’ve had early
> > discussions
> > > on integrations with other partners in the community, and now that the
> > > codebase is fully public, it will be easier for us to make concrete
> > > progress on turning those discussions into actual code contributions
> > (e.g.,
> > > there's already some early Trino integration work happening [1].)
> > >
> > > From a community building perspective, in particular the concern that
> it
> > > can be tough to build a community for a podling in a vacuum, I
> completely
> > > agree. If you start a podling with no community in sight, you may be
> left
> > > floundering and alone for quite some time. That’s why JB and I have
> spent
> > > the last two months bootstrapping that process, finding stakeholders
> who
> > > are interested in helping grown Polaris in some way, making sure we’re
> > > directionally aligned on where we want the project to go, and
> identifying
> > > specific individuals with both a vested interest in contribution and
> > > experience helping grow and run Apache projects in the Apache way. A
> lot
> > of
> > > time, thought, and collaboration went into building this initial
> > community
> > > across a diverse set of stakeholders, and we wanted to reflect that in
> > > calling out the proposed PPMC list separately. As JB said, we’re happy
> to
> > > adjust the lists to something more standard if desired, but we believe
> > the
> > > story behind the lists is important in this case.
> > >
> > > From project overlap perspective, I just want to echo Jack’s take on
> > > things: Polaris for now is fully focused on Iceberg, taking a depth
> first
> > > approach, with the goal of implementing the entire Iceberg REST API
> spec
> > > and helping push forward the state of the art in the Iceberg ecosystem
> > for
> > > features like governance that are highly important for all of our
> > > collective user bases. It’s absolutely adjacent to Gravitino, but as
> > others
> > > have said, it feels to me that they are heading in somewhat different
> > > directions overall. I also think there’s lots of empty space in the
> open
> > > source catalog ecosystem in general at this point, with plenty of room
> > for
> > > both of these efforts to beneficially exist in parallel. And we are
> > > absolutely open to discussing collaborations, with Gravitino, Amoro, or
> > > anyone else; JB has highlighted the importance of this from the very
> > > beginning of our Polaris conversations, and I completely agree.
> > >
> > > And lastly yes, any existing trademark issues should be fixed. I know
> > > there was one batch discovered after the initial push that we were
> > working
> > > on fixing, but I'll go back and see if there are others we haven't
> > > addressed (or if those fixes somehow just haven't made it out yet.)
> > >
> > > Thank you everyone for the feedback and enthusiasm. We appreciate it.
> :-)
> > >
> > > [1] https://github.com/polaris-catalog/polaris/pull/42
> > >
> > > Cheers,
> > > -Tyler
> > >
> > > On 2024/07/31 07:30:07 Justin Mclean wrote:
> > > > Hi,
> > > >
> > > > I sent a reply earlier, but my email is acting up and looks like it
> > > didn’t get through. I have some concerns with this proposal.
> > > >
> > > > In general, the incubator likes projects to have a code base and a
> > small
> > > community, I’m not seeing a community here. Trying to build one during
> > > incubation can be difficult. We have recently rejected proposals in a
> > > similar state, asking them to come back when they have more of a
> > community
> > > around the project.
> > > >
> > > > The PPMC/committer split is unusual.
> > > >
> > > > There seems to be little relation to people who have contributed to
> the
> > > project and the initial committer list. A large number of the people
> > > involved in commits (80+%) are from one vendor, with two exceptions,
> and
> > > two others have made one or two minor commits of a couple of lines.
> > > >
> > > > Adding people to PPMC to help out also seems unusual, as that is the
> > > mentor's job.
> > > >
> > > > In short, this seems to me (and I could be wrong) like a project
> mostly
> > > from a single vendor, but the proposal has been made to make it look
> like
> > > more people are involved. It may well be that these people will be
> > > involved, but I’d prefer if the project was upfront about this and
> added
> > > committers the usual way during incubation.
> > > >
> > > > In short, the initial commit list looks problematic to me.
> > > >
> > > > Kind Regards,
> > > > Justin
> > > >
> > > > P.S. The repo landing page/readme has some ASF trademark issues that
> > > would be good to address.
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> > --
> > Ryan Blue
> > Databricks
> >
>


-- 
Ryan Blue
Databricks

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Thanks for your input Ryan ! Much appreciated.

We would love to have you as mentor ! Do you want me to update the proposal
wiki page or you do ?

Thanks again

Regards
JB

Le mer. 31 juil. 2024 à 22:28, Ryan Blue <bl...@databricks.com.invalid> a
écrit :

> I'm glad to see this proposal because people have been talking about
> working on an implementation of the Iceberg REST catalog spec for a long
> time. I don't think that it is a good idea to put an implementation in the
> Iceberg project itself, so it is great to see a project that intends to
> build one to meet that demand.
>
> I'd like to volunteer to help out and mentor the project. I have a lot of
> context on the REST catalog spec from contributing to the design and client
> implementation, and I've helped both Parquet and Iceberg through incubation
> (which is why I talk about maintaining LICENSE and NOTICE so much).
>
> My take on the PPMC/committer list is that this seems like a reasonable
> choice. I'm also not worried that the project won't be able to attract a
> community given the size of the initial list.
>
> Ryan
>
> On Wed, Jul 31, 2024 at 1:02 PM Tyler Akidau <ta...@apache.org> wrote:
>
> > Hey folks,
> >
> > I wanted to give a little bit of additional context beyond what JB and
> > Jack have said so far in other threads. Everyone’s observations about the
> > level of community code contributions, the committer/PPMC list setup,
> > adjacency to other projects, etc. are spot on. The code has been pretty
> > much entirely delivered by Snowflake at this point. And the
> PPMC/committer
> > division in the proposal is atypical, but as Jack and JB called out, it’s
> > reflective of the collaborative community building that’s been happening
> > over the last two months; more on this below.
> >
> > From a code contribution perspective, we’ve largely been blocked on
> > getting a shareable repo up and running, which I admit took longer than
> > we’d hoped. That was primarily due to Snowflake internal logistics, which
> > as with any large company, is what it is at times. Now that we have that
> in
> > place, I expect to see more material code contributions rolling in over
> the
> > coming weeks. We’ve been having early discussions with the Dremio folks
> > about how Nessie features like catalog level versioning can be integrated
> > into Polaris, and once we align on a concrete design, Robert and JB and
> > crew will be diving in more deeply. Similarly, we’ve had early
> discussions
> > on integrations with other partners in the community, and now that the
> > codebase is fully public, it will be easier for us to make concrete
> > progress on turning those discussions into actual code contributions
> (e.g.,
> > there's already some early Trino integration work happening [1].)
> >
> > From a community building perspective, in particular the concern that it
> > can be tough to build a community for a podling in a vacuum, I completely
> > agree. If you start a podling with no community in sight, you may be left
> > floundering and alone for quite some time. That’s why JB and I have spent
> > the last two months bootstrapping that process, finding stakeholders who
> > are interested in helping grown Polaris in some way, making sure we’re
> > directionally aligned on where we want the project to go, and identifying
> > specific individuals with both a vested interest in contribution and
> > experience helping grow and run Apache projects in the Apache way. A lot
> of
> > time, thought, and collaboration went into building this initial
> community
> > across a diverse set of stakeholders, and we wanted to reflect that in
> > calling out the proposed PPMC list separately. As JB said, we’re happy to
> > adjust the lists to something more standard if desired, but we believe
> the
> > story behind the lists is important in this case.
> >
> > From project overlap perspective, I just want to echo Jack’s take on
> > things: Polaris for now is fully focused on Iceberg, taking a depth first
> > approach, with the goal of implementing the entire Iceberg REST API spec
> > and helping push forward the state of the art in the Iceberg ecosystem
> for
> > features like governance that are highly important for all of our
> > collective user bases. It’s absolutely adjacent to Gravitino, but as
> others
> > have said, it feels to me that they are heading in somewhat different
> > directions overall. I also think there’s lots of empty space in the open
> > source catalog ecosystem in general at this point, with plenty of room
> for
> > both of these efforts to beneficially exist in parallel. And we are
> > absolutely open to discussing collaborations, with Gravitino, Amoro, or
> > anyone else; JB has highlighted the importance of this from the very
> > beginning of our Polaris conversations, and I completely agree.
> >
> > And lastly yes, any existing trademark issues should be fixed. I know
> > there was one batch discovered after the initial push that we were
> working
> > on fixing, but I'll go back and see if there are others we haven't
> > addressed (or if those fixes somehow just haven't made it out yet.)
> >
> > Thank you everyone for the feedback and enthusiasm. We appreciate it. :-)
> >
> > [1] https://github.com/polaris-catalog/polaris/pull/42
> >
> > Cheers,
> > -Tyler
> >
> > On 2024/07/31 07:30:07 Justin Mclean wrote:
> > > Hi,
> > >
> > > I sent a reply earlier, but my email is acting up and looks like it
> > didn’t get through. I have some concerns with this proposal.
> > >
> > > In general, the incubator likes projects to have a code base and a
> small
> > community, I’m not seeing a community here. Trying to build one during
> > incubation can be difficult. We have recently rejected proposals in a
> > similar state, asking them to come back when they have more of a
> community
> > around the project.
> > >
> > > The PPMC/committer split is unusual.
> > >
> > > There seems to be little relation to people who have contributed to the
> > project and the initial committer list. A large number of the people
> > involved in commits (80+%) are from one vendor, with two exceptions, and
> > two others have made one or two minor commits of a couple of lines.
> > >
> > > Adding people to PPMC to help out also seems unusual, as that is the
> > mentor's job.
> > >
> > > In short, this seems to me (and I could be wrong) like a project mostly
> > from a single vendor, but the proposal has been made to make it look like
> > more people are involved. It may well be that these people will be
> > involved, but I’d prefer if the project was upfront about this and added
> > committers the usual way during incubation.
> > >
> > > In short, the initial commit list looks problematic to me.
> > >
> > > Kind Regards,
> > > Justin
> > >
> > > P.S. The repo landing page/readme has some ASF trademark issues that
> > would be good to address.
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
> --
> Ryan Blue
> Databricks
>

Re: [DISCUSS] PolarisProposal to the incubator

[Ryan Blue <bl...@databricks.com.INVALID>]

I'm glad to see this proposal because people have been talking about
working on an implementation of the Iceberg REST catalog spec for a long
time. I don't think that it is a good idea to put an implementation in the
Iceberg project itself, so it is great to see a project that intends to
build one to meet that demand.

I'd like to volunteer to help out and mentor the project. I have a lot of
context on the REST catalog spec from contributing to the design and client
implementation, and I've helped both Parquet and Iceberg through incubation
(which is why I talk about maintaining LICENSE and NOTICE so much).

My take on the PPMC/committer list is that this seems like a reasonable
choice. I'm also not worried that the project won't be able to attract a
community given the size of the initial list.

Ryan

On Wed, Jul 31, 2024 at 1:02 PM Tyler Akidau <ta...@apache.org> wrote:

> Hey folks,
>
> I wanted to give a little bit of additional context beyond what JB and
> Jack have said so far in other threads. Everyone’s observations about the
> level of community code contributions, the committer/PPMC list setup,
> adjacency to other projects, etc. are spot on. The code has been pretty
> much entirely delivered by Snowflake at this point. And the PPMC/committer
> division in the proposal is atypical, but as Jack and JB called out, it’s
> reflective of the collaborative community building that’s been happening
> over the last two months; more on this below.
>
> From a code contribution perspective, we’ve largely been blocked on
> getting a shareable repo up and running, which I admit took longer than
> we’d hoped. That was primarily due to Snowflake internal logistics, which
> as with any large company, is what it is at times. Now that we have that in
> place, I expect to see more material code contributions rolling in over the
> coming weeks. We’ve been having early discussions with the Dremio folks
> about how Nessie features like catalog level versioning can be integrated
> into Polaris, and once we align on a concrete design, Robert and JB and
> crew will be diving in more deeply. Similarly, we’ve had early discussions
> on integrations with other partners in the community, and now that the
> codebase is fully public, it will be easier for us to make concrete
> progress on turning those discussions into actual code contributions (e.g.,
> there's already some early Trino integration work happening [1].)
>
> From a community building perspective, in particular the concern that it
> can be tough to build a community for a podling in a vacuum, I completely
> agree. If you start a podling with no community in sight, you may be left
> floundering and alone for quite some time. That’s why JB and I have spent
> the last two months bootstrapping that process, finding stakeholders who
> are interested in helping grown Polaris in some way, making sure we’re
> directionally aligned on where we want the project to go, and identifying
> specific individuals with both a vested interest in contribution and
> experience helping grow and run Apache projects in the Apache way. A lot of
> time, thought, and collaboration went into building this initial community
> across a diverse set of stakeholders, and we wanted to reflect that in
> calling out the proposed PPMC list separately. As JB said, we’re happy to
> adjust the lists to something more standard if desired, but we believe the
> story behind the lists is important in this case.
>
> From project overlap perspective, I just want to echo Jack’s take on
> things: Polaris for now is fully focused on Iceberg, taking a depth first
> approach, with the goal of implementing the entire Iceberg REST API spec
> and helping push forward the state of the art in the Iceberg ecosystem for
> features like governance that are highly important for all of our
> collective user bases. It’s absolutely adjacent to Gravitino, but as others
> have said, it feels to me that they are heading in somewhat different
> directions overall. I also think there’s lots of empty space in the open
> source catalog ecosystem in general at this point, with plenty of room for
> both of these efforts to beneficially exist in parallel. And we are
> absolutely open to discussing collaborations, with Gravitino, Amoro, or
> anyone else; JB has highlighted the importance of this from the very
> beginning of our Polaris conversations, and I completely agree.
>
> And lastly yes, any existing trademark issues should be fixed. I know
> there was one batch discovered after the initial push that we were working
> on fixing, but I'll go back and see if there are others we haven't
> addressed (or if those fixes somehow just haven't made it out yet.)
>
> Thank you everyone for the feedback and enthusiasm. We appreciate it. :-)
>
> [1] https://github.com/polaris-catalog/polaris/pull/42
>
> Cheers,
> -Tyler
>
> On 2024/07/31 07:30:07 Justin Mclean wrote:
> > Hi,
> >
> > I sent a reply earlier, but my email is acting up and looks like it
> didn’t get through. I have some concerns with this proposal.
> >
> > In general, the incubator likes projects to have a code base and a small
> community, I’m not seeing a community here. Trying to build one during
> incubation can be difficult. We have recently rejected proposals in a
> similar state, asking them to come back when they have more of a community
> around the project.
> >
> > The PPMC/committer split is unusual.
> >
> > There seems to be little relation to people who have contributed to the
> project and the initial committer list. A large number of the people
> involved in commits (80+%) are from one vendor, with two exceptions, and
> two others have made one or two minor commits of a couple of lines.
> >
> > Adding people to PPMC to help out also seems unusual, as that is the
> mentor's job.
> >
> > In short, this seems to me (and I could be wrong) like a project mostly
> from a single vendor, but the proposal has been made to make it look like
> more people are involved. It may well be that these people will be
> involved, but I’d prefer if the project was upfront about this and added
> committers the usual way during incubation.
> >
> > In short, the initial commit list looks problematic to me.
> >
> > Kind Regards,
> > Justin
> >
> > P.S. The repo landing page/readme has some ASF trademark issues that
> would be good to address.
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

-- 
Ryan Blue
Databricks

Re: [DISCUSS] PolarisProposal to the incubator

[Tyler Akidau <ta...@apache.org>]

Hey folks,

I wanted to give a little bit of additional context beyond what JB and Jack have said so far in other threads. Everyone’s observations about the level of community code contributions, the committer/PPMC list setup, adjacency to other projects, etc. are spot on. The code has been pretty much entirely delivered by Snowflake at this point. And the PPMC/committer division in the proposal is atypical, but as Jack and JB called out, it’s reflective of the collaborative community building that’s been happening over the last two months; more on this below.

From a code contribution perspective, we’ve largely been blocked on getting a shareable repo up and running, which I admit took longer than we’d hoped. That was primarily due to Snowflake internal logistics, which as with any large company, is what it is at times. Now that we have that in place, I expect to see more material code contributions rolling in over the coming weeks. We’ve been having early discussions with the Dremio folks about how Nessie features like catalog level versioning can be integrated into Polaris, and once we align on a concrete design, Robert and JB and crew will be diving in more deeply. Similarly, we’ve had early discussions on integrations with other partners in the community, and now that the codebase is fully public, it will be easier for us to make concrete progress on turning those discussions into actual code contributions (e.g., there's already some early Trino integration work happening [1].)

From a community building perspective, in particular the concern that it can be tough to build a community for a podling in a vacuum, I completely agree. If you start a podling with no community in sight, you may be left floundering and alone for quite some time. That’s why JB and I have spent the last two months bootstrapping that process, finding stakeholders who are interested in helping grown Polaris in some way, making sure we’re directionally aligned on where we want the project to go, and identifying specific individuals with both a vested interest in contribution and experience helping grow and run Apache projects in the Apache way. A lot of time, thought, and collaboration went into building this initial community across a diverse set of stakeholders, and we wanted to reflect that in calling out the proposed PPMC list separately. As JB said, we’re happy to adjust the lists to something more standard if desired, but we believe the story behind the lists is important in this case. 

From project overlap perspective, I just want to echo Jack’s take on things: Polaris for now is fully focused on Iceberg, taking a depth first approach, with the goal of implementing the entire Iceberg REST API spec and helping push forward the state of the art in the Iceberg ecosystem for features like governance that are highly important for all of our collective user bases. It’s absolutely adjacent to Gravitino, but as others have said, it feels to me that they are heading in somewhat different directions overall. I also think there’s lots of empty space in the open source catalog ecosystem in general at this point, with plenty of room for both of these efforts to beneficially exist in parallel. And we are absolutely open to discussing collaborations, with Gravitino, Amoro, or anyone else; JB has highlighted the importance of this from the very beginning of our Polaris conversations, and I completely agree.

And lastly yes, any existing trademark issues should be fixed. I know there was one batch discovered after the initial push that we were working on fixing, but I'll go back and see if there are others we haven't addressed (or if those fixes somehow just haven't made it out yet.)

Thank you everyone for the feedback and enthusiasm. We appreciate it. :-)

[1] https://github.com/polaris-catalog/polaris/pull/42

Cheers,
-Tyler

On 2024/07/31 07:30:07 Justin Mclean wrote:
> Hi,
> 
> I sent a reply earlier, but my email is acting up and looks like it didn’t get through. I have some concerns with this proposal.
> 
> In general, the incubator likes projects to have a code base and a small community, I’m not seeing a community here. Trying to build one during incubation can be difficult. We have recently rejected proposals in a similar state, asking them to come back when they have more of a community around the project.
> 
> The PPMC/committer split is unusual.
> 
> There seems to be little relation to people who have contributed to the project and the initial committer list. A large number of the people involved in commits (80+%) are from one vendor, with two exceptions, and two others have made one or two minor commits of a couple of lines.
> 
> Adding people to PPMC to help out also seems unusual, as that is the mentor's job.
> 
> In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.
> 
> In short, the initial commit list looks problematic to me.
> 
> Kind Regards,
> Justin
> 
> P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Dave Fisher <wa...@apache.org>]

> On Aug 1, 2024, at 7:23 PM, Justin Mclean <ju...@classsoftware.com> wrote:
> 
> Hi,
> 
> My issue is not that this is a single-vendor project or that people have been recognised for non-code contributions - both are fine. I would suggest that the project slims down the initial committer list and not split it into PPMC and committers. As part of the incubating process, you can vote in new committers and PPMC members for their contributions.

I suppose that the initial non-ppmc committers can be voted onto the ppmc. To the IPMC I think that the growth to measure is the people beyond those on this list. But as the saying goes its “six of one and half dozen of another”.

Best,
Dave

> 
> Kind Regards,
> Justin
> 
>> On 2 Aug 2024, at 3:33 am, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> 
>> Hi Dave,
>> 
>> Thanks for your message.
>> 
>> I'm a bit confused now: it seems that we have a consensus that the
>> Polaris proposal is solid and correct (according to the number of
>> positive feedback in this discussion so far).
>> 
>> Justin expressed concerns, asking to update the proposal. As said, I
>> understand the comment, and would be happy to change the proposal.
>> 
>> Who's having the same concern and request to change the proposal ?
>> Should we move forward to a vote with the current proposal as Dave proposed ?
>> 
>> I don't want to frustrate anyone here. I have the feeling that if I
>> don't change the proposal, at least Justin would not be happy, and, on
>> the other hand, if I change the proposal, others would not be happy.
>> So, I'm open to suggestions.
>> 
>> I think the Polaris proposal is solid, the contributions (code and non
>> code) are there, the community is small but eager to grow (and that's
>> the purpose of the incubator).
>> 
>> Regards
>> JB
>> 
>> 
>> On Thu, Aug 1, 2024 at 6:39 PM Dave Fisher <wa...@apache.org> wrote:
>>> 
>>> Hi Justin,
>>> 
>>>> In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.
>>> 
>>> When and where should a “single vendor” collaborate on a common code project? If we block “single vendor” projects from Incubation then it’s time to start one of our periodic the Incubator is broken discussions.
>>> 
>>> The Incubator has many retired podlings that failed to form a community. That went from 3 or 4 to 1 or 0 community members. One example was ODFToolkit which came in at the same time as OpenOffice.org. ODFToolkit lingered for about 7 or 8 years. OpenOffice.org had an unusual anyone can signup to be an Initial Committer which brought the number to 73! As the current VP, OpenOffice I would not be wrong to say that the project would be in the Attic if that approach were not taken.
>>> 
>>>> In short, the initial commit list looks problematic to me.
>>> 
>>> Podlings and PMCs are not one size fits all.
>>> 
>>>> 
>>>> P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
>>> 
>>> That is part of Incubation.
>>> 
>>> Best,
>>> Dave
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Justin,

Thanks, it's clearer to me now. I updated the proposal according to
your comments:
- using only "Initial Committers" instead of the split between PPMC
members and committers
- slim down from 18 to 8 names

I give more time to this discussion today, I will move forward to a
vote (as proposed by Dave) tonight or tomorrow morning (my time).

Thanks Justin and Dave for your inputs !

Regards
JB

On Fri, Aug 2, 2024 at 4:23 AM Justin Mclean <ju...@classsoftware.com> wrote:
>
> Hi,
>
> My issue is not that this is a single-vendor project or that people have been recognised for non-code contributions - both are fine. I would suggest that the project slims down the initial committer list and not split it into PPMC and committers. As part of the incubating process, you can vote in new committers and PPMC members for their contributions.
>
> Kind Regards,
> Justin
>
> > On 2 Aug 2024, at 3:33 am, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi Dave,
> >
> > Thanks for your message.
> >
> > I'm a bit confused now: it seems that we have a consensus that the
> > Polaris proposal is solid and correct (according to the number of
> > positive feedback in this discussion so far).
> >
> > Justin expressed concerns, asking to update the proposal. As said, I
> > understand the comment, and would be happy to change the proposal.
> >
> > Who's having the same concern and request to change the proposal ?
> > Should we move forward to a vote with the current proposal as Dave proposed ?
> >
> > I don't want to frustrate anyone here. I have the feeling that if I
> > don't change the proposal, at least Justin would not be happy, and, on
> > the other hand, if I change the proposal, others would not be happy.
> > So, I'm open to suggestions.
> >
> > I think the Polaris proposal is solid, the contributions (code and non
> > code) are there, the community is small but eager to grow (and that's
> > the purpose of the incubator).
> >
> > Regards
> > JB
> >
> >
> > On Thu, Aug 1, 2024 at 6:39 PM Dave Fisher <wa...@apache.org> wrote:
> >>
> >> Hi Justin,
> >>
> >>> In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.
> >>
> >> When and where should a “single vendor” collaborate on a common code project? If we block “single vendor” projects from Incubation then it’s time to start one of our periodic the Incubator is broken discussions.
> >>
> >> The Incubator has many retired podlings that failed to form a community. That went from 3 or 4 to 1 or 0 community members. One example was ODFToolkit which came in at the same time as OpenOffice.org. ODFToolkit lingered for about 7 or 8 years. OpenOffice.org had an unusual anyone can signup to be an Initial Committer which brought the number to 73! As the current VP, OpenOffice I would not be wrong to say that the project would be in the Attic if that approach were not taken.
> >>
> >>> In short, the initial commit list looks problematic to me.
> >>
> >> Podlings and PMCs are not one size fits all.
> >>
> >>>
> >>> P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
> >>
> >> That is part of Incubation.
> >>
> >> Best,
> >> Dave
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Justin Mclean <ju...@classsoftware.com>]

Hi,

My issue is not that this is a single-vendor project or that people have been recognised for non-code contributions - both are fine. I would suggest that the project slims down the initial committer list and not split it into PPMC and committers. As part of the incubating process, you can vote in new committers and PPMC members for their contributions.

Kind Regards,
Justin

> On 2 Aug 2024, at 3:33 am, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> 
> Hi Dave,
> 
> Thanks for your message.
> 
> I'm a bit confused now: it seems that we have a consensus that the
> Polaris proposal is solid and correct (according to the number of
> positive feedback in this discussion so far).
> 
> Justin expressed concerns, asking to update the proposal. As said, I
> understand the comment, and would be happy to change the proposal.
> 
> Who's having the same concern and request to change the proposal ?
> Should we move forward to a vote with the current proposal as Dave proposed ?
> 
> I don't want to frustrate anyone here. I have the feeling that if I
> don't change the proposal, at least Justin would not be happy, and, on
> the other hand, if I change the proposal, others would not be happy.
> So, I'm open to suggestions.
> 
> I think the Polaris proposal is solid, the contributions (code and non
> code) are there, the community is small but eager to grow (and that's
> the purpose of the incubator).
> 
> Regards
> JB
> 
> 
> On Thu, Aug 1, 2024 at 6:39 PM Dave Fisher <wa...@apache.org> wrote:
>> 
>> Hi Justin,
>> 
>>> In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.
>> 
>> When and where should a “single vendor” collaborate on a common code project? If we block “single vendor” projects from Incubation then it’s time to start one of our periodic the Incubator is broken discussions.
>> 
>> The Incubator has many retired podlings that failed to form a community. That went from 3 or 4 to 1 or 0 community members. One example was ODFToolkit which came in at the same time as OpenOffice.org. ODFToolkit lingered for about 7 or 8 years. OpenOffice.org had an unusual anyone can signup to be an Initial Committer which brought the number to 73! As the current VP, OpenOffice I would not be wrong to say that the project would be in the Attic if that approach were not taken.
>> 
>>> In short, the initial commit list looks problematic to me.
>> 
>> Podlings and PMCs are not one size fits all.
>> 
>>> 
>>> P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
>> 
>> That is part of Incubation.
>> 
>> Best,
>> Dave
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Dave,

Thanks for your message.

I'm a bit confused now: it seems that we have a consensus that the
Polaris proposal is solid and correct (according to the number of
positive feedback in this discussion so far).

Justin expressed concerns, asking to update the proposal. As said, I
understand the comment, and would be happy to change the proposal.

Who's having the same concern and request to change the proposal ?
Should we move forward to a vote with the current proposal as Dave proposed ?

I don't want to frustrate anyone here. I have the feeling that if I
don't change the proposal, at least Justin would not be happy, and, on
the other hand, if I change the proposal, others would not be happy.
So, I'm open to suggestions.

I think the Polaris proposal is solid, the contributions (code and non
code) are there, the community is small but eager to grow (and that's
the purpose of the incubator).

Regards
JB


On Thu, Aug 1, 2024 at 6:39 PM Dave Fisher <wa...@apache.org> wrote:
>
> Hi Justin,
>
> > In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.
>
> When and where should a “single vendor” collaborate on a common code project? If we block “single vendor” projects from Incubation then it’s time to start one of our periodic the Incubator is broken discussions.
>
> The Incubator has many retired podlings that failed to form a community. That went from 3 or 4 to 1 or 0 community members. One example was ODFToolkit which came in at the same time as OpenOffice.org. ODFToolkit lingered for about 7 or 8 years. OpenOffice.org had an unusual anyone can signup to be an Initial Committer which brought the number to 73! As the current VP, OpenOffice I would not be wrong to say that the project would be in the Attic if that approach were not taken.
>
> > In short, the initial commit list looks problematic to me.
>
> Podlings and PMCs are not one size fits all.
>
> >
> > P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
>
> That is part of Incubation.
>
> Best,
> Dave
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Dave Fisher <wa...@apache.org>]

Hi Justin,

> In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.

When and where should a “single vendor” collaborate on a common code project? If we block “single vendor” projects from Incubation then it’s time to start one of our periodic the Incubator is broken discussions.

The Incubator has many retired podlings that failed to form a community. That went from 3 or 4 to 1 or 0 community members. One example was ODFToolkit which came in at the same time as OpenOffice.org. ODFToolkit lingered for about 7 or 8 years. OpenOffice.org had an unusual anyone can signup to be an Initial Committer which brought the number to 73! As the current VP, OpenOffice I would not be wrong to say that the project would be in the Attic if that approach were not taken.

> In short, the initial commit list looks problematic to me.

Podlings and PMCs are not one size fits all.

> 
> P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.

That is part of Incubation. 

Best,
Dave


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Justin Mclean <ju...@classsoftware.com>]

Hi,

I sent a reply earlier, but my email is acting up and looks like it didn’t get through. I have some concerns with this proposal.

In general, the incubator likes projects to have a code base and a small community, I’m not seeing a community here. Trying to build one during incubation can be difficult. We have recently rejected proposals in a similar state, asking them to come back when they have more of a community around the project.

The PPMC/committer split is unusual.

There seems to be little relation to people who have contributed to the project and the initial committer list. A large number of the people involved in commits (80+%) are from one vendor, with two exceptions, and two others have made one or two minor commits of a couple of lines.

Adding people to PPMC to help out also seems unusual, as that is the mentor's job.

In short, this seems to me (and I could be wrong) like a project mostly from a single vendor, but the proposal has been made to make it look like more people are involved. It may well be that these people will be involved, but I’d prefer if the project was upfront about this and added committers the usual way during incubation.

In short, the initial commit list looks problematic to me.

Kind Regards,
Justin

P.S. The repo landing page/readme has some ASF trademark issues that would be good to address.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Kent Yao <ya...@apache.org>]

I added myself to the proposal.

Thank you JB,

Kent

On 2024/07/31 10:27:41 Jean-Baptiste Onofré wrote:
> Hi Kent
> 
> Happy to add you as mentor! Do you want to update the proposal wiki or I do
> ?
> 
> Thanks !
> 
> Regards
> JB
> 
> Le mer. 31 juil. 2024 à 09:31, Kent Yao <ya...@apache.org> a écrit :
> 
> > +1 on Poraris entering to Apache. I'm interested in helping as a mentor if
> > needed.
> >
> > Kent Yao
> >
> > On 2024/07/31 07:02:05 ConradJam wrote:
> > > As members of the Amoro project, our team is thrilled to see the growing
> > > attention towards Amoro.
> > >
> > > We are excited about Polaris becoming open source, as it opens up greater
> > > possibilities for future collaboration with the Amoro community.
> > >
> > > Amoro focuses on data lake formats and aims to provide optimization
> > > services and enhancements for the lake. Our primary goal is to offer
> > > optimization services that support multiple table formats (though
> > > currently, Iceberg is the most supported), such as small file
> > optimization,
> > > Z-order sorting optimization, and future index optimization.
> > >
> > > Amoro provides both Internal Catalog and External Catalog methods to
> > > optimize lake tables. To gather optimization information, we have
> > conducted
> > > some catalog management work.
> > >
> > > I often hear people comparing Gravitino and Polaris as potential
> > > competitors to Amoro, which I think is a misconception (I noticed that
> > some
> > > previous discussions about Amoro's positioning seemed unclear, so I
> > wanted
> > > to clarify this).
> > >
> > > While there might be some overlap between Amoro, Gravitino, and Polaris:
> > >
> > > - Gravitino focuses on unified metadata management across various areas,
> > > including Kafka and AI, not just on data lakes.
> > > - Polaris is an interoperable, open-source catalog for Apache Iceberg.
> > >
> > > If there are any errors, please correct them.
> > >
> > > Amoro plans to support both Polaris and Gravitino in the future.
> > > Additionally, the Amoro community will continue to engage with the
> > > Gravitino and Polaris communities to foster more collaborative efforts in
> > > lake optimization.
> > >
> > > [1] Amoro docs: https://amoro.apache.org/docs/latest/
> > > [2] Gravitino docs:  https://datastrato.ai/docs/0.5.1/
> > > [3] Polaris docs: https://polaris.io/
> > >
> > > Jack Ye <ye...@gmail.com> 于2024年7月31日周三 13:22写道：
> > >
> > > > > What's the difference between this project and Amoro
> > > >
> > > > Here is my $0.01, please correct me if I am wrong, especially for
> > people
> > > > working on Amoro and Gravitino.
> > > >
> > > > I think Apache Amoro is focused more on being a self-contained complete
> > > > data lakehouse management and ingestion system. It is a complete
> > solution
> > > > with its own connectors in engines like Spark [1], and customized
> > > > mixed-format integrations in engines like Trino [2]. Polaris is mostly
> > > > focused on the data catalog aspect of a data lakehouse, and offers an
> > open
> > > > source vendor-neutral Iceberg catalog with additional governance
> > support.
> > > > By integrating with the Iceberg REST catalog interface, the intention
> > is
> > > > for it to leverage Iceberg for all the engine integrations to begin
> > with.
> > > > Similarly, any table management or ingestion system that works with
> > Iceberg
> > > > REST API will be able to be plugged in to directly work with Polaris.
> > So
> > > > you could imagine it could be possible for an Iceberg table to be
> > ingested
> > > > and managed by Amoro, but cataloged using Polaris.
> > > >
> > > > This does make Polaris more similar to Apache Gravitino. However, I
> > think
> > > > the key difference between them is that the emphasis of Gravitino is
> > more
> > > > breath-first on aspects like multi-format, multi-catalog,
> > multi-datasource,
> > > > different data catalog objects in AI [3], etc. It exposes different
> > sets of
> > > > APIs for different purposes, with Iceberg REST API being a part of it
> > for
> > > > the Iceberg tables, and other APIs for other data sources [4]. Polaris
> > is
> > > > more depth-first on Iceberg at this moment. Our future plan does say
> > that
> > > > it could extend to non-Iceberg data lakes, and there could be some
> > overlap
> > > > at that time. But even then, there could be different ways to achieve
> > such
> > > > support. For example, we could surface Hive Parquet tables as Iceberg
> > > > tables, if the Iceberg REST catalog standard can be updated to
> > accommodate
> > > > that. There could also be potential collaborations between Polaris and
> > > > Gravitino to achieve the goal together, and I am personally pretty
> > excited
> > > > about that opportunity.
> > > >
> > > > Best,
> > > > Jack Ye
> > > >
> > > > [1] https://amoro.apache.org/docs/latest/spark-configuration/
> > > > [2] https://amoro.apache.org/docs/latest/trino/#mixed-format
> > > > [3]
> > > >
> > > >
> > https://github.com/apache/gravitino-site/blob/10a967f18730c28018e064f3ee1ddd3cc32aa506/src/components/HomepageFeatures/index.tsx#L74
> > > > [4] https://github.com/apache/gravitino/tree/main/catalogs
> > > >
> > > > On Tue, Jul 30, 2024 at 10:06 PM Jean-Baptiste Onofré <jb@nanthrax.net
> > >
> > > > wrote:
> > > >
> > > > > Hi Manu
> > > > >
> > > > > Thanks for the details !
> > > > > I agree with you. As mentor on Gravitino, I would be more than happy
> > > > > to connect the two podlings.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > AFAIK, Amoro is a management system with optimization service,
> > catalog
> > > > > > service, etc. It has a built-in catalog but can also work with
> > other
> > > > > > catalogs like Polaris.
> > > > > > I think Polaris is more comparable to Gravitino which entered the
> > > > > incubator
> > > > > > recently. It would be interesting to see how these two communities
> > can
> > > > > > collaborate.
> > > > > >
> > > > > > Regards,
> > > > > > Manu
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > The proposal is more generic: today it's Apache Iceberg, but
> > after
> > > > the
> > > > > > > discussions with the initial community we agreed it could make
> > sense
> > > > > > > to address other use cases.
> > > > > > >
> > > > > > > I don't know Amoro in details, but I am happy to bridge the
> > > > > > > communities to work together.
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > > Hi, JB
> > > > > > > >
> > > > > > > > Thank you for starting this thread; it's great to see an
> > increasing
> > > > > > > number of projects being developed around Iceberg.
> > > > > > > >
> > > > > > > > I have two questions:
> > > > > > > >
> > > > > > > > - The polaris github repo said it's "an open source catalog for
> > > > > Apache
> > > > > > > Iceberg", but the proposal changed into "a catalog for data
> > lakes".
> > > > > Does it
> > > > > > > mean Polaris's scope has been changed?
> > > > > > > > - What's the difference between this project and Amoro:
> > > > > > > https://github.com/apache/amoro? How do these two communities
> > > > > collaborate?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > > > > > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net>
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Hi Dave,
> > > > > > > > >>
> > > > > > > > >> That's a good question. The main reason is because we wanted
> > > > > people
> > > > > > > > >> with Apache experience in the PPMC to mentor the committers
> > and
> > > > > > > > >> contributors heading to PPMC as well.
> > > > > > > > >> Also, the initial committers worked closely with PPMC
> > guidance
> > > > > > > > >> (explaining the ICLA, good practice, etc).
> > > > > > > > >> So, we wanted to have PPMC acting more as mentor (both
> > > > > technically but
> > > > > > > > >> also with their Apache experience) with committers.
> > > > > > > > >
> > > > > > > > > That makes sense. Are any of the proposed PPMC members also
> > ASF
> > > > > Members
> > > > > > > > > and/or potentially future Mentors?
> > > > > > > > >
> > > > > > > > >> If it's problematic, we can start only with the PPMC group
> > and
> > > > > invite
> > > > > > > > >> new committers/PPMC members during incubation period.
> > > > > > > > >
> > > > > > > > > No problem. It will actually provide the Mentors and later
> > the
> > > > IPMC
> > > > > > > > > additional data to see if the PPMC is properly growing the
> > PPMC
> > > > and
> > > > > > > > > Committer base.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Dave
> > > > > > > > >
> > > > > > > > >>
> > > > > > > > >> Regards
> > > > > > > > >> JB
> > > > > > > > >>
> > > > > > > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <
> > wave@apache.org>
> > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> Hi JB,
> > > > > > > > >>>
> > > > > > > > >>> An interesting project that looks pretty mature.
> > > > > > > > >>>
> > > > > > > > >>> I’m curious about the split between Initial PPMC and
> > initial
> > > > > > > Committer. In the usual case a new podling will have all of the
> > > > Initial
> > > > > > > Committers on the PPMC. Can you tell us why this is not the case
> > with
> > > > > > > Polaris?
> > > > > > > > >>>
> > > > > > > > >>> Best,
> > > > > > > > >>> Dave
> > > > > > > > >>>
> > > > > > > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net>
> > > > > > > wrote:
> > > > > > > > >>>>
> > > > > > > > >>>> Hi folks,
> > > > > > > > >>>>
> > > > > > > > >>>> We would like to propose a new project to the ASF
> > incubator:
> > > > > > > Polaris.
> > > > > > > > >>>>
> > > > > > > > >>>> Polaris is a catalog for data lakes. It provides new
> > levels of
> > > > > > > choice,
> > > > > > > > >>>> flexibility and control over data, with full enterprise
> > > > > security and
> > > > > > > > >>>> Apache Iceberg interoperability across a multitude of
> > engines
> > > > > and
> > > > > > > > >>>> infrastructure. Polaris builds on standards such as those
> > > > > created by
> > > > > > > > >>>> Apache Iceberg, providing the following benefits for the
> > > > > ecosystem:
> > > > > > > > >>>> * Multi-engine interoperability over a single copy of
> > data,
> > > > > > > > >>>> eliminating the need for moving and copying data across
> > > > > different
> > > > > > > > >>>> engines and catalogs.
> > > > > > > > >>>> * An interoperable security model providing a unified
> > > > > authorization
> > > > > > > > >>>> layer independent from the engines processing analytical
> > > > tables.
> > > > > > > > >>>> * For multi-catalog scenarios, a unified catalog level
> > view of
> > > > > data
> > > > > > > > >>>> across multiple catalogs via catalog notification
> > > > integrations.
> > > > > > > > >>>> * The ability to host Polaris Catalog on the
> > infrastructure of
> > > > > your
> > > > > > > choice.
> > > > > > > > >>>>
> > > > > > > > >>>> Here is the proposal:
> > > > > > > > >>>>
> > > > > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > > > > > > >>>>
> > > > > > > > >>>> Comments and feedback are welcome.
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks!
> > > > > > > > >>>> Regards
> > > > > > > > >>>> JB
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >>>> To unsubscribe, e-mail:
> > > > > general-unsubscribe@incubator.apache.org
> > > > > > > > >>>> For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > ---------------------------------------------------------------------
> > > > > > > > >>> To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > > >>> For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > ---------------------------------------------------------------------
> > > > > > > > >> To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > > >> For additional commands, e-mail:
> > > > > general-help@incubator.apache.org
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > >
> > > > > > > > --
> > > > > > > > Xuanwo
> > > > > > > >
> > > > > > > > https://xuanwo.io/
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Kent

Happy to add you as mentor! Do you want to update the proposal wiki or I do
?

Thanks !

Regards
JB

Le mer. 31 juil. 2024 à 09:31, Kent Yao <ya...@apache.org> a écrit :

> +1 on Poraris entering to Apache. I'm interested in helping as a mentor if
> needed.
>
> Kent Yao
>
> On 2024/07/31 07:02:05 ConradJam wrote:
> > As members of the Amoro project, our team is thrilled to see the growing
> > attention towards Amoro.
> >
> > We are excited about Polaris becoming open source, as it opens up greater
> > possibilities for future collaboration with the Amoro community.
> >
> > Amoro focuses on data lake formats and aims to provide optimization
> > services and enhancements for the lake. Our primary goal is to offer
> > optimization services that support multiple table formats (though
> > currently, Iceberg is the most supported), such as small file
> optimization,
> > Z-order sorting optimization, and future index optimization.
> >
> > Amoro provides both Internal Catalog and External Catalog methods to
> > optimize lake tables. To gather optimization information, we have
> conducted
> > some catalog management work.
> >
> > I often hear people comparing Gravitino and Polaris as potential
> > competitors to Amoro, which I think is a misconception (I noticed that
> some
> > previous discussions about Amoro's positioning seemed unclear, so I
> wanted
> > to clarify this).
> >
> > While there might be some overlap between Amoro, Gravitino, and Polaris:
> >
> > - Gravitino focuses on unified metadata management across various areas,
> > including Kafka and AI, not just on data lakes.
> > - Polaris is an interoperable, open-source catalog for Apache Iceberg.
> >
> > If there are any errors, please correct them.
> >
> > Amoro plans to support both Polaris and Gravitino in the future.
> > Additionally, the Amoro community will continue to engage with the
> > Gravitino and Polaris communities to foster more collaborative efforts in
> > lake optimization.
> >
> > [1] Amoro docs: https://amoro.apache.org/docs/latest/
> > [2] Gravitino docs:  https://datastrato.ai/docs/0.5.1/
> > [3] Polaris docs: https://polaris.io/
> >
> > Jack Ye <ye...@gmail.com> 于2024年7月31日周三 13:22写道：
> >
> > > > What's the difference between this project and Amoro
> > >
> > > Here is my $0.01, please correct me if I am wrong, especially for
> people
> > > working on Amoro and Gravitino.
> > >
> > > I think Apache Amoro is focused more on being a self-contained complete
> > > data lakehouse management and ingestion system. It is a complete
> solution
> > > with its own connectors in engines like Spark [1], and customized
> > > mixed-format integrations in engines like Trino [2]. Polaris is mostly
> > > focused on the data catalog aspect of a data lakehouse, and offers an
> open
> > > source vendor-neutral Iceberg catalog with additional governance
> support.
> > > By integrating with the Iceberg REST catalog interface, the intention
> is
> > > for it to leverage Iceberg for all the engine integrations to begin
> with.
> > > Similarly, any table management or ingestion system that works with
> Iceberg
> > > REST API will be able to be plugged in to directly work with Polaris.
> So
> > > you could imagine it could be possible for an Iceberg table to be
> ingested
> > > and managed by Amoro, but cataloged using Polaris.
> > >
> > > This does make Polaris more similar to Apache Gravitino. However, I
> think
> > > the key difference between them is that the emphasis of Gravitino is
> more
> > > breath-first on aspects like multi-format, multi-catalog,
> multi-datasource,
> > > different data catalog objects in AI [3], etc. It exposes different
> sets of
> > > APIs for different purposes, with Iceberg REST API being a part of it
> for
> > > the Iceberg tables, and other APIs for other data sources [4]. Polaris
> is
> > > more depth-first on Iceberg at this moment. Our future plan does say
> that
> > > it could extend to non-Iceberg data lakes, and there could be some
> overlap
> > > at that time. But even then, there could be different ways to achieve
> such
> > > support. For example, we could surface Hive Parquet tables as Iceberg
> > > tables, if the Iceberg REST catalog standard can be updated to
> accommodate
> > > that. There could also be potential collaborations between Polaris and
> > > Gravitino to achieve the goal together, and I am personally pretty
> excited
> > > about that opportunity.
> > >
> > > Best,
> > > Jack Ye
> > >
> > > [1] https://amoro.apache.org/docs/latest/spark-configuration/
> > > [2] https://amoro.apache.org/docs/latest/trino/#mixed-format
> > > [3]
> > >
> > >
> https://github.com/apache/gravitino-site/blob/10a967f18730c28018e064f3ee1ddd3cc32aa506/src/components/HomepageFeatures/index.tsx#L74
> > > [4] https://github.com/apache/gravitino/tree/main/catalogs
> > >
> > > On Tue, Jul 30, 2024 at 10:06 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > wrote:
> > >
> > > > Hi Manu
> > > >
> > > > Thanks for the details !
> > > > I agree with you. As mentor on Gravitino, I would be more than happy
> > > > to connect the two podlings.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com>
> > > > wrote:
> > > > >
> > > > > AFAIK, Amoro is a management system with optimization service,
> catalog
> > > > > service, etc. It has a built-in catalog but can also work with
> other
> > > > > catalogs like Polaris.
> > > > > I think Polaris is more comparable to Gravitino which entered the
> > > > incubator
> > > > > recently. It would be interesting to see how these two communities
> can
> > > > > collaborate.
> > > > >
> > > > > Regards,
> > > > > Manu
> > > > >
> > > > >
> > > > > On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > The proposal is more generic: today it's Apache Iceberg, but
> after
> > > the
> > > > > > discussions with the initial community we agreed it could make
> sense
> > > > > > to address other use cases.
> > > > > >
> > > > > > I don't know Amoro in details, but I am happy to bridge the
> > > > > > communities to work together.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org>
> wrote:
> > > > > > >
> > > > > > > Hi, JB
> > > > > > >
> > > > > > > Thank you for starting this thread; it's great to see an
> increasing
> > > > > > number of projects being developed around Iceberg.
> > > > > > >
> > > > > > > I have two questions:
> > > > > > >
> > > > > > > - The polaris github repo said it's "an open source catalog for
> > > > Apache
> > > > > > Iceberg", but the proposal changed into "a catalog for data
> lakes".
> > > > Does it
> > > > > > mean Polaris's scope has been changed?
> > > > > > > - What's the difference between this project and Amoro:
> > > > > > https://github.com/apache/amoro? How do these two communities
> > > > collaborate?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > > > > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hi Dave,
> > > > > > > >>
> > > > > > > >> That's a good question. The main reason is because we wanted
> > > > people
> > > > > > > >> with Apache experience in the PPMC to mentor the committers
> and
> > > > > > > >> contributors heading to PPMC as well.
> > > > > > > >> Also, the initial committers worked closely with PPMC
> guidance
> > > > > > > >> (explaining the ICLA, good practice, etc).
> > > > > > > >> So, we wanted to have PPMC acting more as mentor (both
> > > > technically but
> > > > > > > >> also with their Apache experience) with committers.
> > > > > > > >
> > > > > > > > That makes sense. Are any of the proposed PPMC members also
> ASF
> > > > Members
> > > > > > > > and/or potentially future Mentors?
> > > > > > > >
> > > > > > > >> If it's problematic, we can start only with the PPMC group
> and
> > > > invite
> > > > > > > >> new committers/PPMC members during incubation period.
> > > > > > > >
> > > > > > > > No problem. It will actually provide the Mentors and later
> the
> > > IPMC
> > > > > > > > additional data to see if the PPMC is properly growing the
> PPMC
> > > and
> > > > > > > > Committer base.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Dave
> > > > > > > >
> > > > > > > >>
> > > > > > > >> Regards
> > > > > > > >> JB
> > > > > > > >>
> > > > > > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <
> wave@apache.org>
> > > > wrote:
> > > > > > > >>>
> > > > > > > >>> Hi JB,
> > > > > > > >>>
> > > > > > > >>> An interesting project that looks pretty mature.
> > > > > > > >>>
> > > > > > > >>> I’m curious about the split between Initial PPMC and
> initial
> > > > > > Committer. In the usual case a new podling will have all of the
> > > Initial
> > > > > > Committers on the PPMC. Can you tell us why this is not the case
> with
> > > > > > Polaris?
> > > > > > > >>>
> > > > > > > >>> Best,
> > > > > > > >>> Dave
> > > > > > > >>>
> > > > > > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>> Hi folks,
> > > > > > > >>>>
> > > > > > > >>>> We would like to propose a new project to the ASF
> incubator:
> > > > > > Polaris.
> > > > > > > >>>>
> > > > > > > >>>> Polaris is a catalog for data lakes. It provides new
> levels of
> > > > > > choice,
> > > > > > > >>>> flexibility and control over data, with full enterprise
> > > > security and
> > > > > > > >>>> Apache Iceberg interoperability across a multitude of
> engines
> > > > and
> > > > > > > >>>> infrastructure. Polaris builds on standards such as those
> > > > created by
> > > > > > > >>>> Apache Iceberg, providing the following benefits for the
> > > > ecosystem:
> > > > > > > >>>> * Multi-engine interoperability over a single copy of
> data,
> > > > > > > >>>> eliminating the need for moving and copying data across
> > > > different
> > > > > > > >>>> engines and catalogs.
> > > > > > > >>>> * An interoperable security model providing a unified
> > > > authorization
> > > > > > > >>>> layer independent from the engines processing analytical
> > > tables.
> > > > > > > >>>> * For multi-catalog scenarios, a unified catalog level
> view of
> > > > data
> > > > > > > >>>> across multiple catalogs via catalog notification
> > > integrations.
> > > > > > > >>>> * The ability to host Polaris Catalog on the
> infrastructure of
> > > > your
> > > > > > choice.
> > > > > > > >>>>
> > > > > > > >>>> Here is the proposal:
> > > > > > > >>>>
> > > > > >
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > > > > > >>>>
> > > > > > > >>>> Comments and feedback are welcome.
> > > > > > > >>>>
> > > > > > > >>>> Thanks!
> > > > > > > >>>> Regards
> > > > > > > >>>> JB
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>> To unsubscribe, e-mail:
> > > > general-unsubscribe@incubator.apache.org
> > > > > > > >>>> For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > > >>> To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > >>> For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > > >> To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > > >> For additional commands, e-mail:
> > > > general-help@incubator.apache.org
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > >
> > > > > > > --
> > > > > > > Xuanwo
> > > > > > >
> > > > > > > https://xuanwo.io/
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > > >
> > > > > >
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [DISCUSS] PolarisProposal to the incubator

[Kent Yao <ya...@apache.org>]

+1 on Poraris entering to Apache. I'm interested in helping as a mentor if needed.

Kent Yao

On 2024/07/31 07:02:05 ConradJam wrote:
> As members of the Amoro project, our team is thrilled to see the growing
> attention towards Amoro.
> 
> We are excited about Polaris becoming open source, as it opens up greater
> possibilities for future collaboration with the Amoro community.
> 
> Amoro focuses on data lake formats and aims to provide optimization
> services and enhancements for the lake. Our primary goal is to offer
> optimization services that support multiple table formats (though
> currently, Iceberg is the most supported), such as small file optimization,
> Z-order sorting optimization, and future index optimization.
> 
> Amoro provides both Internal Catalog and External Catalog methods to
> optimize lake tables. To gather optimization information, we have conducted
> some catalog management work.
> 
> I often hear people comparing Gravitino and Polaris as potential
> competitors to Amoro, which I think is a misconception (I noticed that some
> previous discussions about Amoro's positioning seemed unclear, so I wanted
> to clarify this).
> 
> While there might be some overlap between Amoro, Gravitino, and Polaris:
> 
> - Gravitino focuses on unified metadata management across various areas,
> including Kafka and AI, not just on data lakes.
> - Polaris is an interoperable, open-source catalog for Apache Iceberg.
> 
> If there are any errors, please correct them.
> 
> Amoro plans to support both Polaris and Gravitino in the future.
> Additionally, the Amoro community will continue to engage with the
> Gravitino and Polaris communities to foster more collaborative efforts in
> lake optimization.
> 
> [1] Amoro docs: https://amoro.apache.org/docs/latest/
> [2] Gravitino docs:  https://datastrato.ai/docs/0.5.1/
> [3] Polaris docs: https://polaris.io/
> 
> Jack Ye <ye...@gmail.com> 于2024年7月31日周三 13:22写道：
> 
> > > What's the difference between this project and Amoro
> >
> > Here is my $0.01, please correct me if I am wrong, especially for people
> > working on Amoro and Gravitino.
> >
> > I think Apache Amoro is focused more on being a self-contained complete
> > data lakehouse management and ingestion system. It is a complete solution
> > with its own connectors in engines like Spark [1], and customized
> > mixed-format integrations in engines like Trino [2]. Polaris is mostly
> > focused on the data catalog aspect of a data lakehouse, and offers an open
> > source vendor-neutral Iceberg catalog with additional governance support.
> > By integrating with the Iceberg REST catalog interface, the intention is
> > for it to leverage Iceberg for all the engine integrations to begin with.
> > Similarly, any table management or ingestion system that works with Iceberg
> > REST API will be able to be plugged in to directly work with Polaris. So
> > you could imagine it could be possible for an Iceberg table to be ingested
> > and managed by Amoro, but cataloged using Polaris.
> >
> > This does make Polaris more similar to Apache Gravitino. However, I think
> > the key difference between them is that the emphasis of Gravitino is more
> > breath-first on aspects like multi-format, multi-catalog, multi-datasource,
> > different data catalog objects in AI [3], etc. It exposes different sets of
> > APIs for different purposes, with Iceberg REST API being a part of it for
> > the Iceberg tables, and other APIs for other data sources [4]. Polaris is
> > more depth-first on Iceberg at this moment. Our future plan does say that
> > it could extend to non-Iceberg data lakes, and there could be some overlap
> > at that time. But even then, there could be different ways to achieve such
> > support. For example, we could surface Hive Parquet tables as Iceberg
> > tables, if the Iceberg REST catalog standard can be updated to accommodate
> > that. There could also be potential collaborations between Polaris and
> > Gravitino to achieve the goal together, and I am personally pretty excited
> > about that opportunity.
> >
> > Best,
> > Jack Ye
> >
> > [1] https://amoro.apache.org/docs/latest/spark-configuration/
> > [2] https://amoro.apache.org/docs/latest/trino/#mixed-format
> > [3]
> >
> > https://github.com/apache/gravitino-site/blob/10a967f18730c28018e064f3ee1ddd3cc32aa506/src/components/HomepageFeatures/index.tsx#L74
> > [4] https://github.com/apache/gravitino/tree/main/catalogs
> >
> > On Tue, Jul 30, 2024 at 10:06 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi Manu
> > >
> > > Thanks for the details !
> > > I agree with you. As mentor on Gravitino, I would be more than happy
> > > to connect the two podlings.
> > >
> > > Regards
> > > JB
> > >
> > > On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com>
> > > wrote:
> > > >
> > > > AFAIK, Amoro is a management system with optimization service, catalog
> > > > service, etc. It has a built-in catalog but can also work with other
> > > > catalogs like Polaris.
> > > > I think Polaris is more comparable to Gravitino which entered the
> > > incubator
> > > > recently. It would be interesting to see how these two communities can
> > > > collaborate.
> > > >
> > > > Regards,
> > > > Manu
> > > >
> > > >
> > > > On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <jb@nanthrax.net
> > >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > The proposal is more generic: today it's Apache Iceberg, but after
> > the
> > > > > discussions with the initial community we agreed it could make sense
> > > > > to address other use cases.
> > > > >
> > > > > I don't know Amoro in details, but I am happy to bridge the
> > > > > communities to work together.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
> > > > > >
> > > > > > Hi, JB
> > > > > >
> > > > > > Thank you for starting this thread; it's great to see an increasing
> > > > > number of projects being developed around Iceberg.
> > > > > >
> > > > > > I have two questions:
> > > > > >
> > > > > > - The polaris github repo said it's "an open source catalog for
> > > Apache
> > > > > Iceberg", but the proposal changed into "a catalog for data lakes".
> > > Does it
> > > > > mean Polaris's scope has been changed?
> > > > > > - What's the difference between this project and Amoro:
> > > > > https://github.com/apache/amoro? How do these two communities
> > > collaborate?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > > > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > wrote:
> > > > > > >>
> > > > > > >> Hi Dave,
> > > > > > >>
> > > > > > >> That's a good question. The main reason is because we wanted
> > > people
> > > > > > >> with Apache experience in the PPMC to mentor the committers and
> > > > > > >> contributors heading to PPMC as well.
> > > > > > >> Also, the initial committers worked closely with PPMC guidance
> > > > > > >> (explaining the ICLA, good practice, etc).
> > > > > > >> So, we wanted to have PPMC acting more as mentor (both
> > > technically but
> > > > > > >> also with their Apache experience) with committers.
> > > > > > >
> > > > > > > That makes sense. Are any of the proposed PPMC members also ASF
> > > Members
> > > > > > > and/or potentially future Mentors?
> > > > > > >
> > > > > > >> If it's problematic, we can start only with the PPMC group and
> > > invite
> > > > > > >> new committers/PPMC members during incubation period.
> > > > > > >
> > > > > > > No problem. It will actually provide the Mentors and later the
> > IPMC
> > > > > > > additional data to see if the PPMC is properly growing the PPMC
> > and
> > > > > > > Committer base.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dave
> > > > > > >
> > > > > > >>
> > > > > > >> Regards
> > > > > > >> JB
> > > > > > >>
> > > > > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org>
> > > wrote:
> > > > > > >>>
> > > > > > >>> Hi JB,
> > > > > > >>>
> > > > > > >>> An interesting project that looks pretty mature.
> > > > > > >>>
> > > > > > >>> I’m curious about the split between Initial PPMC and initial
> > > > > Committer. In the usual case a new podling will have all of the
> > Initial
> > > > > Committers on the PPMC. Can you tell us why this is not the case with
> > > > > Polaris?
> > > > > > >>>
> > > > > > >>> Best,
> > > > > > >>> Dave
> > > > > > >>>
> > > > > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > wrote:
> > > > > > >>>>
> > > > > > >>>> Hi folks,
> > > > > > >>>>
> > > > > > >>>> We would like to propose a new project to the ASF incubator:
> > > > > Polaris.
> > > > > > >>>>
> > > > > > >>>> Polaris is a catalog for data lakes. It provides new levels of
> > > > > choice,
> > > > > > >>>> flexibility and control over data, with full enterprise
> > > security and
> > > > > > >>>> Apache Iceberg interoperability across a multitude of engines
> > > and
> > > > > > >>>> infrastructure. Polaris builds on standards such as those
> > > created by
> > > > > > >>>> Apache Iceberg, providing the following benefits for the
> > > ecosystem:
> > > > > > >>>> * Multi-engine interoperability over a single copy of data,
> > > > > > >>>> eliminating the need for moving and copying data across
> > > different
> > > > > > >>>> engines and catalogs.
> > > > > > >>>> * An interoperable security model providing a unified
> > > authorization
> > > > > > >>>> layer independent from the engines processing analytical
> > tables.
> > > > > > >>>> * For multi-catalog scenarios, a unified catalog level view of
> > > data
> > > > > > >>>> across multiple catalogs via catalog notification
> > integrations.
> > > > > > >>>> * The ability to host Polaris Catalog on the infrastructure of
> > > your
> > > > > choice.
> > > > > > >>>>
> > > > > > >>>> Here is the proposal:
> > > > > > >>>>
> > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > > > > >>>>
> > > > > > >>>> Comments and feedback are welcome.
> > > > > > >>>>
> > > > > > >>>> Thanks!
> > > > > > >>>> Regards
> > > > > > >>>> JB
> > > > > > >>>>
> > > > > > >>>>
> > > > > ---------------------------------------------------------------------
> > > > > > >>>> To unsubscribe, e-mail:
> > > general-unsubscribe@incubator.apache.org
> > > > > > >>>> For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > > >>> To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > >>> For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > ---------------------------------------------------------------------
> > > > > > >> To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > > >> For additional commands, e-mail:
> > > general-help@incubator.apache.org
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > >
> > > > > > --
> > > > > > Xuanwo
> > > > > >
> > > > > > https://xuanwo.io/
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> 
> 
> -- 
> Best
> 
> ConradJam
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[ConradJam <ja...@gmail.com>]

As members of the Amoro project, our team is thrilled to see the growing
attention towards Amoro.

We are excited about Polaris becoming open source, as it opens up greater
possibilities for future collaboration with the Amoro community.

Amoro focuses on data lake formats and aims to provide optimization
services and enhancements for the lake. Our primary goal is to offer
optimization services that support multiple table formats (though
currently, Iceberg is the most supported), such as small file optimization,
Z-order sorting optimization, and future index optimization.

Amoro provides both Internal Catalog and External Catalog methods to
optimize lake tables. To gather optimization information, we have conducted
some catalog management work.

I often hear people comparing Gravitino and Polaris as potential
competitors to Amoro, which I think is a misconception (I noticed that some
previous discussions about Amoro's positioning seemed unclear, so I wanted
to clarify this).

While there might be some overlap between Amoro, Gravitino, and Polaris:

- Gravitino focuses on unified metadata management across various areas,
including Kafka and AI, not just on data lakes.
- Polaris is an interoperable, open-source catalog for Apache Iceberg.

If there are any errors, please correct them.

Amoro plans to support both Polaris and Gravitino in the future.
Additionally, the Amoro community will continue to engage with the
Gravitino and Polaris communities to foster more collaborative efforts in
lake optimization.

[1] Amoro docs: https://amoro.apache.org/docs/latest/
[2] Gravitino docs:  https://datastrato.ai/docs/0.5.1/
[3] Polaris docs: https://polaris.io/

Jack Ye <ye...@gmail.com> 于2024年7月31日周三 13:22写道：

> > What's the difference between this project and Amoro
>
> Here is my $0.01, please correct me if I am wrong, especially for people
> working on Amoro and Gravitino.
>
> I think Apache Amoro is focused more on being a self-contained complete
> data lakehouse management and ingestion system. It is a complete solution
> with its own connectors in engines like Spark [1], and customized
> mixed-format integrations in engines like Trino [2]. Polaris is mostly
> focused on the data catalog aspect of a data lakehouse, and offers an open
> source vendor-neutral Iceberg catalog with additional governance support.
> By integrating with the Iceberg REST catalog interface, the intention is
> for it to leverage Iceberg for all the engine integrations to begin with.
> Similarly, any table management or ingestion system that works with Iceberg
> REST API will be able to be plugged in to directly work with Polaris. So
> you could imagine it could be possible for an Iceberg table to be ingested
> and managed by Amoro, but cataloged using Polaris.
>
> This does make Polaris more similar to Apache Gravitino. However, I think
> the key difference between them is that the emphasis of Gravitino is more
> breath-first on aspects like multi-format, multi-catalog, multi-datasource,
> different data catalog objects in AI [3], etc. It exposes different sets of
> APIs for different purposes, with Iceberg REST API being a part of it for
> the Iceberg tables, and other APIs for other data sources [4]. Polaris is
> more depth-first on Iceberg at this moment. Our future plan does say that
> it could extend to non-Iceberg data lakes, and there could be some overlap
> at that time. But even then, there could be different ways to achieve such
> support. For example, we could surface Hive Parquet tables as Iceberg
> tables, if the Iceberg REST catalog standard can be updated to accommodate
> that. There could also be potential collaborations between Polaris and
> Gravitino to achieve the goal together, and I am personally pretty excited
> about that opportunity.
>
> Best,
> Jack Ye
>
> [1] https://amoro.apache.org/docs/latest/spark-configuration/
> [2] https://amoro.apache.org/docs/latest/trino/#mixed-format
> [3]
>
> https://github.com/apache/gravitino-site/blob/10a967f18730c28018e064f3ee1ddd3cc32aa506/src/components/HomepageFeatures/index.tsx#L74
> [4] https://github.com/apache/gravitino/tree/main/catalogs
>
> On Tue, Jul 30, 2024 at 10:06 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Manu
> >
> > Thanks for the details !
> > I agree with you. As mentor on Gravitino, I would be more than happy
> > to connect the two podlings.
> >
> > Regards
> > JB
> >
> > On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com>
> > wrote:
> > >
> > > AFAIK, Amoro is a management system with optimization service, catalog
> > > service, etc. It has a built-in catalog but can also work with other
> > > catalogs like Polaris.
> > > I think Polaris is more comparable to Gravitino which entered the
> > incubator
> > > recently. It would be interesting to see how these two communities can
> > > collaborate.
> > >
> > > Regards,
> > > Manu
> > >
> > >
> > > On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > The proposal is more generic: today it's Apache Iceberg, but after
> the
> > > > discussions with the initial community we agreed it could make sense
> > > > to address other use cases.
> > > >
> > > > I don't know Amoro in details, but I am happy to bridge the
> > > > communities to work together.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
> > > > >
> > > > > Hi, JB
> > > > >
> > > > > Thank you for starting this thread; it's great to see an increasing
> > > > number of projects being developed around Iceberg.
> > > > >
> > > > > I have two questions:
> > > > >
> > > > > - The polaris github repo said it's "an open source catalog for
> > Apache
> > > > Iceberg", but the proposal changed into "a catalog for data lakes".
> > Does it
> > > > mean Polaris's scope has been changed?
> > > > > - What's the difference between this project and Amoro:
> > > > https://github.com/apache/amoro? How do these two communities
> > collaborate?
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > wrote:
> > > > > >>
> > > > > >> Hi Dave,
> > > > > >>
> > > > > >> That's a good question. The main reason is because we wanted
> > people
> > > > > >> with Apache experience in the PPMC to mentor the committers and
> > > > > >> contributors heading to PPMC as well.
> > > > > >> Also, the initial committers worked closely with PPMC guidance
> > > > > >> (explaining the ICLA, good practice, etc).
> > > > > >> So, we wanted to have PPMC acting more as mentor (both
> > technically but
> > > > > >> also with their Apache experience) with committers.
> > > > > >
> > > > > > That makes sense. Are any of the proposed PPMC members also ASF
> > Members
> > > > > > and/or potentially future Mentors?
> > > > > >
> > > > > >> If it's problematic, we can start only with the PPMC group and
> > invite
> > > > > >> new committers/PPMC members during incubation period.
> > > > > >
> > > > > > No problem. It will actually provide the Mentors and later the
> IPMC
> > > > > > additional data to see if the PPMC is properly growing the PPMC
> and
> > > > > > Committer base.
> > > > > >
> > > > > > Best,
> > > > > > Dave
> > > > > >
> > > > > >>
> > > > > >> Regards
> > > > > >> JB
> > > > > >>
> > > > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org>
> > wrote:
> > > > > >>>
> > > > > >>> Hi JB,
> > > > > >>>
> > > > > >>> An interesting project that looks pretty mature.
> > > > > >>>
> > > > > >>> I’m curious about the split between Initial PPMC and initial
> > > > Committer. In the usual case a new podling will have all of the
> Initial
> > > > Committers on the PPMC. Can you tell us why this is not the case with
> > > > Polaris?
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Dave
> > > > > >>>
> > > > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > wrote:
> > > > > >>>>
> > > > > >>>> Hi folks,
> > > > > >>>>
> > > > > >>>> We would like to propose a new project to the ASF incubator:
> > > > Polaris.
> > > > > >>>>
> > > > > >>>> Polaris is a catalog for data lakes. It provides new levels of
> > > > choice,
> > > > > >>>> flexibility and control over data, with full enterprise
> > security and
> > > > > >>>> Apache Iceberg interoperability across a multitude of engines
> > and
> > > > > >>>> infrastructure. Polaris builds on standards such as those
> > created by
> > > > > >>>> Apache Iceberg, providing the following benefits for the
> > ecosystem:
> > > > > >>>> * Multi-engine interoperability over a single copy of data,
> > > > > >>>> eliminating the need for moving and copying data across
> > different
> > > > > >>>> engines and catalogs.
> > > > > >>>> * An interoperable security model providing a unified
> > authorization
> > > > > >>>> layer independent from the engines processing analytical
> tables.
> > > > > >>>> * For multi-catalog scenarios, a unified catalog level view of
> > data
> > > > > >>>> across multiple catalogs via catalog notification
> integrations.
> > > > > >>>> * The ability to host Polaris Catalog on the infrastructure of
> > your
> > > > choice.
> > > > > >>>>
> > > > > >>>> Here is the proposal:
> > > > > >>>>
> > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > > > >>>>
> > > > > >>>> Comments and feedback are welcome.
> > > > > >>>>
> > > > > >>>> Thanks!
> > > > > >>>> Regards
> > > > > >>>> JB
> > > > > >>>>
> > > > > >>>>
> > > > ---------------------------------------------------------------------
> > > > > >>>> To unsubscribe, e-mail:
> > general-unsubscribe@incubator.apache.org
> > > > > >>>> For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > ---------------------------------------------------------------------
> > > > > >>> To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > >>> For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > >>>
> > > > > >>
> > > > > >>
> > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > >> For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >
> > > > > --
> > > > > Xuanwo
> > > > >
> > > > > https://xuanwo.io/
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>


-- 
Best

ConradJam

Re: [DISCUSS] PolarisProposal to the incubator

[Jack Ye <ye...@gmail.com>]

> What's the difference between this project and Amoro

Here is my $0.01, please correct me if I am wrong, especially for people
working on Amoro and Gravitino.

I think Apache Amoro is focused more on being a self-contained complete
data lakehouse management and ingestion system. It is a complete solution
with its own connectors in engines like Spark [1], and customized
mixed-format integrations in engines like Trino [2]. Polaris is mostly
focused on the data catalog aspect of a data lakehouse, and offers an open
source vendor-neutral Iceberg catalog with additional governance support.
By integrating with the Iceberg REST catalog interface, the intention is
for it to leverage Iceberg for all the engine integrations to begin with.
Similarly, any table management or ingestion system that works with Iceberg
REST API will be able to be plugged in to directly work with Polaris. So
you could imagine it could be possible for an Iceberg table to be ingested
and managed by Amoro, but cataloged using Polaris.

This does make Polaris more similar to Apache Gravitino. However, I think
the key difference between them is that the emphasis of Gravitino is more
breath-first on aspects like multi-format, multi-catalog, multi-datasource,
different data catalog objects in AI [3], etc. It exposes different sets of
APIs for different purposes, with Iceberg REST API being a part of it for
the Iceberg tables, and other APIs for other data sources [4]. Polaris is
more depth-first on Iceberg at this moment. Our future plan does say that
it could extend to non-Iceberg data lakes, and there could be some overlap
at that time. But even then, there could be different ways to achieve such
support. For example, we could surface Hive Parquet tables as Iceberg
tables, if the Iceberg REST catalog standard can be updated to accommodate
that. There could also be potential collaborations between Polaris and
Gravitino to achieve the goal together, and I am personally pretty excited
about that opportunity.

Best,
Jack Ye

[1] https://amoro.apache.org/docs/latest/spark-configuration/
[2] https://amoro.apache.org/docs/latest/trino/#mixed-format
[3]
https://github.com/apache/gravitino-site/blob/10a967f18730c28018e064f3ee1ddd3cc32aa506/src/components/HomepageFeatures/index.tsx#L74
[4] https://github.com/apache/gravitino/tree/main/catalogs

On Tue, Jul 30, 2024 at 10:06 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi Manu
>
> Thanks for the details !
> I agree with you. As mentor on Gravitino, I would be more than happy
> to connect the two podlings.
>
> Regards
> JB
>
> On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com>
> wrote:
> >
> > AFAIK, Amoro is a management system with optimization service, catalog
> > service, etc. It has a built-in catalog but can also work with other
> > catalogs like Polaris.
> > I think Polaris is more comparable to Gravitino which entered the
> incubator
> > recently. It would be interesting to see how these two communities can
> > collaborate.
> >
> > Regards,
> > Manu
> >
> >
> > On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi
> > >
> > > The proposal is more generic: today it's Apache Iceberg, but after the
> > > discussions with the initial community we agreed it could make sense
> > > to address other use cases.
> > >
> > > I don't know Amoro in details, but I am happy to bridge the
> > > communities to work together.
> > >
> > > Regards
> > > JB
> > >
> > > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
> > > >
> > > > Hi, JB
> > > >
> > > > Thank you for starting this thread; it's great to see an increasing
> > > number of projects being developed around Iceberg.
> > > >
> > > > I have two questions:
> > > >
> > > > - The polaris github repo said it's "an open source catalog for
> Apache
> > > Iceberg", but the proposal changed into "a catalog for data lakes".
> Does it
> > > mean Polaris's scope has been changed?
> > > > - What's the difference between this project and Amoro:
> > > https://github.com/apache/amoro? How do these two communities
> collaborate?
> > > >
> > > >
> > > >
> > > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > wrote:
> > > > >>
> > > > >> Hi Dave,
> > > > >>
> > > > >> That's a good question. The main reason is because we wanted
> people
> > > > >> with Apache experience in the PPMC to mentor the committers and
> > > > >> contributors heading to PPMC as well.
> > > > >> Also, the initial committers worked closely with PPMC guidance
> > > > >> (explaining the ICLA, good practice, etc).
> > > > >> So, we wanted to have PPMC acting more as mentor (both
> technically but
> > > > >> also with their Apache experience) with committers.
> > > > >
> > > > > That makes sense. Are any of the proposed PPMC members also ASF
> Members
> > > > > and/or potentially future Mentors?
> > > > >
> > > > >> If it's problematic, we can start only with the PPMC group and
> invite
> > > > >> new committers/PPMC members during incubation period.
> > > > >
> > > > > No problem. It will actually provide the Mentors and later the IPMC
> > > > > additional data to see if the PPMC is properly growing the PPMC and
> > > > > Committer base.
> > > > >
> > > > > Best,
> > > > > Dave
> > > > >
> > > > >>
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org>
> wrote:
> > > > >>>
> > > > >>> Hi JB,
> > > > >>>
> > > > >>> An interesting project that looks pretty mature.
> > > > >>>
> > > > >>> I’m curious about the split between Initial PPMC and initial
> > > Committer. In the usual case a new podling will have all of the Initial
> > > Committers on the PPMC. Can you tell us why this is not the case with
> > > Polaris?
> > > > >>>
> > > > >>> Best,
> > > > >>> Dave
> > > > >>>
> > > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > wrote:
> > > > >>>>
> > > > >>>> Hi folks,
> > > > >>>>
> > > > >>>> We would like to propose a new project to the ASF incubator:
> > > Polaris.
> > > > >>>>
> > > > >>>> Polaris is a catalog for data lakes. It provides new levels of
> > > choice,
> > > > >>>> flexibility and control over data, with full enterprise
> security and
> > > > >>>> Apache Iceberg interoperability across a multitude of engines
> and
> > > > >>>> infrastructure. Polaris builds on standards such as those
> created by
> > > > >>>> Apache Iceberg, providing the following benefits for the
> ecosystem:
> > > > >>>> * Multi-engine interoperability over a single copy of data,
> > > > >>>> eliminating the need for moving and copying data across
> different
> > > > >>>> engines and catalogs.
> > > > >>>> * An interoperable security model providing a unified
> authorization
> > > > >>>> layer independent from the engines processing analytical tables.
> > > > >>>> * For multi-catalog scenarios, a unified catalog level view of
> data
> > > > >>>> across multiple catalogs via catalog notification integrations.
> > > > >>>> * The ability to host Polaris Catalog on the infrastructure of
> your
> > > choice.
> > > > >>>>
> > > > >>>> Here is the proposal:
> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > > >>>>
> > > > >>>> Comments and feedback are welcome.
> > > > >>>>
> > > > >>>> Thanks!
> > > > >>>> Regards
> > > > >>>> JB
> > > > >>>>
> > > > >>>>
> > > ---------------------------------------------------------------------
> > > > >>>> To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > >>>> For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> ---------------------------------------------------------------------
> > > > >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >>> For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >>>
> > > > >>
> > > > >>
> ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >> For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > > --
> > > > Xuanwo
> > > >
> > > > https://xuanwo.io/
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Manu

Thanks for the details !
I agree with you. As mentor on Gravitino, I would be more than happy
to connect the two podlings.

Regards
JB

On Wed, Jul 31, 2024 at 7:00 AM Manu Zhang <ow...@gmail.com> wrote:
>
> AFAIK, Amoro is a management system with optimization service, catalog
> service, etc. It has a built-in catalog but can also work with other
> catalogs like Polaris.
> I think Polaris is more comparable to Gravitino which entered the incubator
> recently. It would be interesting to see how these two communities can
> collaborate.
>
> Regards,
> Manu
>
>
> On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi
> >
> > The proposal is more generic: today it's Apache Iceberg, but after the
> > discussions with the initial community we agreed it could make sense
> > to address other use cases.
> >
> > I don't know Amoro in details, but I am happy to bridge the
> > communities to work together.
> >
> > Regards
> > JB
> >
> > On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
> > >
> > > Hi, JB
> > >
> > > Thank you for starting this thread; it's great to see an increasing
> > number of projects being developed around Iceberg.
> > >
> > > I have two questions:
> > >
> > > - The polaris github repo said it's "an open source catalog for Apache
> > Iceberg", but the proposal changed into "a catalog for data lakes". Does it
> > mean Polaris's scope has been changed?
> > > - What's the difference between this project and Amoro:
> > https://github.com/apache/amoro? How do these two communities collaborate?
> > >
> > >
> > >
> > > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > > >>
> > > >> Hi Dave,
> > > >>
> > > >> That's a good question. The main reason is because we wanted people
> > > >> with Apache experience in the PPMC to mentor the committers and
> > > >> contributors heading to PPMC as well.
> > > >> Also, the initial committers worked closely with PPMC guidance
> > > >> (explaining the ICLA, good practice, etc).
> > > >> So, we wanted to have PPMC acting more as mentor (both technically but
> > > >> also with their Apache experience) with committers.
> > > >
> > > > That makes sense. Are any of the proposed PPMC members also ASF Members
> > > > and/or potentially future Mentors?
> > > >
> > > >> If it's problematic, we can start only with the PPMC group and invite
> > > >> new committers/PPMC members during incubation period.
> > > >
> > > > No problem. It will actually provide the Mentors and later the IPMC
> > > > additional data to see if the PPMC is properly growing the PPMC and
> > > > Committer base.
> > > >
> > > > Best,
> > > > Dave
> > > >
> > > >>
> > > >> Regards
> > > >> JB
> > > >>
> > > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
> > > >>>
> > > >>> Hi JB,
> > > >>>
> > > >>> An interesting project that looks pretty mature.
> > > >>>
> > > >>> I’m curious about the split between Initial PPMC and initial
> > Committer. In the usual case a new podling will have all of the Initial
> > Committers on the PPMC. Can you tell us why this is not the case with
> > Polaris?
> > > >>>
> > > >>> Best,
> > > >>> Dave
> > > >>>
> > > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > > >>>>
> > > >>>> Hi folks,
> > > >>>>
> > > >>>> We would like to propose a new project to the ASF incubator:
> > Polaris.
> > > >>>>
> > > >>>> Polaris is a catalog for data lakes. It provides new levels of
> > choice,
> > > >>>> flexibility and control over data, with full enterprise security and
> > > >>>> Apache Iceberg interoperability across a multitude of engines and
> > > >>>> infrastructure. Polaris builds on standards such as those created by
> > > >>>> Apache Iceberg, providing the following benefits for the ecosystem:
> > > >>>> * Multi-engine interoperability over a single copy of data,
> > > >>>> eliminating the need for moving and copying data across different
> > > >>>> engines and catalogs.
> > > >>>> * An interoperable security model providing a unified authorization
> > > >>>> layer independent from the engines processing analytical tables.
> > > >>>> * For multi-catalog scenarios, a unified catalog level view of data
> > > >>>> across multiple catalogs via catalog notification integrations.
> > > >>>> * The ability to host Polaris Catalog on the infrastructure of your
> > choice.
> > > >>>>
> > > >>>> Here is the proposal:
> > > >>>>
> > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > > >>>>
> > > >>>> Comments and feedback are welcome.
> > > >>>>
> > > >>>> Thanks!
> > > >>>> Regards
> > > >>>> JB
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >>>> For additional commands, e-mail: general-help@incubator.apache.org
> > > >>>>
> > > >>>
> > > >>>
> > > >>> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >>> For additional commands, e-mail: general-help@incubator.apache.org
> > > >>>
> > > >>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >> For additional commands, e-mail: general-help@incubator.apache.org
> > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > > --
> > > Xuanwo
> > >
> > > https://xuanwo.io/
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Manu Zhang <ow...@gmail.com>]

AFAIK, Amoro is a management system with optimization service, catalog
service, etc. It has a built-in catalog but can also work with other
catalogs like Polaris.
I think Polaris is more comparable to Gravitino which entered the incubator
recently. It would be interesting to see how these two communities can
collaborate.

Regards,
Manu


On Wed, Jul 31, 2024 at 12:36 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi
>
> The proposal is more generic: today it's Apache Iceberg, but after the
> discussions with the initial community we agreed it could make sense
> to address other use cases.
>
> I don't know Amoro in details, but I am happy to bridge the
> communities to work together.
>
> Regards
> JB
>
> On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
> >
> > Hi, JB
> >
> > Thank you for starting this thread; it's great to see an increasing
> number of projects being developed around Iceberg.
> >
> > I have two questions:
> >
> > - The polaris github repo said it's "an open source catalog for Apache
> Iceberg", but the proposal changed into "a catalog for data lakes". Does it
> mean Polaris's scope has been changed?
> > - What's the difference between this project and Amoro:
> https://github.com/apache/amoro? How do these two communities collaborate?
> >
> >
> >
> > On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> > >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> That's a good question. The main reason is because we wanted people
> > >> with Apache experience in the PPMC to mentor the committers and
> > >> contributors heading to PPMC as well.
> > >> Also, the initial committers worked closely with PPMC guidance
> > >> (explaining the ICLA, good practice, etc).
> > >> So, we wanted to have PPMC acting more as mentor (both technically but
> > >> also with their Apache experience) with committers.
> > >
> > > That makes sense. Are any of the proposed PPMC members also ASF Members
> > > and/or potentially future Mentors?
> > >
> > >> If it's problematic, we can start only with the PPMC group and invite
> > >> new committers/PPMC members during incubation period.
> > >
> > > No problem. It will actually provide the Mentors and later the IPMC
> > > additional data to see if the PPMC is properly growing the PPMC and
> > > Committer base.
> > >
> > > Best,
> > > Dave
> > >
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
> > >>>
> > >>> Hi JB,
> > >>>
> > >>> An interesting project that looks pretty mature.
> > >>>
> > >>> I’m curious about the split between Initial PPMC and initial
> Committer. In the usual case a new podling will have all of the Initial
> Committers on the PPMC. Can you tell us why this is not the case with
> Polaris?
> > >>>
> > >>> Best,
> > >>> Dave
> > >>>
> > >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > >>>>
> > >>>> Hi folks,
> > >>>>
> > >>>> We would like to propose a new project to the ASF incubator:
> Polaris.
> > >>>>
> > >>>> Polaris is a catalog for data lakes. It provides new levels of
> choice,
> > >>>> flexibility and control over data, with full enterprise security and
> > >>>> Apache Iceberg interoperability across a multitude of engines and
> > >>>> infrastructure. Polaris builds on standards such as those created by
> > >>>> Apache Iceberg, providing the following benefits for the ecosystem:
> > >>>> * Multi-engine interoperability over a single copy of data,
> > >>>> eliminating the need for moving and copying data across different
> > >>>> engines and catalogs.
> > >>>> * An interoperable security model providing a unified authorization
> > >>>> layer independent from the engines processing analytical tables.
> > >>>> * For multi-catalog scenarios, a unified catalog level view of data
> > >>>> across multiple catalogs via catalog notification integrations.
> > >>>> * The ability to host Polaris Catalog on the infrastructure of your
> choice.
> > >>>>
> > >>>> Here is the proposal:
> > >>>>
> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > >>>>
> > >>>> Comments and feedback are welcome.
> > >>>>
> > >>>> Thanks!
> > >>>> Regards
> > >>>> JB
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >>>> For additional commands, e-mail: general-help@incubator.apache.org
> > >>>>
> > >>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >>> For additional commands, e-mail: general-help@incubator.apache.org
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: general-help@incubator.apache.org
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> >
> > --
> > Xuanwo
> >
> > https://xuanwo.io/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi

The proposal is more generic: today it's Apache Iceberg, but after the
discussions with the initial community we agreed it could make sense
to address other use cases.

I don't know Amoro in details, but I am happy to bridge the
communities to work together.

Regards
JB

On Wed, Jul 31, 2024 at 5:16 AM Xuanwo <xu...@apache.org> wrote:
>
> Hi, JB
>
> Thank you for starting this thread; it's great to see an increasing number of projects being developed around Iceberg.
>
> I have two questions:
>
> - The polaris github repo said it's "an open source catalog for Apache Iceberg", but the proposal changed into "a catalog for data lakes". Does it mean Polaris's scope has been changed?
> - What's the difference between this project and Amoro: https://github.com/apache/amoro? How do these two communities collaborate?
>
>
>
> On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
> >> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>
> >> Hi Dave,
> >>
> >> That's a good question. The main reason is because we wanted people
> >> with Apache experience in the PPMC to mentor the committers and
> >> contributors heading to PPMC as well.
> >> Also, the initial committers worked closely with PPMC guidance
> >> (explaining the ICLA, good practice, etc).
> >> So, we wanted to have PPMC acting more as mentor (both technically but
> >> also with their Apache experience) with committers.
> >
> > That makes sense. Are any of the proposed PPMC members also ASF Members
> > and/or potentially future Mentors?
> >
> >> If it's problematic, we can start only with the PPMC group and invite
> >> new committers/PPMC members during incubation period.
> >
> > No problem. It will actually provide the Mentors and later the IPMC
> > additional data to see if the PPMC is properly growing the PPMC and
> > Committer base.
> >
> > Best,
> > Dave
> >
> >>
> >> Regards
> >> JB
> >>
> >> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
> >>>
> >>> Hi JB,
> >>>
> >>> An interesting project that looks pretty mature.
> >>>
> >>> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
> >>>
> >>> Best,
> >>> Dave
> >>>
> >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>>>
> >>>> Hi folks,
> >>>>
> >>>> We would like to propose a new project to the ASF incubator: Polaris.
> >>>>
> >>>> Polaris is a catalog for data lakes. It provides new levels of choice,
> >>>> flexibility and control over data, with full enterprise security and
> >>>> Apache Iceberg interoperability across a multitude of engines and
> >>>> infrastructure. Polaris builds on standards such as those created by
> >>>> Apache Iceberg, providing the following benefits for the ecosystem:
> >>>> * Multi-engine interoperability over a single copy of data,
> >>>> eliminating the need for moving and copying data across different
> >>>> engines and catalogs.
> >>>> * An interoperable security model providing a unified authorization
> >>>> layer independent from the engines processing analytical tables.
> >>>> * For multi-catalog scenarios, a unified catalog level view of data
> >>>> across multiple catalogs via catalog notification integrations.
> >>>> * The ability to host Polaris Catalog on the infrastructure of your choice.
> >>>>
> >>>> Here is the proposal:
> >>>> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >>>>
> >>>> Comments and feedback are welcome.
> >>>>
> >>>> Thanks!
> >>>> Regards
> >>>> JB
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
>
> --
> Xuanwo
>
> https://xuanwo.io/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Xuanwo <xu...@apache.org>]

Hi, JB

Thank you for starting this thread; it's great to see an increasing number of projects being developed around Iceberg.

I have two questions:

- The polaris github repo said it's "an open source catalog for Apache Iceberg", but the proposal changed into "a catalog for data lakes". Does it mean Polaris's scope has been changed?
- What's the difference between this project and Amoro: https://github.com/apache/amoro? How do these two communities collaborate?



On Wed, Jul 31, 2024, at 04:19, Dave Fisher wrote:
>> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> 
>> Hi Dave,
>> 
>> That's a good question. The main reason is because we wanted people
>> with Apache experience in the PPMC to mentor the committers and
>> contributors heading to PPMC as well.
>> Also, the initial committers worked closely with PPMC guidance
>> (explaining the ICLA, good practice, etc).
>> So, we wanted to have PPMC acting more as mentor (both technically but
>> also with their Apache experience) with committers.
>
> That makes sense. Are any of the proposed PPMC members also ASF Members 
> and/or potentially future Mentors?
>
>> If it's problematic, we can start only with the PPMC group and invite
>> new committers/PPMC members during incubation period.
>
> No problem. It will actually provide the Mentors and later the IPMC 
> additional data to see if the PPMC is properly growing the PPMC and 
> Committer base.
>
> Best,
> Dave
>
>> 
>> Regards
>> JB
>> 
>> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
>>> 
>>> Hi JB,
>>> 
>>> An interesting project that looks pretty mature.
>>> 
>>> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
>>> 
>>> Best,
>>> Dave
>>> 
>>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>> 
>>>> Hi folks,
>>>> 
>>>> We would like to propose a new project to the ASF incubator: Polaris.
>>>> 
>>>> Polaris is a catalog for data lakes. It provides new levels of choice,
>>>> flexibility and control over data, with full enterprise security and
>>>> Apache Iceberg interoperability across a multitude of engines and
>>>> infrastructure. Polaris builds on standards such as those created by
>>>> Apache Iceberg, providing the following benefits for the ecosystem:
>>>> * Multi-engine interoperability over a single copy of data,
>>>> eliminating the need for moving and copying data across different
>>>> engines and catalogs.
>>>> * An interoperable security model providing a unified authorization
>>>> layer independent from the engines processing analytical tables.
>>>> * For multi-catalog scenarios, a unified catalog level view of data
>>>> across multiple catalogs via catalog notification integrations.
>>>> * The ability to host Polaris Catalog on the infrastructure of your choice.
>>>> 
>>>> Here is the proposal:
>>>> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
>>>> 
>>>> Comments and feedback are welcome.
>>>> 
>>>> Thanks!
>>>> Regards
>>>> JB
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org

-- 
Xuanwo

https://xuanwo.io/

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Dave

Most of the listed PPMC members are not ASF member, but they could be
interested to mentors.
So, even if some of them don't have code-contributions on Polaris,
they are helping a lot in the roadmap, use cases, design, etc. They
would like to help in terms of governance and community building.
That's why I think it's a good starting list.

Thoughts ?

Regards
JB

On Tue, Jul 30, 2024 at 10:19 PM Dave Fisher <wa...@apache.org> wrote:
>
>
>
> > On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi Dave,
> >
> > That's a good question. The main reason is because we wanted people
> > with Apache experience in the PPMC to mentor the committers and
> > contributors heading to PPMC as well.
> > Also, the initial committers worked closely with PPMC guidance
> > (explaining the ICLA, good practice, etc).
> > So, we wanted to have PPMC acting more as mentor (both technically but
> > also with their Apache experience) with committers.
>
> That makes sense. Are any of the proposed PPMC members also ASF Members and/or potentially future Mentors?
>
> > If it's problematic, we can start only with the PPMC group and invite
> > new committers/PPMC members during incubation period.
>
> No problem. It will actually provide the Mentors and later the IPMC additional data to see if the PPMC is properly growing the PPMC and Committer base.
>
> Best,
> Dave
>
> >
> > Regards
> > JB
> >
> > On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
> >>
> >> Hi JB,
> >>
> >> An interesting project that looks pretty mature.
> >>
> >> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
> >>
> >> Best,
> >> Dave
> >>
> >>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>>
> >>> Hi folks,
> >>>
> >>> We would like to propose a new project to the ASF incubator: Polaris.
> >>>
> >>> Polaris is a catalog for data lakes. It provides new levels of choice,
> >>> flexibility and control over data, with full enterprise security and
> >>> Apache Iceberg interoperability across a multitude of engines and
> >>> infrastructure. Polaris builds on standards such as those created by
> >>> Apache Iceberg, providing the following benefits for the ecosystem:
> >>> * Multi-engine interoperability over a single copy of data,
> >>> eliminating the need for moving and copying data across different
> >>> engines and catalogs.
> >>> * An interoperable security model providing a unified authorization
> >>> layer independent from the engines processing analytical tables.
> >>> * For multi-catalog scenarios, a unified catalog level view of data
> >>> across multiple catalogs via catalog notification integrations.
> >>> * The ability to host Polaris Catalog on the infrastructure of your choice.
> >>>
> >>> Here is the proposal:
> >>> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >>>
> >>> Comments and feedback are welcome.
> >>>
> >>> Thanks!
> >>> Regards
> >>> JB
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Dave Fisher <wa...@apache.org>]

> On Jul 30, 2024, at 11:34 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> 
> Hi Dave,
> 
> That's a good question. The main reason is because we wanted people
> with Apache experience in the PPMC to mentor the committers and
> contributors heading to PPMC as well.
> Also, the initial committers worked closely with PPMC guidance
> (explaining the ICLA, good practice, etc).
> So, we wanted to have PPMC acting more as mentor (both technically but
> also with their Apache experience) with committers.

That makes sense. Are any of the proposed PPMC members also ASF Members and/or potentially future Mentors?

> If it's problematic, we can start only with the PPMC group and invite
> new committers/PPMC members during incubation period.

No problem. It will actually provide the Mentors and later the IPMC additional data to see if the PPMC is properly growing the PPMC and Committer base.

Best,
Dave

> 
> Regards
> JB
> 
> On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
>> 
>> Hi JB,
>> 
>> An interesting project that looks pretty mature.
>> 
>> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
>> 
>> Best,
>> Dave
>> 
>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>> 
>>> Hi folks,
>>> 
>>> We would like to propose a new project to the ASF incubator: Polaris.
>>> 
>>> Polaris is a catalog for data lakes. It provides new levels of choice,
>>> flexibility and control over data, with full enterprise security and
>>> Apache Iceberg interoperability across a multitude of engines and
>>> infrastructure. Polaris builds on standards such as those created by
>>> Apache Iceberg, providing the following benefits for the ecosystem:
>>> * Multi-engine interoperability over a single copy of data,
>>> eliminating the need for moving and copying data across different
>>> engines and catalogs.
>>> * An interoperable security model providing a unified authorization
>>> layer independent from the engines processing analytical tables.
>>> * For multi-catalog scenarios, a unified catalog level view of data
>>> across multiple catalogs via catalog notification integrations.
>>> * The ability to host Polaris Catalog on the infrastructure of your choice.
>>> 
>>> Here is the proposal:
>>> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
>>> 
>>> Comments and feedback are welcome.
>>> 
>>> Thanks!
>>> Regards
>>> JB
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Dave,

That's a good question. The main reason is because we wanted people
with Apache experience in the PPMC to mentor the committers and
contributors heading to PPMC as well.
Also, the initial committers worked closely with PPMC guidance
(explaining the ICLA, good practice, etc).
So, we wanted to have PPMC acting more as mentor (both technically but
also with their Apache experience) with committers.

If it's problematic, we can start only with the PPMC group and invite
new committers/PPMC members during incubation period.

Regards
JB

On Tue, Jul 30, 2024 at 8:19 PM Dave Fisher <wa...@apache.org> wrote:
>
> Hi JB,
>
> An interesting project that looks pretty mature.
>
> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
>
> Best,
> Dave
>
> > On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi folks,
> >
> > We would like to propose a new project to the ASF incubator: Polaris.
> >
> > Polaris is a catalog for data lakes. It provides new levels of choice,
> > flexibility and control over data, with full enterprise security and
> > Apache Iceberg interoperability across a multitude of engines and
> > infrastructure. Polaris builds on standards such as those created by
> > Apache Iceberg, providing the following benefits for the ecosystem:
> > * Multi-engine interoperability over a single copy of data,
> > eliminating the need for moving and copying data across different
> > engines and catalogs.
> > * An interoperable security model providing a unified authorization
> > layer independent from the engines processing analytical tables.
> > * For multi-catalog scenarios, a unified catalog level view of data
> > across multiple catalogs via catalog notification integrations.
> > * The ability to host Polaris Catalog on the infrastructure of your choice.
> >
> > Here is the proposal:
> > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >
> > Comments and feedback are welcome.
> >
> > Thanks!
> > Regards
> > JB
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Alex,

Correct me if I'm wrong (even if I know Wayang :)), Wayang is more on
the data processing (with features like zero-copy, etc).
Polaris is a catalog, basically providing the metadata for a
table/view name and dealing with access to table/view.
So Wayang could work with Polaris to retrieve metadata of tables he
wants to process.

Regards
JB

On Tue, Jul 30, 2024 at 8:52 PM Alexander Alten-Lorenz <al...@scalytics.io> wrote:
>
> Hey JB,
>
> What’s the difference to Wayang?
>
> —a.
>
> Sent from my mobile, please excuse typos
>
> > On Jul 30, 2024, at 8:40 PM, Ayush Saxena <ay...@gmail.com> wrote:
> >
> > Hi JB,
> > Apart from the questions from Dave, I have an additional question
> > about the initial committers & PPMC members, Did those folks
> > contribute anything to the project so far? I checked the repository
> > [1], most of them here have some trivial 1-2 contributions. & then the
> > initial PR [2], it does mention folks from Snowflake but where did the
> > Dremio folks & couple of other non Snowflake folks mentioned in the
> > initial committer & PPMC list contributed?
> >
> > I started checking the PPMC list, I found only one commit for the
> > first member mentioned in the list [3], which is a typo :-)
> >
> > -Ayush
> >
> > [1] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commits?author%3Danoopj&source=gmail-imap&ust=1722969606000000&usg=AOvVaw0BUI4BAqk0thBM6MevcPi_
> > [2] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commit/88fbb9ae77548257c885ab194ef6ef43a66ffbe9&source=gmail-imap&ust=1722969606000000&usg=AOvVaw1qX5wUW17ImJBTQJM79FHl
> > [3] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commits?author%3Danoopj&source=gmail-imap&ust=1722969606000000&usg=AOvVaw0BUI4BAqk0thBM6MevcPi_
> >
> >> On Tue, 30 Jul 2024 at 23:49, Dave Fisher <wa...@apache.org> wrote:
> >>
> >> Hi JB,
> >>
> >> An interesting project that looks pretty mature.
> >>
> >> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
> >>
> >> Best,
> >> Dave
> >>
> >>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >>>
> >>> Hi folks,
> >>>
> >>> We would like to propose a new project to the ASF incubator: Polaris.
> >>>
> >>> Polaris is a catalog for data lakes. It provides new levels of choice,
> >>> flexibility and control over data, with full enterprise security and
> >>> Apache Iceberg interoperability across a multitude of engines and
> >>> infrastructure. Polaris builds on standards such as those created by
> >>> Apache Iceberg, providing the following benefits for the ecosystem:
> >>> * Multi-engine interoperability over a single copy of data,
> >>> eliminating the need for moving and copying data across different
> >>> engines and catalogs.
> >>> * An interoperable security model providing a unified authorization
> >>> layer independent from the engines processing analytical tables.
> >>> * For multi-catalog scenarios, a unified catalog level view of data
> >>> across multiple catalogs via catalog notification integrations.
> >>> * The ability to host Polaris Catalog on the infrastructure of your choice.
> >>>
> >>> Here is the proposal:
> >>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal&source=gmail-imap&ust=1722969606000000&usg=AOvVaw2E4PeaUjK6OLK5VYmrPgiv
> >>>
> >>> Comments and feedback are welcome.
> >>>
> >>> Thanks!
> >>> Regards
> >>> JB
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> --
> *Why Scalytics:*
> Break data silos. Put your data to work. All without ever
> sharing your data. That's the Scalytics difference.
>
>
> --
> 3401 N. MIAMI AVE.
> STE 230
> 33127 Miami, Florida
> United States
> www.scalytics.io
> <http://www.scalytics.io>
>
> --  Please consider the environment before
> printing this email --
>
> Disclaimer:
> The content of this message is
> confidential. If you have received it by mistake, please inform us by an
> email reply and then delete the message. It is forbidden to copy, forward,
> or in any way reveal the contents of this message to anyone. The integrity
> and security of this email cannot be guaranteed over the Internet.
> Therefore, the sender will not be held liable for any damage caused by the
> message.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Alexander Alten-Lorenz <al...@scalytics.io>]

Hey JB,

What’s the difference to Wayang?

—a. 

Sent from my mobile, please excuse typos 

> On Jul 30, 2024, at 8:40 PM, Ayush Saxena <ay...@gmail.com> wrote:
> 
> Hi JB,
> Apart from the questions from Dave, I have an additional question
> about the initial committers & PPMC members, Did those folks
> contribute anything to the project so far? I checked the repository
> [1], most of them here have some trivial 1-2 contributions. & then the
> initial PR [2], it does mention folks from Snowflake but where did the
> Dremio folks & couple of other non Snowflake folks mentioned in the
> initial committer & PPMC list contributed?
> 
> I started checking the PPMC list, I found only one commit for the
> first member mentioned in the list [3], which is a typo :-)
> 
> -Ayush
> 
> [1] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commits?author%3Danoopj&source=gmail-imap&ust=1722969606000000&usg=AOvVaw0BUI4BAqk0thBM6MevcPi_
> [2] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commit/88fbb9ae77548257c885ab194ef6ef43a66ffbe9&source=gmail-imap&ust=1722969606000000&usg=AOvVaw1qX5wUW17ImJBTQJM79FHl
> [3] https://www.google.com/url?q=https://github.com/polaris-catalog/polaris/commits?author%3Danoopj&source=gmail-imap&ust=1722969606000000&usg=AOvVaw0BUI4BAqk0thBM6MevcPi_
> 
>> On Tue, 30 Jul 2024 at 23:49, Dave Fisher <wa...@apache.org> wrote:
>> 
>> Hi JB,
>> 
>> An interesting project that looks pretty mature.
>> 
>> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
>> 
>> Best,
>> Dave
>> 
>>>> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>> 
>>> Hi folks,
>>> 
>>> We would like to propose a new project to the ASF incubator: Polaris.
>>> 
>>> Polaris is a catalog for data lakes. It provides new levels of choice,
>>> flexibility and control over data, with full enterprise security and
>>> Apache Iceberg interoperability across a multitude of engines and
>>> infrastructure. Polaris builds on standards such as those created by
>>> Apache Iceberg, providing the following benefits for the ecosystem:
>>> * Multi-engine interoperability over a single copy of data,
>>> eliminating the need for moving and copying data across different
>>> engines and catalogs.
>>> * An interoperable security model providing a unified authorization
>>> layer independent from the engines processing analytical tables.
>>> * For multi-catalog scenarios, a unified catalog level view of data
>>> across multiple catalogs via catalog notification integrations.
>>> * The ability to host Polaris Catalog on the infrastructure of your choice.
>>> 
>>> Here is the proposal:
>>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal&source=gmail-imap&ust=1722969606000000&usg=AOvVaw2E4PeaUjK6OLK5VYmrPgiv
>>> 
>>> Comments and feedback are welcome.
>>> 
>>> Thanks!
>>> Regards
>>> JB
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 

-- 
*Why Scalytics:* 
Break data silos. Put your data to work. All without ever 
sharing your data. That's the Scalytics difference.


--
3401 N. MIAMI AVE. 
STE 230
33127 Miami, Florida 
United States
www.scalytics.io 
<http://www.scalytics.io>

--  Please consider the environment before 
printing this email --

Disclaimer:
The content of this message is 
confidential. If you have received it by mistake, please inform us by an 
email reply and then delete the message. It is forbidden to copy, forward, 
or in any way reveal the contents of this message to anyone. The integrity 
and security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by the 
message.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi,

Good question too :)
The project codebase went public, and we squashed a bunch of commits
and changed the history, and we lost some commits.
That said, you are right: if everyone contributed on the ideas,
visions, draft etc, not everyone on the lists (both PPMC members and
committers) contributed code. However all are involved in the project
(discussions, objectives, goals, design).

Regards
JB

On Tue, Jul 30, 2024 at 8:39 PM Ayush Saxena <ay...@gmail.com> wrote:
>
> Hi JB,
> Apart from the questions from Dave, I have an additional question
> about the initial committers & PPMC members, Did those folks
> contribute anything to the project so far? I checked the repository
> [1], most of them here have some trivial 1-2 contributions. & then the
> initial PR [2], it does mention folks from Snowflake but where did the
> Dremio folks & couple of other non Snowflake folks mentioned in the
> initial committer & PPMC list contributed?
>
> I started checking the PPMC list, I found only one commit for the
> first member mentioned in the list [3], which is a typo :-)
>
> -Ayush
>
> [1] https://github.com/polaris-catalog/polaris/commits?author=anoopj
> [2] https://github.com/polaris-catalog/polaris/commit/88fbb9ae77548257c885ab194ef6ef43a66ffbe9
> [3] https://github.com/polaris-catalog/polaris/commits?author=anoopj
>
> On Tue, 30 Jul 2024 at 23:49, Dave Fisher <wa...@apache.org> wrote:
> >
> > Hi JB,
> >
> > An interesting project that looks pretty mature.
> >
> > I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
> >
> > Best,
> > Dave
> >
> > > On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> > >
> > > Hi folks,
> > >
> > > We would like to propose a new project to the ASF incubator: Polaris.
> > >
> > > Polaris is a catalog for data lakes. It provides new levels of choice,
> > > flexibility and control over data, with full enterprise security and
> > > Apache Iceberg interoperability across a multitude of engines and
> > > infrastructure. Polaris builds on standards such as those created by
> > > Apache Iceberg, providing the following benefits for the ecosystem:
> > > * Multi-engine interoperability over a single copy of data,
> > > eliminating the need for moving and copying data across different
> > > engines and catalogs.
> > > * An interoperable security model providing a unified authorization
> > > layer independent from the engines processing analytical tables.
> > > * For multi-catalog scenarios, a unified catalog level view of data
> > > across multiple catalogs via catalog notification integrations.
> > > * The ability to host Polaris Catalog on the infrastructure of your choice.
> > >
> > > Here is the proposal:
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> > >
> > > Comments and feedback are welcome.
> > >
> > > Thanks!
> > > Regards
> > > JB
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Ayush Saxena <ay...@gmail.com>]

Hi JB,
Apart from the questions from Dave, I have an additional question
about the initial committers & PPMC members, Did those folks
contribute anything to the project so far? I checked the repository
[1], most of them here have some trivial 1-2 contributions. & then the
initial PR [2], it does mention folks from Snowflake but where did the
Dremio folks & couple of other non Snowflake folks mentioned in the
initial committer & PPMC list contributed?

I started checking the PPMC list, I found only one commit for the
first member mentioned in the list [3], which is a typo :-)

-Ayush

[1] https://github.com/polaris-catalog/polaris/commits?author=anoopj
[2] https://github.com/polaris-catalog/polaris/commit/88fbb9ae77548257c885ab194ef6ef43a66ffbe9
[3] https://github.com/polaris-catalog/polaris/commits?author=anoopj

On Tue, 30 Jul 2024 at 23:49, Dave Fisher <wa...@apache.org> wrote:
>
> Hi JB,
>
> An interesting project that looks pretty mature.
>
> I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?
>
> Best,
> Dave
>
> > On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi folks,
> >
> > We would like to propose a new project to the ASF incubator: Polaris.
> >
> > Polaris is a catalog for data lakes. It provides new levels of choice,
> > flexibility and control over data, with full enterprise security and
> > Apache Iceberg interoperability across a multitude of engines and
> > infrastructure. Polaris builds on standards such as those created by
> > Apache Iceberg, providing the following benefits for the ecosystem:
> > * Multi-engine interoperability over a single copy of data,
> > eliminating the need for moving and copying data across different
> > engines and catalogs.
> > * An interoperable security model providing a unified authorization
> > layer independent from the engines processing analytical tables.
> > * For multi-catalog scenarios, a unified catalog level view of data
> > across multiple catalogs via catalog notification integrations.
> > * The ability to host Polaris Catalog on the infrastructure of your choice.
> >
> > Here is the proposal:
> > https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> >
> > Comments and feedback are welcome.
> >
> > Thanks!
> > Regards
> > JB
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Dave Fisher <wa...@apache.org>]

Hi JB,

An interesting project that looks pretty mature.

I’m curious about the split between Initial PPMC and initial Committer. In the usual case a new podling will have all of the Initial Committers on the PPMC. Can you tell us why this is not the case with Polaris?

Best,
Dave

> On Jul 30, 2024, at 10:33 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> 
> Hi folks,
> 
> We would like to propose a new project to the ASF incubator: Polaris.
> 
> Polaris is a catalog for data lakes. It provides new levels of choice,
> flexibility and control over data, with full enterprise security and
> Apache Iceberg interoperability across a multitude of engines and
> infrastructure. Polaris builds on standards such as those created by
> Apache Iceberg, providing the following benefits for the ecosystem:
> * Multi-engine interoperability over a single copy of data,
> eliminating the need for moving and copying data across different
> engines and catalogs.
> * An interoperable security model providing a unified authorization
> layer independent from the engines processing analytical tables.
> * For multi-catalog scenarios, a unified catalog level view of data
> across multiple catalogs via catalog notification integrations.
> * The ability to host Polaris Catalog on the infrastructure of your choice.
> 
> Here is the proposal:
> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> 
> Comments and feedback are welcome.
> 
> Thanks!
> Regards
> JB
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] PolarisProposal to the incubator

[Jeff Jirsa <jj...@apache.org>]

This is an exciting proposal and I'm really happy to see it, would love to see it incubate and eventually graduate!

- Jef

On 2024/07/30 17:33:57 Jean-Baptiste Onofré wrote:
> Hi folks,
> 
> We would like to propose a new project to the ASF incubator: Polaris.
> 
> Polaris is a catalog for data lakes. It provides new levels of choice,
> flexibility and control over data, with full enterprise security and
> Apache Iceberg interoperability across a multitude of engines and
> infrastructure. Polaris builds on standards such as those created by
> Apache Iceberg, providing the following benefits for the ecosystem:
> * Multi-engine interoperability over a single copy of data,
> eliminating the need for moving and copying data across different
> engines and catalogs.
> * An interoperable security model providing a unified authorization
> layer independent from the engines processing analytical tables.
> * For multi-catalog scenarios, a unified catalog level view of data
> across multiple catalogs via catalog notification integrations.
> * The ability to host Polaris Catalog on the infrastructure of your choice.
> 
> Here is the proposal:
> https://cwiki.apache.org/confluence/display/INCUBATOR/PolarisProposal
> 
> Comments and feedback are welcome.
> 
> Thanks!
> Regards
> JB
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org