You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2024/10/18 09:17:13 UTC
Heading to Polaris 0.9 release and beyond
Hi folks,
First of all, thanks everyone for the first community meeting
yesterday. It was a great meeting, very interesting discussions.
You can find the meeting notes on the Polaris website
(https://polaris.apache.org/community/meetings/):
https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
As discussed during the meeting, here's the release proposal:
- the first Polaris release will be 0.9. I'm doing a new pass on the
preparation steps (due diligence on
NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
we are mid October, I propose to actually release 0.9 at the beginning
of November.
This release is to verify our distribution with both the Polaris
community and also the Apache Incubator (IPMC).
- After 0.9 release, I propose to move forward to 1.0 release (I think
it would be great to have 1.0 release before end of the year)
- I also think we should use the "fast" major version as soon as we
introduce any breaking change. I don't see a problem with having
Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
Regards
JB
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
The release process has started (I created the 0.9.x branch and bumped
main to 1.0.0-incubating-SNAPSHOT).
However, I've just realised that I forgot to create the Polaris
stagingProfile on Nexus (link to org.apache.polaris groupId) to upload
the artifacts. I'm fixing it now.
Sorry for the delay due to that.
Regards
JB
On Fri, Nov 15, 2024 at 3:51 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi folks,
>
> As said yesterday during the community meeting, I'm moving on the
> 0.9.0-incubating release today.
>
> First, I realized that the version was not correct: as ASF incubator
> podling, Polaris has to use -incubating in the version.
> I created this PR: https://github.com/apache/polaris/pull/451
>
> I will merge as soon as I have a review ;)
>
> I'm documenting all steps I'm using for the release (preparation
> steps, git commands, etc). The purpose is to document the release
> process on the website and I will also create a script to
> automatically does this.
>
> I will keep you posted soon.
>
> Regards
> JB
>
> On Tue, Nov 12, 2024 at 2:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi folks,
> >
> > I created KEYS file
> > (https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS) to
> > start the 0.9.0 release preparation.
> > I will also create a PR to "document" the current release process.
> >
> > I will keep you posted about the 0.9.0 release progress (stay tuned
> > for the vote email :)).
> >
> > Thanks !
> > Regards
> >
> > JB
> >
> >
> > On Fri, Oct 18, 2024 at 11:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> > >
> > > Hi folks,
> > >
> > > First of all, thanks everyone for the first community meeting
> > > yesterday. It was a great meeting, very interesting discussions.
> > > You can find the meeting notes on the Polaris website
> > > (https://polaris.apache.org/community/meetings/):
> > >
> > > https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > >
> > > As discussed during the meeting, here's the release proposal:
> > > - the first Polaris release will be 0.9. I'm doing a new pass on the
> > > preparation steps (due diligence on
> > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> > > we are mid October, I propose to actually release 0.9 at the beginning
> > > of November.
> > > This release is to verify our distribution with both the Polaris
> > > community and also the Apache Incubator (IPMC).
> > > - After 0.9 release, I propose to move forward to 1.0 release (I think
> > > it would be great to have 1.0 release before end of the year)
> > > - I also think we should use the "fast" major version as soon as we
> > > introduce any breaking change. I don't see a problem with having
> > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > >
> > > Regards
> > > JB
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi folks,
As said yesterday during the community meeting, I'm moving on the
0.9.0-incubating release today.
First, I realized that the version was not correct: as ASF incubator
podling, Polaris has to use -incubating in the version.
I created this PR: https://github.com/apache/polaris/pull/451
I will merge as soon as I have a review ;)
I'm documenting all steps I'm using for the release (preparation
steps, git commands, etc). The purpose is to document the release
process on the website and I will also create a script to
automatically does this.
I will keep you posted soon.
Regards
JB
On Tue, Nov 12, 2024 at 2:34 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi folks,
>
> I created KEYS file
> (https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS) to
> start the 0.9.0 release preparation.
> I will also create a PR to "document" the current release process.
>
> I will keep you posted about the 0.9.0 release progress (stay tuned
> for the vote email :)).
>
> Thanks !
> Regards
>
> JB
>
>
> On Fri, Oct 18, 2024 at 11:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi folks,
> >
> > First of all, thanks everyone for the first community meeting
> > yesterday. It was a great meeting, very interesting discussions.
> > You can find the meeting notes on the Polaris website
> > (https://polaris.apache.org/community/meetings/):
> >
> > https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> >
> > As discussed during the meeting, here's the release proposal:
> > - the first Polaris release will be 0.9. I'm doing a new pass on the
> > preparation steps (due diligence on
> > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> > we are mid October, I propose to actually release 0.9 at the beginning
> > of November.
> > This release is to verify our distribution with both the Polaris
> > community and also the Apache Incubator (IPMC).
> > - After 0.9 release, I propose to move forward to 1.0 release (I think
> > it would be great to have 1.0 release before end of the year)
> > - I also think we should use the "fast" major version as soon as we
> > introduce any breaking change. I don't see a problem with having
> > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> >
> > Regards
> > JB
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi folks,
I created KEYS file
(https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS) to
start the 0.9.0 release preparation.
I will also create a PR to "document" the current release process.
I will keep you posted about the 0.9.0 release progress (stay tuned
for the vote email :)).
Thanks !
Regards
JB
On Fri, Oct 18, 2024 at 11:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi folks,
>
> First of all, thanks everyone for the first community meeting
> yesterday. It was a great meeting, very interesting discussions.
> You can find the meeting notes on the Polaris website
> (https://polaris.apache.org/community/meetings/):
>
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
>
> As discussed during the meeting, here's the release proposal:
> - the first Polaris release will be 0.9. I'm doing a new pass on the
> preparation steps (due diligence on
> NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> we are mid October, I propose to actually release 0.9 at the beginning
> of November.
> This release is to verify our distribution with both the Polaris
> community and also the Apache Incubator (IPMC).
> - After 0.9 release, I propose to move forward to 1.0 release (I think
> it would be great to have 1.0 release before end of the year)
> - I also think we should use the "fast" major version as soon as we
> introduce any breaking change. I don't see a problem with having
> Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
>
> Regards
> JB
Re: Heading to Polaris 0.9 release and beyond
Posted by Robert Stupp <sn...@snazy.de>.
API/model (r)evolution and functionalities are different topics, but
very related when it comes to (semantic) versioning and releases.
(**flame off**) IMHO versions are rather just arbitrary,
somewhat subjectively chosen numbers, sometimes driven by marketing-y
thoughts... (**flame on**)
Re: Maintaining major+minor version-branches: I'd also like to not have
to maintain more than one branch, because backporting changes can be a
pain. However, I also think that it would be wrong to not have the
_ability_ to maintain multiple version branches, even if it's just for
relevant security updates or critical bug fixes. Having (too) many
active release version branches also lets an
compatibility/upgrade/downgrade matrix explode.
I think that we should at least try to stick with a single branch and
only release from there. Or a mixed model, where there are version
branches, but those only get (patch) version dependency upgrades -
mostly automated.
Re: API compatibility: I think, it's worth to define what a user-facing
API is. For me, it's really only the user-facing and documented REST
API(s) and nothing more. Those user-facing Polaris APIs must be designed
with backwards/forwards compatibility in mind. Assuming that the (REST)
API resources and service interfaces and service implementations are
"properly" abstracted and there's good-enough test coverage, it's
possible to do a "major REST API version bump" and maintain those for
quite some time.
Re: Deprecating (for removal): this is probably a case-by-case decision.
Whole REST API endpoints need to be retained much longer (years maybe?).
Some administrative functionality however can potentially be
removed/replaced relatively quickly.
SPIs, think: interfaces that are used by Polaris extensions/plugins, for
example pluggable authN/Z, have a different compatibility guarantee -
rather a "best effort" approach based on technical
needs/requirements/efforts. All other code, that's not explicitly
declared to be some kind of SPI, is *not* a public API. The Java
'public' modifier does not make a Java type a public API. Unlike
Iceberg/Parquet/etc, Polaris code is "only" a server/service
implementation - but not a dependency that users pull into their projects.
Backend (database persistence) however is another different animal.
Upgrades and downgrades and concurrently running mixed-version scenarios
must work.
Robert
On 24.10.24 19:25, Yufei Gu wrote:
> I believe it would be helpful to clearly define the following for our API
> management along with versioning:
>
> 1. *Valid APIs*: It's common to support multiple versions of a valid
> API. For example, both v1 and v2 of loadTable should remain functional.
> 2. *Deprecated APIs*: Even after deprecation, APIs should continue to be
> supported to avoid disruptions.
> 3. *Removing Deprecated APIs*: We need a careful approach for retiring
> deprecated APIs, as removing them can cause breaking changes for existing
> clients. This step should be treated as a breaking change and handled with
> caution.
>
> Additionally, it is helpful to differentiate between *specs* (e.g., REST
> endpoints) and *APIs* (public classes and methods). REST endpoints tend to
> have longer lifecycles and will be deprecated or removed according to a
> well-defined plan(discussion, announcement, etc), while public methods
> should adhere to semantic versioning.
>
>
> Yufei
>
>
> On Wed, Oct 23, 2024 at 4:55 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> By experience, it's very difficult to maintain two API versions in the
>> same codebase: very soon, we have a lot of "conditional statements" in
>> the code, which is not super elegant. Supporting previous API versions
>> (with routing) is possible as soon as we don't introduce "conditional
>> statements" per version.
>> The maintenance of branches is not necessarily painful: all new
>> features and improvements go on the latest active branch, other
>> branches just receive security fixes, dependency updates or major
>> fixes.
>> Else, we clearly state that we support only the latest active branch.
>> Several Apache projects maintain a couple of active branches (Apache
>> Camel, Apache ActiveMQ, etc).
>>
>> I think it makes sense to bump the Polaris version with the Iceberg
>> REST Spec version. The versions don't necessarily match (like Polaris
>> 3 can implement Iceberg REST Spec 2), but any new major REST Spec
>> version should result in a new major Polaris version.
>>
>> Regards
>> JB
>>
>> On Wed, Oct 23, 2024 at 1:05 PM Michael Collado <co...@gmail.com>
>> wrote:
>>> Personally, I’d rather not maintain two branches. I think it’s easier to
>>> maintain two API versions in the same codebase rather than have to back
>>> port changes into stale branches.
>>>
>>> This also leads to the question of what we do for Iceberg spec API
>> version
>>> changes. I can imagine if the whole REST spec changes to v2, we might
>>> release Polaris 2.0, but what do we do if Iceberg only increments the
>>> version for one or two APIs?
>>>
>>> Mike
>>>
>>> On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Good point.
>>>>
>>>> Imho, we should increment API versions when we introduce "breaking
>>>> changes".
>>>> On major versions, I don't see a need to support previous API
>>>> versions. Users who want to support previous API can always use
>>>> previous Polaris versions.
>>>> For us, we can "maintain" two major versions in parallel (the last one
>>>> with active new features, improvements, and bug fixes on both the last
>>>> version and previous one).
>>>>
>>>> For instance, in the Apache ActiveMQ project maintains both ActiveMQ
>>>> 5.x and 6.x, with complete different API support (ActiveMQ 5.x
>>>> implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
>>>> 3.1).
>>>>
>>>> So, regarding your example, I would say that Polaris 2.0 should not
>>>> necessary support v1 APIs, but focus on vX APIs and important changes.
>>>> We can still maintain Polaris 1.x for v1 API (with dependency updates
>>>> and bug fixes).
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <
>> collado.mike@gmail.com>
>>>> wrote:
>>>>> What's the support policy for API versions? Do we increment API
>> versions
>>>>> for breaking API changes (this seems reasonable and likely)? And do
>> we
>>>>> support previous API versions for some period of time?
>>>>>
>>>>> If the answer to both of those questions is "yes" , then what is the
>>>>> purpose of fast major verison releases? If we ship Polaris 2.0,
>> shouldn't
>>>>> the service still support the v1 APIs? For me, a 2.0 release should
>> mean
>>>>> pretty radical changes - new features, such as non-Iceberg table
>> support,
>>>>> delegated Authz policies, etc.
>>>>>
>>>>> On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com>
>> wrote:
>>>>>> I'm not opposed to rapid evolution, but I’m trying to understand
>> the
>>>>>> reasoning behind the fast major version. It’s ultimately a tradeoff
>>>> between
>>>>>> maintaining backward compatibility and advancing quickly through
>>>> breaking
>>>>>> changes—we can’t achieve both at the same time. A faster major
>> version
>>>>>> release seems to indicate a preference for rapid progress over
>>>> stability.
>>>>>> Otherwise, why push for it? Am I missing something?
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>
>>>>>>> IMHO, production compatibility concerns are valid, but they are
>> not
>>>>>> related
>>>>>>> to the versioning scheme.
>>>>>>>
>>>>>>> Backward-compatibility breakage should be declared anyway,
>> whether
>>>> the
>>>>>> move
>>>>>>> is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
>>>>>>>
>>>>>>> Still, bumping a major version indicates major change. It does
>> not
>>>>>>> automatically indicate incompatibility. My reading of Semver [1]
>>>>>> guidelines
>>>>>>> is that major must be incremented when incompatible changes are
>>>> present,
>>>>>>> but there is no restriction on incrementing major and keeping the
>>>> product
>>>>>>> backward compatible.
>>>>>>>
>>>>>>> On the other hand, if we avoid major code changes because of the
>>>> risk of
>>>>>>> incompatibility, that will slow down the project's evolution.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Dmitri.
>>>>>>>
>>>>>>> [1] https://semver.org/
>>>>>>>
>>>>>>> On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com>
>>>> wrote:
>>>>>>>> +1 on moving forward with 0.9 as the first version!
>>>>>>>>
>>>>>>>> I have some concerns about fast major versioning, especially
>> with
>>>>>>>> compatibility. Since the catalog is crucial in production,
>> teams
>>>> may be
>>>>>>>> reluctant to upgrade, particularly with breaking changes.
>>>>>>>>
>>>>>>>> Does fast versioning help in this case, or could it make things
>>>> harder?
>>>>>>>> Does it encourage more frequent upgrades, and is there a plan
>> to
>>>> manage
>>>>>>>> production impact? While we can't completely avoid user forks,
>> we
>>>> want
>>>>>> to
>>>>>>>> minimize divergence.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> +1 to "fast" major version.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Dmitri.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> First of all, thanks everyone for the first community
>> meeting
>>>>>>>>>> yesterday. It was a great meeting, very interesting
>>>> discussions.
>>>>>>>>>> You can find the meeting notes on the Polaris website
>>>>>>>>>> (https://polaris.apache.org/community/meetings/):
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
>>>>>>>>>> As discussed during the meeting, here's the release
>> proposal:
>>>>>>>>>> - the first Polaris release will be 0.9. I'm doing a new
>> pass
>>>> on
>>>>>> the
>>>>>>>>>> preparation steps (due diligence on
>>>>>>>>>> NOTICE/LICENSE/DISCLAIMER/dependencies, automatic
>> "scripts",
>>>> etc).
>>>>>> As
>>>>>>>>>> we are mid October, I propose to actually release 0.9 at
>> the
>>>>>>> beginning
>>>>>>>>>> of November.
>>>>>>>>>> This release is to verify our distribution with both the
>>>> Polaris
>>>>>>>>>> community and also the Apache Incubator (IPMC).
>>>>>>>>>> - After 0.9 release, I propose to move forward to 1.0
>> release
>>>> (I
>>>>>>> think
>>>>>>>>>> it would be great to have 1.0 release before end of the
>> year)
>>>>>>>>>> - I also think we should use the "fast" major version as
>> soon
>>>> as we
>>>>>>>>>> introduce any breaking change. I don't see a problem with
>>>> having
>>>>>>>>>> Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> JB
>>>>>>>>>>
--
Robert Stupp
@snazy
Re: Heading to Polaris 0.9 release and beyond
Posted by Yufei Gu <fl...@gmail.com>.
I believe it would be helpful to clearly define the following for our API
management along with versioning:
1. *Valid APIs*: It's common to support multiple versions of a valid
API. For example, both v1 and v2 of loadTable should remain functional.
2. *Deprecated APIs*: Even after deprecation, APIs should continue to be
supported to avoid disruptions.
3. *Removing Deprecated APIs*: We need a careful approach for retiring
deprecated APIs, as removing them can cause breaking changes for existing
clients. This step should be treated as a breaking change and handled with
caution.
Additionally, it is helpful to differentiate between *specs* (e.g., REST
endpoints) and *APIs* (public classes and methods). REST endpoints tend to
have longer lifecycles and will be deprecated or removed according to a
well-defined plan(discussion, announcement, etc), while public methods
should adhere to semantic versioning.
Yufei
On Wed, Oct 23, 2024 at 4:55 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> By experience, it's very difficult to maintain two API versions in the
> same codebase: very soon, we have a lot of "conditional statements" in
> the code, which is not super elegant. Supporting previous API versions
> (with routing) is possible as soon as we don't introduce "conditional
> statements" per version.
> The maintenance of branches is not necessarily painful: all new
> features and improvements go on the latest active branch, other
> branches just receive security fixes, dependency updates or major
> fixes.
> Else, we clearly state that we support only the latest active branch.
> Several Apache projects maintain a couple of active branches (Apache
> Camel, Apache ActiveMQ, etc).
>
> I think it makes sense to bump the Polaris version with the Iceberg
> REST Spec version. The versions don't necessarily match (like Polaris
> 3 can implement Iceberg REST Spec 2), but any new major REST Spec
> version should result in a new major Polaris version.
>
> Regards
> JB
>
> On Wed, Oct 23, 2024 at 1:05 PM Michael Collado <co...@gmail.com>
> wrote:
> >
> > Personally, I’d rather not maintain two branches. I think it’s easier to
> > maintain two API versions in the same codebase rather than have to back
> > port changes into stale branches.
> >
> > This also leads to the question of what we do for Iceberg spec API
> version
> > changes. I can imagine if the whole REST spec changes to v2, we might
> > release Polaris 2.0, but what do we do if Iceberg only increments the
> > version for one or two APIs?
> >
> > Mike
> >
> > On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Good point.
> > >
> > > Imho, we should increment API versions when we introduce "breaking
> > > changes".
> > > On major versions, I don't see a need to support previous API
> > > versions. Users who want to support previous API can always use
> > > previous Polaris versions.
> > > For us, we can "maintain" two major versions in parallel (the last one
> > > with active new features, improvements, and bug fixes on both the last
> > > version and previous one).
> > >
> > > For instance, in the Apache ActiveMQ project maintains both ActiveMQ
> > > 5.x and 6.x, with complete different API support (ActiveMQ 5.x
> > > implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
> > > 3.1).
> > >
> > > So, regarding your example, I would say that Polaris 2.0 should not
> > > necessary support v1 APIs, but focus on vX APIs and important changes.
> > > We can still maintain Polaris 1.x for v1 API (with dependency updates
> > > and bug fixes).
> > >
> > > Regards
> > > JB
> > >
> > > On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <
> collado.mike@gmail.com>
> > > wrote:
> > > >
> > > > What's the support policy for API versions? Do we increment API
> versions
> > > > for breaking API changes (this seems reasonable and likely)? And do
> we
> > > > support previous API versions for some period of time?
> > > >
> > > > If the answer to both of those questions is "yes" , then what is the
> > > > purpose of fast major verison releases? If we ship Polaris 2.0,
> shouldn't
> > > > the service still support the v1 APIs? For me, a 2.0 release should
> mean
> > > > pretty radical changes - new features, such as non-Iceberg table
> support,
> > > > delegated Authz policies, etc.
> > > >
> > > > On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > I'm not opposed to rapid evolution, but I’m trying to understand
> the
> > > > > reasoning behind the fast major version. It’s ultimately a tradeoff
> > > between
> > > > > maintaining backward compatibility and advancing quickly through
> > > breaking
> > > > > changes—we can’t achieve both at the same time. A faster major
> version
> > > > > release seems to indicate a preference for rapid progress over
> > > stability.
> > > > > Otherwise, why push for it? Am I missing something?
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
> > > > > <dm...@dremio.com.invalid> wrote:
> > > > >
> > > > > > IMHO, production compatibility concerns are valid, but they are
> not
> > > > > related
> > > > > > to the versioning scheme.
> > > > > >
> > > > > > Backward-compatibility breakage should be declared anyway,
> whether
> > > the
> > > > > move
> > > > > > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
> > > > > >
> > > > > > Still, bumping a major version indicates major change. It does
> not
> > > > > > automatically indicate incompatibility. My reading of Semver [1]
> > > > > guidelines
> > > > > > is that major must be incremented when incompatible changes are
> > > present,
> > > > > > but there is no restriction on incrementing major and keeping the
> > > product
> > > > > > backward compatible.
> > > > > >
> > > > > > On the other hand, if we avoid major code changes because of the
> > > risk of
> > > > > > incompatibility, that will slow down the project's evolution.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > [1] https://semver.org/
> > > > > >
> > > > > > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > +1 on moving forward with 0.9 as the first version!
> > > > > > >
> > > > > > > I have some concerns about fast major versioning, especially
> with
> > > > > > > compatibility. Since the catalog is crucial in production,
> teams
> > > may be
> > > > > > > reluctant to upgrade, particularly with breaking changes.
> > > > > > >
> > > > > > > Does fast versioning help in this case, or could it make things
> > > harder?
> > > > > > > Does it encourage more frequent upgrades, and is there a plan
> to
> > > manage
> > > > > > > production impact? While we can't completely avoid user forks,
> we
> > > want
> > > > > to
> > > > > > > minimize divergence.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > > > > > > <dm...@dremio.com.invalid> wrote:
> > > > > > >
> > > > > > > > +1 to "fast" major version.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
> > > > > jb@nanthrax.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > First of all, thanks everyone for the first community
> meeting
> > > > > > > > > yesterday. It was a great meeting, very interesting
> > > discussions.
> > > > > > > > > You can find the meeting notes on the Polaris website
> > > > > > > > > (https://polaris.apache.org/community/meetings/):
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > > > > > > >
> > > > > > > > > As discussed during the meeting, here's the release
> proposal:
> > > > > > > > > - the first Polaris release will be 0.9. I'm doing a new
> pass
> > > on
> > > > > the
> > > > > > > > > preparation steps (due diligence on
> > > > > > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic
> "scripts",
> > > etc).
> > > > > As
> > > > > > > > > we are mid October, I propose to actually release 0.9 at
> the
> > > > > > beginning
> > > > > > > > > of November.
> > > > > > > > > This release is to verify our distribution with both the
> > > Polaris
> > > > > > > > > community and also the Apache Incubator (IPMC).
> > > > > > > > > - After 0.9 release, I propose to move forward to 1.0
> release
> > > (I
> > > > > > think
> > > > > > > > > it would be great to have 1.0 release before end of the
> year)
> > > > > > > > > - I also think we should use the "fast" major version as
> soon
> > > as we
> > > > > > > > > introduce any breaking change. I don't see a problem with
> > > having
> > > > > > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > JB
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
By experience, it's very difficult to maintain two API versions in the
same codebase: very soon, we have a lot of "conditional statements" in
the code, which is not super elegant. Supporting previous API versions
(with routing) is possible as soon as we don't introduce "conditional
statements" per version.
The maintenance of branches is not necessarily painful: all new
features and improvements go on the latest active branch, other
branches just receive security fixes, dependency updates or major
fixes.
Else, we clearly state that we support only the latest active branch.
Several Apache projects maintain a couple of active branches (Apache
Camel, Apache ActiveMQ, etc).
I think it makes sense to bump the Polaris version with the Iceberg
REST Spec version. The versions don't necessarily match (like Polaris
3 can implement Iceberg REST Spec 2), but any new major REST Spec
version should result in a new major Polaris version.
Regards
JB
On Wed, Oct 23, 2024 at 1:05 PM Michael Collado <co...@gmail.com> wrote:
>
> Personally, I’d rather not maintain two branches. I think it’s easier to
> maintain two API versions in the same codebase rather than have to back
> port changes into stale branches.
>
> This also leads to the question of what we do for Iceberg spec API version
> changes. I can imagine if the whole REST spec changes to v2, we might
> release Polaris 2.0, but what do we do if Iceberg only increments the
> version for one or two APIs?
>
> Mike
>
> On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Good point.
> >
> > Imho, we should increment API versions when we introduce "breaking
> > changes".
> > On major versions, I don't see a need to support previous API
> > versions. Users who want to support previous API can always use
> > previous Polaris versions.
> > For us, we can "maintain" two major versions in parallel (the last one
> > with active new features, improvements, and bug fixes on both the last
> > version and previous one).
> >
> > For instance, in the Apache ActiveMQ project maintains both ActiveMQ
> > 5.x and 6.x, with complete different API support (ActiveMQ 5.x
> > implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
> > 3.1).
> >
> > So, regarding your example, I would say that Polaris 2.0 should not
> > necessary support v1 APIs, but focus on vX APIs and important changes.
> > We can still maintain Polaris 1.x for v1 API (with dependency updates
> > and bug fixes).
> >
> > Regards
> > JB
> >
> > On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <co...@gmail.com>
> > wrote:
> > >
> > > What's the support policy for API versions? Do we increment API versions
> > > for breaking API changes (this seems reasonable and likely)? And do we
> > > support previous API versions for some period of time?
> > >
> > > If the answer to both of those questions is "yes" , then what is the
> > > purpose of fast major verison releases? If we ship Polaris 2.0, shouldn't
> > > the service still support the v1 APIs? For me, a 2.0 release should mean
> > > pretty radical changes - new features, such as non-Iceberg table support,
> > > delegated Authz policies, etc.
> > >
> > > On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > I'm not opposed to rapid evolution, but I’m trying to understand the
> > > > reasoning behind the fast major version. It’s ultimately a tradeoff
> > between
> > > > maintaining backward compatibility and advancing quickly through
> > breaking
> > > > changes—we can’t achieve both at the same time. A faster major version
> > > > release seems to indicate a preference for rapid progress over
> > stability.
> > > > Otherwise, why push for it? Am I missing something?
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
> > > > <dm...@dremio.com.invalid> wrote:
> > > >
> > > > > IMHO, production compatibility concerns are valid, but they are not
> > > > related
> > > > > to the versioning scheme.
> > > > >
> > > > > Backward-compatibility breakage should be declared anyway, whether
> > the
> > > > move
> > > > > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
> > > > >
> > > > > Still, bumping a major version indicates major change. It does not
> > > > > automatically indicate incompatibility. My reading of Semver [1]
> > > > guidelines
> > > > > is that major must be incremented when incompatible changes are
> > present,
> > > > > but there is no restriction on incrementing major and keeping the
> > product
> > > > > backward compatible.
> > > > >
> > > > > On the other hand, if we avoid major code changes because of the
> > risk of
> > > > > incompatibility, that will slow down the project's evolution.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > [1] https://semver.org/
> > > > >
> > > > > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 on moving forward with 0.9 as the first version!
> > > > > >
> > > > > > I have some concerns about fast major versioning, especially with
> > > > > > compatibility. Since the catalog is crucial in production, teams
> > may be
> > > > > > reluctant to upgrade, particularly with breaking changes.
> > > > > >
> > > > > > Does fast versioning help in this case, or could it make things
> > harder?
> > > > > > Does it encourage more frequent upgrades, and is there a plan to
> > manage
> > > > > > production impact? While we can't completely avoid user forks, we
> > want
> > > > to
> > > > > > minimize divergence.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > > > > > <dm...@dremio.com.invalid> wrote:
> > > > > >
> > > > > > > +1 to "fast" major version.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
> > > > jb@nanthrax.net>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > First of all, thanks everyone for the first community meeting
> > > > > > > > yesterday. It was a great meeting, very interesting
> > discussions.
> > > > > > > > You can find the meeting notes on the Polaris website
> > > > > > > > (https://polaris.apache.org/community/meetings/):
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > > > > > >
> > > > > > > > As discussed during the meeting, here's the release proposal:
> > > > > > > > - the first Polaris release will be 0.9. I'm doing a new pass
> > on
> > > > the
> > > > > > > > preparation steps (due diligence on
> > > > > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts",
> > etc).
> > > > As
> > > > > > > > we are mid October, I propose to actually release 0.9 at the
> > > > > beginning
> > > > > > > > of November.
> > > > > > > > This release is to verify our distribution with both the
> > Polaris
> > > > > > > > community and also the Apache Incubator (IPMC).
> > > > > > > > - After 0.9 release, I propose to move forward to 1.0 release
> > (I
> > > > > think
> > > > > > > > it would be great to have 1.0 release before end of the year)
> > > > > > > > - I also think we should use the "fast" major version as soon
> > as we
> > > > > > > > introduce any breaking change. I don't see a problem with
> > having
> > > > > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > JB
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: Heading to Polaris 0.9 release and beyond
Posted by Michael Collado <co...@gmail.com>.
Personally, I’d rather not maintain two branches. I think it’s easier to
maintain two API versions in the same codebase rather than have to back
port changes into stale branches.
This also leads to the question of what we do for Iceberg spec API version
changes. I can imagine if the whole REST spec changes to v2, we might
release Polaris 2.0, but what do we do if Iceberg only increments the
version for one or two APIs?
Mike
On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Good point.
>
> Imho, we should increment API versions when we introduce "breaking
> changes".
> On major versions, I don't see a need to support previous API
> versions. Users who want to support previous API can always use
> previous Polaris versions.
> For us, we can "maintain" two major versions in parallel (the last one
> with active new features, improvements, and bug fixes on both the last
> version and previous one).
>
> For instance, in the Apache ActiveMQ project maintains both ActiveMQ
> 5.x and 6.x, with complete different API support (ActiveMQ 5.x
> implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
> 3.1).
>
> So, regarding your example, I would say that Polaris 2.0 should not
> necessary support v1 APIs, but focus on vX APIs and important changes.
> We can still maintain Polaris 1.x for v1 API (with dependency updates
> and bug fixes).
>
> Regards
> JB
>
> On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <co...@gmail.com>
> wrote:
> >
> > What's the support policy for API versions? Do we increment API versions
> > for breaking API changes (this seems reasonable and likely)? And do we
> > support previous API versions for some period of time?
> >
> > If the answer to both of those questions is "yes" , then what is the
> > purpose of fast major verison releases? If we ship Polaris 2.0, shouldn't
> > the service still support the v1 APIs? For me, a 2.0 release should mean
> > pretty radical changes - new features, such as non-Iceberg table support,
> > delegated Authz policies, etc.
> >
> > On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > I'm not opposed to rapid evolution, but I’m trying to understand the
> > > reasoning behind the fast major version. It’s ultimately a tradeoff
> between
> > > maintaining backward compatibility and advancing quickly through
> breaking
> > > changes—we can’t achieve both at the same time. A faster major version
> > > release seems to indicate a preference for rapid progress over
> stability.
> > > Otherwise, why push for it? Am I missing something?
> > >
> > > Yufei
> > >
> > >
> > > On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
> > > <dm...@dremio.com.invalid> wrote:
> > >
> > > > IMHO, production compatibility concerns are valid, but they are not
> > > related
> > > > to the versioning scheme.
> > > >
> > > > Backward-compatibility breakage should be declared anyway, whether
> the
> > > move
> > > > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
> > > >
> > > > Still, bumping a major version indicates major change. It does not
> > > > automatically indicate incompatibility. My reading of Semver [1]
> > > guidelines
> > > > is that major must be incremented when incompatible changes are
> present,
> > > > but there is no restriction on incrementing major and keeping the
> product
> > > > backward compatible.
> > > >
> > > > On the other hand, if we avoid major code changes because of the
> risk of
> > > > incompatibility, that will slow down the project's evolution.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > [1] https://semver.org/
> > > >
> > > > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > +1 on moving forward with 0.9 as the first version!
> > > > >
> > > > > I have some concerns about fast major versioning, especially with
> > > > > compatibility. Since the catalog is crucial in production, teams
> may be
> > > > > reluctant to upgrade, particularly with breaking changes.
> > > > >
> > > > > Does fast versioning help in this case, or could it make things
> harder?
> > > > > Does it encourage more frequent upgrades, and is there a plan to
> manage
> > > > > production impact? While we can't completely avoid user forks, we
> want
> > > to
> > > > > minimize divergence.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > > > > <dm...@dremio.com.invalid> wrote:
> > > > >
> > > > > > +1 to "fast" major version.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > First of all, thanks everyone for the first community meeting
> > > > > > > yesterday. It was a great meeting, very interesting
> discussions.
> > > > > > > You can find the meeting notes on the Polaris website
> > > > > > > (https://polaris.apache.org/community/meetings/):
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > > > > >
> > > > > > > As discussed during the meeting, here's the release proposal:
> > > > > > > - the first Polaris release will be 0.9. I'm doing a new pass
> on
> > > the
> > > > > > > preparation steps (due diligence on
> > > > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts",
> etc).
> > > As
> > > > > > > we are mid October, I propose to actually release 0.9 at the
> > > > beginning
> > > > > > > of November.
> > > > > > > This release is to verify our distribution with both the
> Polaris
> > > > > > > community and also the Apache Incubator (IPMC).
> > > > > > > - After 0.9 release, I propose to move forward to 1.0 release
> (I
> > > > think
> > > > > > > it would be great to have 1.0 release before end of the year)
> > > > > > > - I also think we should use the "fast" major version as soon
> as we
> > > > > > > introduce any breaking change. I don't see a problem with
> having
> > > > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Good point.
Imho, we should increment API versions when we introduce "breaking changes".
On major versions, I don't see a need to support previous API
versions. Users who want to support previous API can always use
previous Polaris versions.
For us, we can "maintain" two major versions in parallel (the last one
with active new features, improvements, and bug fixes on both the last
version and previous one).
For instance, in the Apache ActiveMQ project maintains both ActiveMQ
5.x and 6.x, with complete different API support (ActiveMQ 5.x
implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
3.1).
So, regarding your example, I would say that Polaris 2.0 should not
necessary support v1 APIs, but focus on vX APIs and important changes.
We can still maintain Polaris 1.x for v1 API (with dependency updates
and bug fixes).
Regards
JB
On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <co...@gmail.com> wrote:
>
> What's the support policy for API versions? Do we increment API versions
> for breaking API changes (this seems reasonable and likely)? And do we
> support previous API versions for some period of time?
>
> If the answer to both of those questions is "yes" , then what is the
> purpose of fast major verison releases? If we ship Polaris 2.0, shouldn't
> the service still support the v1 APIs? For me, a 2.0 release should mean
> pretty radical changes - new features, such as non-Iceberg table support,
> delegated Authz policies, etc.
>
> On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > I'm not opposed to rapid evolution, but I’m trying to understand the
> > reasoning behind the fast major version. It’s ultimately a tradeoff between
> > maintaining backward compatibility and advancing quickly through breaking
> > changes—we can’t achieve both at the same time. A faster major version
> > release seems to indicate a preference for rapid progress over stability.
> > Otherwise, why push for it? Am I missing something?
> >
> > Yufei
> >
> >
> > On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
> > <dm...@dremio.com.invalid> wrote:
> >
> > > IMHO, production compatibility concerns are valid, but they are not
> > related
> > > to the versioning scheme.
> > >
> > > Backward-compatibility breakage should be declared anyway, whether the
> > move
> > > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
> > >
> > > Still, bumping a major version indicates major change. It does not
> > > automatically indicate incompatibility. My reading of Semver [1]
> > guidelines
> > > is that major must be incremented when incompatible changes are present,
> > > but there is no restriction on incrementing major and keeping the product
> > > backward compatible.
> > >
> > > On the other hand, if we avoid major code changes because of the risk of
> > > incompatibility, that will slow down the project's evolution.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > [1] https://semver.org/
> > >
> > > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > +1 on moving forward with 0.9 as the first version!
> > > >
> > > > I have some concerns about fast major versioning, especially with
> > > > compatibility. Since the catalog is crucial in production, teams may be
> > > > reluctant to upgrade, particularly with breaking changes.
> > > >
> > > > Does fast versioning help in this case, or could it make things harder?
> > > > Does it encourage more frequent upgrades, and is there a plan to manage
> > > > production impact? While we can't completely avoid user forks, we want
> > to
> > > > minimize divergence.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > > > <dm...@dremio.com.invalid> wrote:
> > > >
> > > > > +1 to "fast" major version.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > First of all, thanks everyone for the first community meeting
> > > > > > yesterday. It was a great meeting, very interesting discussions.
> > > > > > You can find the meeting notes on the Polaris website
> > > > > > (https://polaris.apache.org/community/meetings/):
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > > > >
> > > > > > As discussed during the meeting, here's the release proposal:
> > > > > > - the first Polaris release will be 0.9. I'm doing a new pass on
> > the
> > > > > > preparation steps (due diligence on
> > > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc).
> > As
> > > > > > we are mid October, I propose to actually release 0.9 at the
> > > beginning
> > > > > > of November.
> > > > > > This release is to verify our distribution with both the Polaris
> > > > > > community and also the Apache Incubator (IPMC).
> > > > > > - After 0.9 release, I propose to move forward to 1.0 release (I
> > > think
> > > > > > it would be great to have 1.0 release before end of the year)
> > > > > > - I also think we should use the "fast" major version as soon as we
> > > > > > introduce any breaking change. I don't see a problem with having
> > > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Heading to Polaris 0.9 release and beyond
Posted by Michael Collado <co...@gmail.com>.
What's the support policy for API versions? Do we increment API versions
for breaking API changes (this seems reasonable and likely)? And do we
support previous API versions for some period of time?
If the answer to both of those questions is "yes" , then what is the
purpose of fast major verison releases? If we ship Polaris 2.0, shouldn't
the service still support the v1 APIs? For me, a 2.0 release should mean
pretty radical changes - new features, such as non-Iceberg table support,
delegated Authz policies, etc.
On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <fl...@gmail.com> wrote:
> I'm not opposed to rapid evolution, but I’m trying to understand the
> reasoning behind the fast major version. It’s ultimately a tradeoff between
> maintaining backward compatibility and advancing quickly through breaking
> changes—we can’t achieve both at the same time. A faster major version
> release seems to indicate a preference for rapid progress over stability.
> Otherwise, why push for it? Am I missing something?
>
> Yufei
>
>
> On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
> > IMHO, production compatibility concerns are valid, but they are not
> related
> > to the versioning scheme.
> >
> > Backward-compatibility breakage should be declared anyway, whether the
> move
> > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
> >
> > Still, bumping a major version indicates major change. It does not
> > automatically indicate incompatibility. My reading of Semver [1]
> guidelines
> > is that major must be incremented when incompatible changes are present,
> > but there is no restriction on incrementing major and keeping the product
> > backward compatible.
> >
> > On the other hand, if we avoid major code changes because of the risk of
> > incompatibility, that will slow down the project's evolution.
> >
> > Cheers,
> > Dmitri.
> >
> > [1] https://semver.org/
> >
> > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > +1 on moving forward with 0.9 as the first version!
> > >
> > > I have some concerns about fast major versioning, especially with
> > > compatibility. Since the catalog is crucial in production, teams may be
> > > reluctant to upgrade, particularly with breaking changes.
> > >
> > > Does fast versioning help in this case, or could it make things harder?
> > > Does it encourage more frequent upgrades, and is there a plan to manage
> > > production impact? While we can't completely avoid user forks, we want
> to
> > > minimize divergence.
> > >
> > > Yufei
> > >
> > >
> > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > > <dm...@dremio.com.invalid> wrote:
> > >
> > > > +1 to "fast" major version.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > First of all, thanks everyone for the first community meeting
> > > > > yesterday. It was a great meeting, very interesting discussions.
> > > > > You can find the meeting notes on the Polaris website
> > > > > (https://polaris.apache.org/community/meetings/):
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > > >
> > > > > As discussed during the meeting, here's the release proposal:
> > > > > - the first Polaris release will be 0.9. I'm doing a new pass on
> the
> > > > > preparation steps (due diligence on
> > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc).
> As
> > > > > we are mid October, I propose to actually release 0.9 at the
> > beginning
> > > > > of November.
> > > > > This release is to verify our distribution with both the Polaris
> > > > > community and also the Apache Incubator (IPMC).
> > > > > - After 0.9 release, I propose to move forward to 1.0 release (I
> > think
> > > > > it would be great to have 1.0 release before end of the year)
> > > > > - I also think we should use the "fast" major version as soon as we
> > > > > introduce any breaking change. I don't see a problem with having
> > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > >
> > >
> >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Yufei Gu <fl...@gmail.com>.
I'm not opposed to rapid evolution, but I’m trying to understand the
reasoning behind the fast major version. It’s ultimately a tradeoff between
maintaining backward compatibility and advancing quickly through breaking
changes—we can’t achieve both at the same time. A faster major version
release seems to indicate a preference for rapid progress over stability.
Otherwise, why push for it? Am I missing something?
Yufei
On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
<dm...@dremio.com.invalid> wrote:
> IMHO, production compatibility concerns are valid, but they are not related
> to the versioning scheme.
>
> Backward-compatibility breakage should be declared anyway, whether the move
> is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
>
> Still, bumping a major version indicates major change. It does not
> automatically indicate incompatibility. My reading of Semver [1] guidelines
> is that major must be incremented when incompatible changes are present,
> but there is no restriction on incrementing major and keeping the product
> backward compatible.
>
> On the other hand, if we avoid major code changes because of the risk of
> incompatibility, that will slow down the project's evolution.
>
> Cheers,
> Dmitri.
>
> [1] https://semver.org/
>
> On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > +1 on moving forward with 0.9 as the first version!
> >
> > I have some concerns about fast major versioning, especially with
> > compatibility. Since the catalog is crucial in production, teams may be
> > reluctant to upgrade, particularly with breaking changes.
> >
> > Does fast versioning help in this case, or could it make things harder?
> > Does it encourage more frequent upgrades, and is there a plan to manage
> > production impact? While we can't completely avoid user forks, we want to
> > minimize divergence.
> >
> > Yufei
> >
> >
> > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> > <dm...@dremio.com.invalid> wrote:
> >
> > > +1 to "fast" major version.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > First of all, thanks everyone for the first community meeting
> > > > yesterday. It was a great meeting, very interesting discussions.
> > > > You can find the meeting notes on the Polaris website
> > > > (https://polaris.apache.org/community/meetings/):
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > > >
> > > > As discussed during the meeting, here's the release proposal:
> > > > - the first Polaris release will be 0.9. I'm doing a new pass on the
> > > > preparation steps (due diligence on
> > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> > > > we are mid October, I propose to actually release 0.9 at the
> beginning
> > > > of November.
> > > > This release is to verify our distribution with both the Polaris
> > > > community and also the Apache Incubator (IPMC).
> > > > - After 0.9 release, I propose to move forward to 1.0 release (I
> think
> > > > it would be great to have 1.0 release before end of the year)
> > > > - I also think we should use the "fast" major version as soon as we
> > > > introduce any breaking change. I don't see a problem with having
> > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > > >
> > > > Regards
> > > > JB
> > > >
> > >
> >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
IMHO, production compatibility concerns are valid, but they are not related
to the versioning scheme.
Backward-compatibility breakage should be declared anyway, whether the move
is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
Still, bumping a major version indicates major change. It does not
automatically indicate incompatibility. My reading of Semver [1] guidelines
is that major must be incremented when incompatible changes are present,
but there is no restriction on incrementing major and keeping the product
backward compatible.
On the other hand, if we avoid major code changes because of the risk of
incompatibility, that will slow down the project's evolution.
Cheers,
Dmitri.
[1] https://semver.org/
On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <fl...@gmail.com> wrote:
> +1 on moving forward with 0.9 as the first version!
>
> I have some concerns about fast major versioning, especially with
> compatibility. Since the catalog is crucial in production, teams may be
> reluctant to upgrade, particularly with breaking changes.
>
> Does fast versioning help in this case, or could it make things harder?
> Does it encourage more frequent upgrades, and is there a plan to manage
> production impact? While we can't completely avoid user forks, we want to
> minimize divergence.
>
> Yufei
>
>
> On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
> > +1 to "fast" major version.
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi folks,
> > >
> > > First of all, thanks everyone for the first community meeting
> > > yesterday. It was a great meeting, very interesting discussions.
> > > You can find the meeting notes on the Polaris website
> > > (https://polaris.apache.org/community/meetings/):
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> > >
> > > As discussed during the meeting, here's the release proposal:
> > > - the first Polaris release will be 0.9. I'm doing a new pass on the
> > > preparation steps (due diligence on
> > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> > > we are mid October, I propose to actually release 0.9 at the beginning
> > > of November.
> > > This release is to verify our distribution with both the Polaris
> > > community and also the Apache Incubator (IPMC).
> > > - After 0.9 release, I propose to move forward to 1.0 release (I think
> > > it would be great to have 1.0 release before end of the year)
> > > - I also think we should use the "fast" major version as soon as we
> > > introduce any breaking change. I don't see a problem with having
> > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> > >
> > > Regards
> > > JB
> > >
> >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Yufei Gu <fl...@gmail.com>.
+1 on moving forward with 0.9 as the first version!
I have some concerns about fast major versioning, especially with
compatibility. Since the catalog is crucial in production, teams may be
reluctant to upgrade, particularly with breaking changes.
Does fast versioning help in this case, or could it make things harder?
Does it encourage more frequent upgrades, and is there a plan to manage
production impact? While we can't completely avoid user forks, we want to
minimize divergence.
Yufei
On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
<dm...@dremio.com.invalid> wrote:
> +1 to "fast" major version.
>
> Cheers,
> Dmitri.
>
> On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi folks,
> >
> > First of all, thanks everyone for the first community meeting
> > yesterday. It was a great meeting, very interesting discussions.
> > You can find the meeting notes on the Polaris website
> > (https://polaris.apache.org/community/meetings/):
> >
> >
> >
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
> >
> > As discussed during the meeting, here's the release proposal:
> > - the first Polaris release will be 0.9. I'm doing a new pass on the
> > preparation steps (due diligence on
> > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> > we are mid October, I propose to actually release 0.9 at the beginning
> > of November.
> > This release is to verify our distribution with both the Polaris
> > community and also the Apache Incubator (IPMC).
> > - After 0.9 release, I propose to move forward to 1.0 release (I think
> > it would be great to have 1.0 release before end of the year)
> > - I also think we should use the "fast" major version as soon as we
> > introduce any breaking change. I don't see a problem with having
> > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
> >
> > Regards
> > JB
> >
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
+1 to "fast" major version.
Cheers,
Dmitri.
On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi folks,
>
> First of all, thanks everyone for the first community meeting
> yesterday. It was a great meeting, very interesting discussions.
> You can find the meeting notes on the Polaris website
> (https://polaris.apache.org/community/meetings/):
>
>
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
>
> As discussed during the meeting, here's the release proposal:
> - the first Polaris release will be 0.9. I'm doing a new pass on the
> preparation steps (due diligence on
> NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> we are mid October, I propose to actually release 0.9 at the beginning
> of November.
> This release is to verify our distribution with both the Polaris
> community and also the Apache Incubator (IPMC).
> - After 0.9 release, I propose to move forward to 1.0 release (I think
> it would be great to have 1.0 release before end of the year)
> - I also think we should use the "fast" major version as soon as we
> introduce any breaking change. I don't see a problem with having
> Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
>
> Regards
> JB
>
Re: Heading to Polaris 0.9 release and beyond
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi folks,
Quick update about the releases:
- I sent a message yesterday about Quarkus effort on Polaris. A
working branch is in progress, a "formal" PR should be opened soon.
The goal is to have Polaris powered by Quarkus for Polaris 1.0 release
- I'm working on 0.9 release preparation. As reminder, 0.9 is a kind
of "test" release to document and test our release process and verify
the distributions (source, binary, etc). I plan to submit 0.9 to vote
next week.
Regards
JB
On Fri, Oct 18, 2024 at 11:17 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi folks,
>
> First of all, thanks everyone for the first community meeting
> yesterday. It was a great meeting, very interesting discussions.
> You can find the meeting notes on the Polaris website
> (https://polaris.apache.org/community/meetings/):
>
> https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
>
> As discussed during the meeting, here's the release proposal:
> - the first Polaris release will be 0.9. I'm doing a new pass on the
> preparation steps (due diligence on
> NOTICE/LICENSE/DISCLAIMER/dependencies, automatic "scripts", etc). As
> we are mid October, I propose to actually release 0.9 at the beginning
> of November.
> This release is to verify our distribution with both the Polaris
> community and also the Apache Incubator (IPMC).
> - After 0.9 release, I propose to move forward to 1.0 release (I think
> it would be great to have 1.0 release before end of the year)
> - I also think we should use the "fast" major version as soon as we
> introduce any breaking change. I don't see a problem with having
> Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
>
> Regards
> JB