You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/01 12:52:40 UTC

[GitHub] [arrow-rs] alamb opened a new issue #1120: More frequent major releases for arrow-rs

alamb opened a new issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120


   TDLR: I propose doing major releases for arrow-rs more frequently (up to every other week) directly from `master`, breaking the correspondence with the main arrow release
   
   For example, the release cadence might look like 
   * 2022-01-01: `7.0.0`
   * 2022-01-15: `7.1.0` (no backwards incompatible changes)
   * 2022-02-01: `8.0.0` (new backwards incompatible change)
   * 2022-02-15: `9.0.0` (new backwards incompatible change)
   * 2022-03-01: `9.1.0` etc
   * 2022-03-15: `9.2.0` 
   * 2022-04-01: `9.3.0` 
   * 2022-04-15: `10.0.0` 
   ...
   
   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   The rust ecosystem as a whole is "fast changing". This means:
   1. Many pieces of the ecosystem change frequently that require downstream updates (e.g. new clippy lints released in stable versions of rust that require code fixes to pass CI)
   2. Many crate authors don't spend effort maintaining stable releases, but instead fix issues by pushing new releases
   
   This means the ability to quickly upgrade to new downstream libraries is critical to help fix issues like https://github.com/apache/arrow-rs/issues/1101
   
   With a few notable exceptions (e.g. tokio) most of the rust ecosystem pushes new major releases frequently (monthly if not more so), often via `0.X` releases.
   
   The current [release cadence and versioning scheme](https://github.com/apache/arrow-rs/blob/master/dev/release/README.md) of `arrow-rs` inherits from the C/C++ and python ecosystems with a major release every 3 months while maintaining a backwards compatible branch.
   
   I propose moving the rust implementation in closer alignment with the rest of the rust ecosystem with more frequent major version releases.
   
   **Describe the solution you'd like**
   
   Continue to release every 2 weeks; However, release all new versions directly from the `master` branch, picking a new version number based on the changes in the crate (a new major version if there were semver changes,  minor if not)
   
   Pros:
   1. Same number of releases as today (1 every other week)
   2. Less branch maintenance overhead (selfishly good for me)
   3. Major changes to dependencies(e.g. upgrade to newer versions of prost) available sooner
   
   Cons:
   1. downstream crates would likely have to do major updates more frequently (`cargo update` doesn't pick up new major versions)
   2. Rust versions would no longer necessarily align with  C++/etc mono repo release version
   
   Much of the rust ecosystem, as described above, is used to the "do major updates frequently" mindset, and furthermore tools such as `dependabot` reduce the effort required to do so, I believe the first con is manageable. 
   
   **Describe alternatives you've considered**
   
   Option: 1 "major release on demand", where we released when there were "enough" changes built up to do a regular release;
   
   Option 2: keep the same structure (a `master` and `active_release` branch); do a major release every month instead of every 3 months, and do a minor release every other week.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1042811407


   (reddit post https://www.reddit.com/r/rust/comments/ss8vjy/february_2022_rust_apache_arrow_and_parquet/)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb closed issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb closed issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1006541097


   > Maybe we need more active PMC for arrow in the rust ecosystem.
   
   Yeah -- I don't really know how to improve this. We do have several representatives of the Rust implementation on the PMC now (@jorgecarleitao @nevi-me @andygrove and @Dandandan) but their time is limited (as all of ours is)
   
   I do wonder how much of the issue is the current process, which suggests running an automated script. I am not convinced this extra level of "quality assurance" is a real value add -- the real value add to me is having more than one person look at the release's content and say that the "content seems reasonable."
   
   Are there any other thoughts on this thread?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] liukun4515 commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1006202007


   > > I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release.
   > 
   > The major thing that prevents more frequent and/or automated releases is the release voting process. I often have to chase (via slack, etc) to get 3 PMC members to approve a release;
   
   Maybe we need more active PMC for arrow in the rust ecosystem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] liukun4515 commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1006200930


   Waiting for the release of 7.0.0!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1042814463


   Thank you for your feedback @tafia -- it is nice to hear from someone who valued the incremental updates!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] tafia commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
tafia commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1042672561


   I know this is an old issue but just giving my 2 cents. 
   
   While I definitely agree working out of master directly, bumping a major version so frequently is not super convenient.
   
   > Much of the rust ecosystem, as described above, is used to the "do major updates frequently" mindset
   
   I am not sure where this is coming from. From my experience there are many important crates which are trying hard not to release new major versions unless necessary (rocket, actix-web, chrono, tokio, log for the most important ones). 
   
   As explained in a reddit post, I have a hard time keeping up with the major version changes. Updating one library is easy, updating all its reverse dependency much harder.
   
   Anyway, thanks a lot for the hard work!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005605490


   > I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release. 
   
   The major thing that prevents more frequent and/or automated releases is the release voting process. I often have to chase (via slack, etc) to get 3 PMC members to approve a release;
   
   In terms of automation,  sqlparser-rs uses `cargo release` -- https://github.com/sqlparser-rs/sqlparser-rs/blob/main/docs/releasing.md and it is pretty sweeet. 
   
   > Edit: a somewhat related question is must all the crates in this repo advance their version numbers in lockstep? Should a breaking release to parquet mandate a breaking semver bump to arrow?
   
   I don't know of any technical reason they not need to advance their numbers in lockstep -- it is a convenience so that we (mostly so I) don't have to scrutinize the changes to each crate and determine if/when a new version is warranted on each release


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] liukun4515 commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005367698


   If the master has the API-change commit after the last release, how to handle this?
   @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] tustvold commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005421434


   I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release. The major benefit would be this could be completely automated, potentially using existing tooling.
   
   In the past I've used this approach for binaries, in particular using goreleaser, but I see no obvious reason something similar couldn't be done here...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] tustvold commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1003613448


   > often via 0.X releases
   
   I think this touches on something key. In my opinion, the crates in this repository are still somewhat beta if not alpha software, with major APIs still getting fleshed out. As such it would perhaps be more expected for it to be 0.x software with the accompanying frequency of breaking releases. I accept the version ship has sailed, but I think it makes sense to pretend it hasn't and adopt a more rapid breaking release cadence as suggested :+1:
   
   FWIW similar arguments apply to many of the arrow-rs dependencies. It isn't so much that the ecosystem itself is "fast changing", just a lot of the ecosystem isn't yet v1 :laughing:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005601790


   > If the master has the API-change commit after the last release, how to handle this?
   
   @liukun4515  the proposal is that the following release would have a new major (rather than minor version)
   
   So for example, if there was an API change on Jan 12 (after 7.0.0 was released) the next release after Jan 12 would become 8.0.0 (rather than 7.1.0)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005188297


   In conclusion, from my perspective maintaining an active release branch with stable patches and major releases every 3 months doesn't provide enough value for the cost 
   
   The cost is borne both by me actually creating the releases as well as users who are slowed down picking up updates for dependent libraries (the latest version of tonic, for example) or waiting for their changes to be released.
   
   So unless there is significant pushback (ideally along with a volunteer to help maintain a stable branch) I will plan to start doing releases for arrow-rs directly from master as proposed on [1] , starting with arrow-rs 7.0 (will make a candidate later this week and hope to release early next)
   
   I think the only downside to this will be the oft discussed "arrow will end up with versions 22.0.0 rather than version 0.22.0" which may lead to confusion about its relative stability. We can add some documentation about this and adjust if it causes too much confusion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005189688


   
   TLDR is I plan to release arrow-rs 7.0 as normally scheduled at the end of this week, and then after that only release from master every other week (rather than maintaining a more stable active_release branch)
   
   So the first time something different will happen is in 2.5 weeks time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1012387837


   Given there are no more comments I'll proceed with this proposal 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] paddyhoran commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
paddyhoran commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1004170916


   Also another pro is that we can hopefully bring arrow2 back into the fold (though that would need confirmation/additional work).  As I remember it, @jorgecarleitao's main sticking point was that the Rust version of arrow was sticking with the C++/Python versioning scheme.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] houqp commented on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1007086385


   I don't have anything to add other than that I fully support this move :) It will certain reduce @alamb 's maintenance burden and make it easier for downstream to get access to breaking changes :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] tustvold edited a comment on issue #1120: More frequent major releases for arrow-rs

Posted by GitBox <gi...@apache.org>.
tustvold edited a comment on issue #1120:
URL: https://github.com/apache/arrow-rs/issues/1120#issuecomment-1005421434


   I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release. The major benefit would be this could be completely automated, potentially using existing tooling.
   
   In the past I've used this approach for binaries, in particular using goreleaser, but I see no obvious reason something similar couldn't be done here...
   
   Edit: a somewhat related question is must all the crates in this repo advance their version numbers in lockstep? Should a breaking release to parquet mandate a breaking semver bump to arrow?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org