You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/24 12:45:38 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #2327: More frequent DataFusion releases to crates.io (discussion)
alamb opened a new issue, #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
DataFusion's mission statement is to be "easy to embed" from https://arrow.apache.org/datafusion/user-guide/introduction.html#why-datafusion
Within the rust ecosystem, publishing to crates.io is a key way to make something easy to embed (other projects can then use it with a single line of `toml`). However, we have been releasing about once every three months, which limits the speed at which we can publish new features to crates. io for use in downstream projects
The most active projects of the users today of datafuson use a fork or work off the master branch directly (see presentation linked in #2323).
**Describe the solution you'd like**
It would be awesome to release DataFusion more frequently, perhaps in a semver compatible way, to encourage more community use.
For example, as suggested by @martinitus https://github.com/apache/arrow-datafusion/issues/37#issuecomment-1106605474 when there is a new major version of arrow-rs:
> I think general best practice (given you use semver) is that if you have a major increase in a dependency that appears in your public API, then you also need a major increase in your library version, as the stuff that appear in your public API may break backwards compatibility for the users of your API.
**Describe alternatives you've considered**
Keep with the current approximately quarterly release schedule, at least until the APIs have stabilized more.
**Additional context**
Managing regular releases is a substantial undertaking:
1. Managing the actual release process requires significant time
2. Managing what code changes can go where and still maintain semver is non trivial (e.g. backporting 'semver' compatible changes -- see discussion on https://github.com/apache/arrow-rs/issues/1120 for some flavor). We (I) abandoned doing this in arrow-rs due to lack of time.
I don't personally have the bandwidth at this time to organize / manage this process; However, I wanted to get it on our collective radar, and I would be happy to help guide and support anyone who wants to invest the time making this happen for the community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1110838451
> Another side note: Other rust projects used pre v1.0.0 minor commits to stabilize their APIs for a while. During that phase, they break compatibility also during minor version increments. Once they are "happy" with things they stabilize it in a version 1.0.0 promising strong backwards compatibility from there on.
Yes, I agree this would be the ideal versioning scheme. The reason DataFusion is already in > 1.0.0 version is partly due to history, as in the past it was released with the rest of the arrow implementations and matched version numbers.
> That said, I see no real downsides in frequent major releases during stabilization :)
The only downside I see is that there may be some misaligned expectations with some users. However, it hasn't seemed to cause any major issues yet, that I know of.
> Awesome work! <3
❤️ thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1115486289
Another option to consider would be [Calendar Versioning](https://calver.org/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
jychen7 commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1120341778
> More frequent DataFusion releases to crates.io
do we mean more frequent [major release](https://github.com/apache/arrow-datafusion/blob/master/dev/release/README.md#major-release) from `master` branch?
> Another option to consider would be [Calendar Versioning](https://calver.org/)
I like it!
> andygrove commented 12 days ago
> I would like to volunteer to help with this
I also can help with this, but from last time 7.1.0 release ([PR](https://github.com/apache/arrow-datafusion/pull/2187)), though I can help prepare release note, it still require committer to take time to prepare release candidate artifacts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1108937303
I would like to volunteer to help with this. I think that the dask-sql projection is going to benefit from more frequent releases of DataFusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1399513677
Given @andygrove is now running releases every month or so I think we can close this particular issue and open other tickets for other discussiosn
Discussion of "nightly" releases: https://github.com/apache/arrow-datafusion/issues/5023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #2327: More frequent DataFusion releases to crates.io (discussion)
URL: https://github.com/apache/arrow-datafusion/issues/2327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1120415130
> I also can help with this, but from last time 7.1.0 release (https://github.com/apache/arrow-datafusion/pull/2187), though I can help prepare release note, it still require committe
Thank you @jychen7 -- your assistance for 7.1.0 was very helpful
I find the mechanics of creating the artifacts and voting takes relatively minimal time compared to creating release notes and ensuring everything is ready. The assistance of the community on release notes and documentation is *amazing*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] martinitus commented on issue #2327: More frequent DataFusion releases to crates.io (discussion)
Posted by GitBox <gi...@apache.org>.
martinitus commented on issue #2327:
URL: https://github.com/apache/arrow-datafusion/issues/2327#issuecomment-1109966426
Another side note: Other rust projects used pre v1.0.0 minor commits to stabilize their APIs for a while. During that phase, they break compatibility also during minor version increments. Once they are "happy" with things they stabilize it in a version 1.0.0 promising strong backwards compatibility from there on.
I guess examples for this would be tokio or hyper?!
That said, I see no real downsides in frequent major releases during stabilization :)
Awesome work! <3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org