You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Neal Richardson <ne...@gmail.com> on 2020/03/24 15:00:53 UTC
Preparing for 0.17.0 Arrow release
Hi all,
A few weeks ago, there seemed to be consensus (lazy, at least) for a 0.17
release at the end of the month. Judging from
https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release, it
looks like we're getting closer.
I'd encourage everyone to review their backlogs and (1) bump from 0.17
scope any tickets they don't plan to finish this week, and (2) if there are
any issues that should block release, make sure they are flagged as
"blockers".
Neal
On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com> wrote:
> It seems like the consensus is to push for a 0.17.0 major release
> sooner rather than doing a patch release, since releases in general
> are costly. This is fine with me. I see that a 0.17.0 milestone has
> been created in JIRA and some JIRA gardening has begun. Do you think
> we can be in a position to release by the week of March 23 or the week
> of March 30?
>
> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > If people are generally on board with accelerating a 0.17.0 major
> > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > beginning to do issue gardening to whittle things down to
> > critical-looking bugs and high probability patches for the next couple
> > of weeks.
> >
> > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > I recall there are some other issues that have been reported or fixed
> > > that are critical and not yet marked with 0.16.1.
> > >
> > > I'm also OK with doing a 0.17.0 release sooner
> > >
> > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > <ne...@gmail.com> wrote:
> > > >
> > > > I would also be more supportive of doing 0.17 earlier instead of a
> patch
> > > > release.
> > > >
> > > > Neal
> > > >
> > > >
> > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> neal.p.richardson@gmail.com>
> > > > wrote:
> > > >
> > > > > If releases were costless to make, I'd be all for it, but it's not
> clear
> > > > > to me that it's worth the diversion from other priorities to make
> a release
> > > > > right now. Nothing on
> > > > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > jumps out to me as super urgent--what are you seeing as critical?
> > > > >
> > > > > If we did decide to go forward, would it be possible to do a
> release that
> > > > > is limited to the affected implementations (say, do a Python-only
> release)?
> > > > > That might reduce the cost of building and verifying enough to
> make it
> > > > > reasonable to consider.
> > > > >
> > > > > Neal
> > > > >
> > > > >
> > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> szucs.krisztian@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > > >> >
> > > > >> > hi folks,
> > > > >> >
> > > > >> > There have been a number of critical issues reported (many of
> them
> > > > >> > fixed already) since 0.16.0 was released. Is there interest in
> > > > >> > preparing a patch 0.16.1 release (with backported patches onto a
> > > > >> > maint-0.16.x branch as with 0.15.1) since the next major
> release is a
> > > > >> > minimum of 6-8 weeks away from general availability?
> > > > >> >
> > > > >> > Did the 0.15.1 patch release helper script that Krisztian wrote
> get
> > > > >> > contributed as a PR?
> > > > >> Not yet, but it is available at
> > > > >> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > >> >
> > > > >> > Thanks
> > > > >> > Wes
> > > > >>
> > > > >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Krisztián Szűcs <sz...@gmail.com>.
if you mean he connection error that's a flake. The kartothek build
nightly passes in the last 3 reports.
On Sun, Apr 12, 2020 at 3:57 AM Wes McKinney <we...@gmail.com> wrote:
>
> I'm not able to reproduce the test-conda-python-3.7-kartothek-master
> failure locally with docker-compose, is that a flake or real?
>
> On Sat, Apr 11, 2020 at 7:29 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > On Sat, Apr 11, 2020 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:
> > >
> > >
> > > Le 11/04/2020 à 12:34, Krisztián Szűcs a écrit :
> > > > - test-conda-python-3.7-turbodbc-latest:
> > > > - test-conda-python-3.7-turbodbc-master:
> > > > The latest is important here, because the release would break the
> > > > interoperability with turbodbc.
> > > > I need feedback on this from Uwe.
> > >
> > > For transparency, this was diagnosed and two issues were filed for turbodbc:
> > > https://github.com/blue-yonder/turbodbc/issues/251
> > > https://github.com/blue-yonder/turbodbc/issues/250
> > >
> > > I'm not sure there's anything Arrow can do to restore compatibility. If
> > > we relax array validation to avoid the validation failure that turbodbc
> > > is getting, the erroneously-constructed Arrow array could lead to
> > > further errors or crashes later on.
> > Than we shouldn't consider the turbodbc integration tests as blocker.
> >
> > Thanks Antoine!
> > >
> > > Regards
> > >
> > > Antoine.
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
I'm not able to reproduce the test-conda-python-3.7-kartothek-master
failure locally with docker-compose, is that a flake or real?
On Sat, Apr 11, 2020 at 7:29 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> On Sat, Apr 11, 2020 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > Le 11/04/2020 à 12:34, Krisztián Szűcs a écrit :
> > > - test-conda-python-3.7-turbodbc-latest:
> > > - test-conda-python-3.7-turbodbc-master:
> > > The latest is important here, because the release would break the
> > > interoperability with turbodbc.
> > > I need feedback on this from Uwe.
> >
> > For transparency, this was diagnosed and two issues were filed for turbodbc:
> > https://github.com/blue-yonder/turbodbc/issues/251
> > https://github.com/blue-yonder/turbodbc/issues/250
> >
> > I'm not sure there's anything Arrow can do to restore compatibility. If
> > we relax array validation to avoid the validation failure that turbodbc
> > is getting, the erroneously-constructed Arrow array could lead to
> > further errors or crashes later on.
> Than we shouldn't consider the turbodbc integration tests as blocker.
>
> Thanks Antoine!
> >
> > Regards
> >
> > Antoine.
Re: Preparing for 0.17.0 Arrow release
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Sat, Apr 11, 2020 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> Le 11/04/2020 à 12:34, Krisztián Szűcs a écrit :
> > - test-conda-python-3.7-turbodbc-latest:
> > - test-conda-python-3.7-turbodbc-master:
> > The latest is important here, because the release would break the
> > interoperability with turbodbc.
> > I need feedback on this from Uwe.
>
> For transparency, this was diagnosed and two issues were filed for turbodbc:
> https://github.com/blue-yonder/turbodbc/issues/251
> https://github.com/blue-yonder/turbodbc/issues/250
>
> I'm not sure there's anything Arrow can do to restore compatibility. If
> we relax array validation to avoid the validation failure that turbodbc
> is getting, the erroneously-constructed Arrow array could lead to
> further errors or crashes later on.
Than we shouldn't consider the turbodbc integration tests as blocker.
Thanks Antoine!
>
> Regards
>
> Antoine.
Re: Preparing for 0.17.0 Arrow release
Posted by Antoine Pitrou <an...@python.org>.
Le 11/04/2020 à 12:34, Krisztián Szűcs a écrit :
> - test-conda-python-3.7-turbodbc-latest:
> - test-conda-python-3.7-turbodbc-master:
> The latest is important here, because the release would break the
> interoperability with turbodbc.
> I need feedback on this from Uwe.
For transparency, this was diagnosed and two issues were filed for turbodbc:
https://github.com/blue-yonder/turbodbc/issues/251
https://github.com/blue-yonder/turbodbc/issues/250
I'm not sure there's anything Arrow can do to restore compatibility. If
we relax array validation to avoid the validation failure that turbodbc
is getting, the erroneously-constructed Arrow array could lead to
further errors or crashes later on.
Regards
Antoine.
Re: Preparing for 0.17.0 Arrow release
Posted by Krisztián Szűcs <sz...@gmail.com>.
We'll receive three nightly reports per day until the release, see the
failing tasks there.
Failed Tasks:
- conda-osx-*:
- conda-win-*:
- wheel-osx-cp38:
- wheel-win-*:
Python dataset tests are failing, possible related to silently
ignoring paths when reading via the new ParquetDataset.
Not trivial to reproduce.
- test-conda-python-3.7-hdfs-2.9.2:
Unable to download hadoop from the apache mirror, probably not a
blocker but would be nice to see that hdfs tests are passing.
- test-conda-python-3.7-kartothek-master:
- test-conda-python-3.7-turbodbc-latest:
- test-conda-python-3.7-turbodbc-master:
The latest is important here, because the release would break the
interoperability with turbodbc.
I need feedback on this from Uwe.
On Fri, Apr 10, 2020 at 10:18 PM Wes McKinney <we...@gmail.com> wrote:
>
> None of the open issues look like they should prevent a release
> candidate from being cut.
>
> Fixing the nightly and packaging builds seems like the last remaining
> task, but some of them need to be tracked by JIRA issues. Can you
> write a list here of what's definitely currently broken?
>
> On Fri, Apr 10, 2020 at 1:35 PM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > We still have 12 open issues, about half of them are not essential.
> > The more pressing problem is the number of failing builds including
> > packaging builds (see the nightly reports).
> >
> > Releasing today doesn't look realistic, but if we're able to resolve
> > the issues over the weekend I can start the release procedure on
> > Monday.
> >
> > Thanks, Krisztian
> >
> > On Tue, Apr 7, 2020 at 4:31 AM Andy Grove <an...@gmail.com> wrote:
> > >
> > > There are two trivial Rust PRs pending that I would like to see merged for
> > > the release.
> > >
> > > ARROW-7794: [Rust] Support releasing arrow-flight
> > >
> > > https://github.com/apache/arrow/pull/6858
> > >
> > > ARROW-8357: [Rust] [DataFusion] Dockerfile for CLI is missing format dir
> > >
> > > https://github.com/apache/arrow/pull/6860
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > > On Mon, Apr 6, 2020 at 6:55 AM Antoine Pitrou <an...@python.org> wrote:
> > >
> > > >
> > > > Also nice to have perhaps (PR available and several back-and-forths
> > > > already):
> > > >
> > > > * ARROW-7610: [Java] Finish support for 64 bit int allocations
> > > >
> > > > Needs a Java committer to decide...
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > > Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> > > > > We are getting close to the 0.17.0 endgame.
> > > > >
> > > > > Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> > > > > issues without patches yet so we should decide quickly whether they
> > > > > need to be included. Are they any blocking issues not accounted for in
> > > > > the milestone?
> > > > >
> > > > > * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
> > > > >
> > > > > Patch available
> > > > >
> > > > > * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> > > > > relative path to Flight.proto
> > > > >
> > > > > No patch yet
> > > > >
> > > > > * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> > > > > documentation when updating website
> > > > >
> > > > > This issue needs to be addressed by the release manager and the
> > > > > Confluence instructions must be updated.
> > > > >
> > > > > * ARROW-7891 [C++] RecordBatch->Equals should also have a
> > > > > check_metadata argument
> > > > >
> > > > > Patch available that needs to be reviewed and approved
> > > > >
> > > > > * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> > > > schema
> > > > >
> > > > > Patch available, but failures to be resolved
> > > > >
> > > > > * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> > > > reuse
> > > > >
> > > > > Depends on ARROW-8164, will require rebase
> > > > >
> > > > > * ARROW-8039: [Python][Dataset] Support using dataset API in
> > > > > pyarrow.parquet with a minimal ParquetDataset shim
> > > > >
> > > > > Patch pending
> > > > >
> > > > > * ARROW-8047: [Python][Documentation] Document migration from
> > > > > ParquetDataset to pyarrow.datasets
> > > > >
> > > > > May be tackled beyond 0.17.0
> > > > >
> > > > > * ARROW-8063: [Python] Add user guide documentation for Datasets API
> > > > >
> > > > > May be tackled beyond 0.17.0
> > > > >
> > > > > * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
> > > > >
> > > > > Does not seem strictly necessary for release, since a packaging issue
> > > > >
> > > > > * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
> > > > >
> > > > > Patch available, but needs review. May
> > > > >
> > > > > * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> > > > > incorrect path gives confusing error message
> > > > >
> > > > > Nice to have, but not essential
> > > > >
> > > > > * ARROW-8266: [C++] Add backup mirrors for external project source
> > > > downloads
> > > > >
> > > > > Patch available, nice to have
> > > > >
> > > > > * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> > > > > per "Feather V2" changes
> > > > >
> > > > > Patch available
> > > > >
> > > > > * ARROW-8300 [R] Documentation and changelog updates for 0.17
> > > > >
> > > > > Patch available
> > > > >
> > > > > * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> > > > > requirements in C data interface
> > > > >
> > > > > Patch available
> > > > >
> > > > > * ARROW-8330: [Documentation] The post release script generates the
> > > > > documentation with a development version
> > > > >
> > > > > Patch available
> > > > >
> > > > > * ARROW-8335: [Release] Add crossbow jobs to run release verification
> > > > >
> > > > > Patch in progress
> > > > >
> > > > > On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> > > > >>
> > > > >> I see ARROW-6871 in the list.
> > > > >> It seems it has some bugs, which are being fixed by ARROW-8239.
> > > > >> So I have added ARROW-8239 to the list.
> > > > >>
> > > > >> The PR for ARROW-8239 is already approved, so it is expected to be
> > > > resolved
> > > > >> soon.
> > > > >>
> > > > >> Best,
> > > > >> Liya Fan
> > > > >>
> > > > >> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> I moved the Java issues out of 0.17.0, they seem complex enough or not
> > > > of
> > > > >>> enough significance to make them blockers for 0.17.0 release. If
> > > > owners of
> > > > >>> the issues disagree please move them back int.
> > > > >>>
> > > > >>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > >>>
> > > > >>>> We've made good progress, but there are still 35 issues in the
> > > > >>>> backlog. Some of them are documentation related, but there are some
> > > > >>>> functionality-related patches that could be at risk. If all could
> > > > >>>> review again to trim out anything that isn't going to make the cut for
> > > > >>>> 0.17.0, please do
> > > > >>>>
> > > > >>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>> I just took a first pass at reviewing the Java and Rust issues and
> > > > >>>> removed
> > > > >>>>> some from the 0.17.0 release. There are a few small Rust issues that
> > > > I
> > > > >>> am
> > > > >>>>> actively working on for this release.
> > > > >>>>>
> > > > >>>>> Thanks.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> hi Neal,
> > > > >>>>>>
> > > > >>>>>> Thanks for helping coordinate. I agree we should be in a position to
> > > > >>>>>> release sometime next week.
> > > > >>>>>>
> > > > >>>>>> Can folks from the Rust and Java side review issues in the backlog?
> > > > >>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
> > > > >>>>>> issues.
> > > > >>>>>>
> > > > >>>>>> Thanks
> > > > >>>>>>
> > > > >>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > > >>>>>> <ne...@gmail.com> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Hi all,
> > > > >>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
> > > > >>> a
> > > > >>>> 0.17
> > > > >>>>>>> release at the end of the month. Judging from
> > > > >>>>>>>
> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > > >>>>>> it
> > > > >>>>>>> looks like we're getting closer.
> > > > >>>>>>>
> > > > >>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
> > > > >>>> 0.17
> > > > >>>>>>> scope any tickets they don't plan to finish this week, and (2) if
> > > > >>>> there
> > > > >>>>>> are
> > > > >>>>>>> any issues that should block release, make sure they are flagged as
> > > > >>>>>>> "blockers".
> > > > >>>>>>>
> > > > >>>>>>> Neal
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
> > > > >>>>>>>> sooner rather than doing a patch release, since releases in
> > > > >>> general
> > > > >>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
> > > > >>> has
> > > > >>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
> > > > >>>> think
> > > > >>>>>>>> we can be in a position to release by the week of March 23 or the
> > > > >>>> week
> > > > >>>>>>>> of March 30?
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> > > > >>>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>> If people are generally on board with accelerating a 0.17.0
> > > > >>> major
> > > > >>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > >>>>>>>>> beginning to do issue gardening to whittle things down to
> > > > >>>>>>>>> critical-looking bugs and high probability patches for the next
> > > > >>>>>> couple
> > > > >>>>>>>>> of weeks.
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > > > >>>> wesmckinn@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I recall there are some other issues that have been reported
> > > > >>> or
> > > > >>>>>> fixed
> > > > >>>>>>>>>> that are critical and not yet marked with 0.16.1.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > >>>>>>>>>> <ne...@gmail.com> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
> > > > >>>> instead of
> > > > >>>>>> a
> > > > >>>>>>>> patch
> > > > >>>>>>>>>>> release.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Neal
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > >>>>>>>> neal.p.richardson@gmail.com>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
> > > > >>>> it's
> > > > >>>>>> not
> > > > >>>>>>>> clear
> > > > >>>>>>>>>>>> to me that it's worth the diversion from other priorities
> > > > >>>> to
> > > > >>>>>> make
> > > > >>>>>>>> a release
> > > > >>>>>>>>>>>> right now. Nothing on
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > >>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
> > > > >>>>>> critical?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> If we did decide to go forward, would it be possible to
> > > > >>> do
> > > > >>>> a
> > > > >>>>>>>> release that
> > > > >>>>>>>>>>>> is limited to the affected implementations (say, do a
> > > > >>>>>> Python-only
> > > > >>>>>>>> release)?
> > > > >>>>>>>>>>>> That might reduce the cost of building and verifying
> > > > >>>> enough to
> > > > >>>>>>>> make it
> > > > >>>>>>>>>>>> reasonable to consider.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Neal
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > >>>>>>>> szucs.krisztian@gmail.com>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > > >>>>>> wesmckinn@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> hi folks,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> There have been a number of critical issues reported
> > > > >>>> (many
> > > > >>>>>> of
> > > > >>>>>>>> them
> > > > >>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
> > > > >>>> interest
> > > > >>>>>> in
> > > > >>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
> > > > >>>> patches
> > > > >>>>>> onto a
> > > > >>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
> > > > >>> major
> > > > >>>>>>>> release is a
> > > > >>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
> > > > >>>> Krisztian
> > > > >>>>>> wrote
> > > > >>>>>>>> get
> > > > >>>>>>>>>>>>>> contributed as a PR?
> > > > >>>>>>>>>>>>> Not yet, but it is available at
> > > > >>>>>>>>>>>>>
> > > > >>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>> Wes
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > >
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
None of the open issues look like they should prevent a release
candidate from being cut.
Fixing the nightly and packaging builds seems like the last remaining
task, but some of them need to be tracked by JIRA issues. Can you
write a list here of what's definitely currently broken?
On Fri, Apr 10, 2020 at 1:35 PM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> We still have 12 open issues, about half of them are not essential.
> The more pressing problem is the number of failing builds including
> packaging builds (see the nightly reports).
>
> Releasing today doesn't look realistic, but if we're able to resolve
> the issues over the weekend I can start the release procedure on
> Monday.
>
> Thanks, Krisztian
>
> On Tue, Apr 7, 2020 at 4:31 AM Andy Grove <an...@gmail.com> wrote:
> >
> > There are two trivial Rust PRs pending that I would like to see merged for
> > the release.
> >
> > ARROW-7794: [Rust] Support releasing arrow-flight
> >
> > https://github.com/apache/arrow/pull/6858
> >
> > ARROW-8357: [Rust] [DataFusion] Dockerfile for CLI is missing format dir
> >
> > https://github.com/apache/arrow/pull/6860
> >
> > Thanks,
> >
> > Andy.
> >
> >
> > On Mon, Apr 6, 2020 at 6:55 AM Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Also nice to have perhaps (PR available and several back-and-forths
> > > already):
> > >
> > > * ARROW-7610: [Java] Finish support for 64 bit int allocations
> > >
> > > Needs a Java committer to decide...
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> > > > We are getting close to the 0.17.0 endgame.
> > > >
> > > > Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> > > > issues without patches yet so we should decide quickly whether they
> > > > need to be included. Are they any blocking issues not accounted for in
> > > > the milestone?
> > > >
> > > > * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
> > > >
> > > > Patch available
> > > >
> > > > * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> > > > relative path to Flight.proto
> > > >
> > > > No patch yet
> > > >
> > > > * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> > > > documentation when updating website
> > > >
> > > > This issue needs to be addressed by the release manager and the
> > > > Confluence instructions must be updated.
> > > >
> > > > * ARROW-7891 [C++] RecordBatch->Equals should also have a
> > > > check_metadata argument
> > > >
> > > > Patch available that needs to be reviewed and approved
> > > >
> > > > * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> > > schema
> > > >
> > > > Patch available, but failures to be resolved
> > > >
> > > > * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> > > reuse
> > > >
> > > > Depends on ARROW-8164, will require rebase
> > > >
> > > > * ARROW-8039: [Python][Dataset] Support using dataset API in
> > > > pyarrow.parquet with a minimal ParquetDataset shim
> > > >
> > > > Patch pending
> > > >
> > > > * ARROW-8047: [Python][Documentation] Document migration from
> > > > ParquetDataset to pyarrow.datasets
> > > >
> > > > May be tackled beyond 0.17.0
> > > >
> > > > * ARROW-8063: [Python] Add user guide documentation for Datasets API
> > > >
> > > > May be tackled beyond 0.17.0
> > > >
> > > > * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
> > > >
> > > > Does not seem strictly necessary for release, since a packaging issue
> > > >
> > > > * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
> > > >
> > > > Patch available, but needs review. May
> > > >
> > > > * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> > > > incorrect path gives confusing error message
> > > >
> > > > Nice to have, but not essential
> > > >
> > > > * ARROW-8266: [C++] Add backup mirrors for external project source
> > > downloads
> > > >
> > > > Patch available, nice to have
> > > >
> > > > * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> > > > per "Feather V2" changes
> > > >
> > > > Patch available
> > > >
> > > > * ARROW-8300 [R] Documentation and changelog updates for 0.17
> > > >
> > > > Patch available
> > > >
> > > > * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> > > > requirements in C data interface
> > > >
> > > > Patch available
> > > >
> > > > * ARROW-8330: [Documentation] The post release script generates the
> > > > documentation with a development version
> > > >
> > > > Patch available
> > > >
> > > > * ARROW-8335: [Release] Add crossbow jobs to run release verification
> > > >
> > > > Patch in progress
> > > >
> > > > On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> > > >>
> > > >> I see ARROW-6871 in the list.
> > > >> It seems it has some bugs, which are being fixed by ARROW-8239.
> > > >> So I have added ARROW-8239 to the list.
> > > >>
> > > >> The PR for ARROW-8239 is already approved, so it is expected to be
> > > resolved
> > > >> soon.
> > > >>
> > > >> Best,
> > > >> Liya Fan
> > > >>
> > > >> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I moved the Java issues out of 0.17.0, they seem complex enough or not
> > > of
> > > >>> enough significance to make them blockers for 0.17.0 release. If
> > > owners of
> > > >>> the issues disagree please move them back int.
> > > >>>
> > > >>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >>>
> > > >>>> We've made good progress, but there are still 35 issues in the
> > > >>>> backlog. Some of them are documentation related, but there are some
> > > >>>> functionality-related patches that could be at risk. If all could
> > > >>>> review again to trim out anything that isn't going to make the cut for
> > > >>>> 0.17.0, please do
> > > >>>>
> > > >>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> > > >>> wrote:
> > > >>>>>
> > > >>>>> I just took a first pass at reviewing the Java and Rust issues and
> > > >>>> removed
> > > >>>>> some from the 0.17.0 release. There are a few small Rust issues that
> > > I
> > > >>> am
> > > >>>>> actively working on for this release.
> > > >>>>>
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> hi Neal,
> > > >>>>>>
> > > >>>>>> Thanks for helping coordinate. I agree we should be in a position to
> > > >>>>>> release sometime next week.
> > > >>>>>>
> > > >>>>>> Can folks from the Rust and Java side review issues in the backlog?
> > > >>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
> > > >>>>>> issues.
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>
> > > >>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > >>>>>> <ne...@gmail.com> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi all,
> > > >>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
> > > >>> a
> > > >>>> 0.17
> > > >>>>>>> release at the end of the month. Judging from
> > > >>>>>>>
> > > >>>>
> > > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > >>>>>> it
> > > >>>>>>> looks like we're getting closer.
> > > >>>>>>>
> > > >>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
> > > >>>> 0.17
> > > >>>>>>> scope any tickets they don't plan to finish this week, and (2) if
> > > >>>> there
> > > >>>>>> are
> > > >>>>>>> any issues that should block release, make sure they are flagged as
> > > >>>>>>> "blockers".
> > > >>>>>>>
> > > >>>>>>> Neal
> > > >>>>>>>
> > > >>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
> > > >>>>>>>> sooner rather than doing a patch release, since releases in
> > > >>> general
> > > >>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
> > > >>> has
> > > >>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
> > > >>>> think
> > > >>>>>>>> we can be in a position to release by the week of March 23 or the
> > > >>>> week
> > > >>>>>>>> of March 30?
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> > > >>>>
> > > >>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> If people are generally on board with accelerating a 0.17.0
> > > >>> major
> > > >>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > >>>>>>>>> beginning to do issue gardening to whittle things down to
> > > >>>>>>>>> critical-looking bugs and high probability patches for the next
> > > >>>>>> couple
> > > >>>>>>>>> of weeks.
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > > >>>> wesmckinn@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> I recall there are some other issues that have been reported
> > > >>> or
> > > >>>>>> fixed
> > > >>>>>>>>>> that are critical and not yet marked with 0.16.1.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > >>>>>>>>>> <ne...@gmail.com> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
> > > >>>> instead of
> > > >>>>>> a
> > > >>>>>>>> patch
> > > >>>>>>>>>>> release.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Neal
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > >>>>>>>> neal.p.richardson@gmail.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
> > > >>>> it's
> > > >>>>>> not
> > > >>>>>>>> clear
> > > >>>>>>>>>>>> to me that it's worth the diversion from other priorities
> > > >>>> to
> > > >>>>>> make
> > > >>>>>>>> a release
> > > >>>>>>>>>>>> right now. Nothing on
> > > >>>>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > >>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
> > > >>>>>> critical?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> If we did decide to go forward, would it be possible to
> > > >>> do
> > > >>>> a
> > > >>>>>>>> release that
> > > >>>>>>>>>>>> is limited to the affected implementations (say, do a
> > > >>>>>> Python-only
> > > >>>>>>>> release)?
> > > >>>>>>>>>>>> That might reduce the cost of building and verifying
> > > >>>> enough to
> > > >>>>>>>> make it
> > > >>>>>>>>>>>> reasonable to consider.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Neal
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > >>>>>>>> szucs.krisztian@gmail.com>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > >>>>>> wesmckinn@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> hi folks,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> There have been a number of critical issues reported
> > > >>>> (many
> > > >>>>>> of
> > > >>>>>>>> them
> > > >>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
> > > >>>> interest
> > > >>>>>> in
> > > >>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
> > > >>>> patches
> > > >>>>>> onto a
> > > >>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
> > > >>> major
> > > >>>>>>>> release is a
> > > >>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
> > > >>>> Krisztian
> > > >>>>>> wrote
> > > >>>>>>>> get
> > > >>>>>>>>>>>>>> contributed as a PR?
> > > >>>>>>>>>>>>> Not yet, but it is available at
> > > >>>>>>>>>>>>>
> > > >>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>> Wes
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > >
Re: Preparing for 0.17.0 Arrow release
Posted by Krisztián Szűcs <sz...@gmail.com>.
We still have 12 open issues, about half of them are not essential.
The more pressing problem is the number of failing builds including
packaging builds (see the nightly reports).
Releasing today doesn't look realistic, but if we're able to resolve
the issues over the weekend I can start the release procedure on
Monday.
Thanks, Krisztian
On Tue, Apr 7, 2020 at 4:31 AM Andy Grove <an...@gmail.com> wrote:
>
> There are two trivial Rust PRs pending that I would like to see merged for
> the release.
>
> ARROW-7794: [Rust] Support releasing arrow-flight
>
> https://github.com/apache/arrow/pull/6858
>
> ARROW-8357: [Rust] [DataFusion] Dockerfile for CLI is missing format dir
>
> https://github.com/apache/arrow/pull/6860
>
> Thanks,
>
> Andy.
>
>
> On Mon, Apr 6, 2020 at 6:55 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Also nice to have perhaps (PR available and several back-and-forths
> > already):
> >
> > * ARROW-7610: [Java] Finish support for 64 bit int allocations
> >
> > Needs a Java committer to decide...
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> > > We are getting close to the 0.17.0 endgame.
> > >
> > > Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> > > issues without patches yet so we should decide quickly whether they
> > > need to be included. Are they any blocking issues not accounted for in
> > > the milestone?
> > >
> > > * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
> > >
> > > Patch available
> > >
> > > * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> > > relative path to Flight.proto
> > >
> > > No patch yet
> > >
> > > * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> > > documentation when updating website
> > >
> > > This issue needs to be addressed by the release manager and the
> > > Confluence instructions must be updated.
> > >
> > > * ARROW-7891 [C++] RecordBatch->Equals should also have a
> > > check_metadata argument
> > >
> > > Patch available that needs to be reviewed and approved
> > >
> > > * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> > schema
> > >
> > > Patch available, but failures to be resolved
> > >
> > > * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> > reuse
> > >
> > > Depends on ARROW-8164, will require rebase
> > >
> > > * ARROW-8039: [Python][Dataset] Support using dataset API in
> > > pyarrow.parquet with a minimal ParquetDataset shim
> > >
> > > Patch pending
> > >
> > > * ARROW-8047: [Python][Documentation] Document migration from
> > > ParquetDataset to pyarrow.datasets
> > >
> > > May be tackled beyond 0.17.0
> > >
> > > * ARROW-8063: [Python] Add user guide documentation for Datasets API
> > >
> > > May be tackled beyond 0.17.0
> > >
> > > * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
> > >
> > > Does not seem strictly necessary for release, since a packaging issue
> > >
> > > * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
> > >
> > > Patch available, but needs review. May
> > >
> > > * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> > > incorrect path gives confusing error message
> > >
> > > Nice to have, but not essential
> > >
> > > * ARROW-8266: [C++] Add backup mirrors for external project source
> > downloads
> > >
> > > Patch available, nice to have
> > >
> > > * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> > > per "Feather V2" changes
> > >
> > > Patch available
> > >
> > > * ARROW-8300 [R] Documentation and changelog updates for 0.17
> > >
> > > Patch available
> > >
> > > * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> > > requirements in C data interface
> > >
> > > Patch available
> > >
> > > * ARROW-8330: [Documentation] The post release script generates the
> > > documentation with a development version
> > >
> > > Patch available
> > >
> > > * ARROW-8335: [Release] Add crossbow jobs to run release verification
> > >
> > > Patch in progress
> > >
> > > On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> > >>
> > >> I see ARROW-6871 in the list.
> > >> It seems it has some bugs, which are being fixed by ARROW-8239.
> > >> So I have added ARROW-8239 to the list.
> > >>
> > >> The PR for ARROW-8239 is already approved, so it is expected to be
> > resolved
> > >> soon.
> > >>
> > >> Best,
> > >> Liya Fan
> > >>
> > >> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> > >> wrote:
> > >>
> > >>> I moved the Java issues out of 0.17.0, they seem complex enough or not
> > of
> > >>> enough significance to make them blockers for 0.17.0 release. If
> > owners of
> > >>> the issues disagree please move them back int.
> > >>>
> > >>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>>
> > >>>> We've made good progress, but there are still 35 issues in the
> > >>>> backlog. Some of them are documentation related, but there are some
> > >>>> functionality-related patches that could be at risk. If all could
> > >>>> review again to trim out anything that isn't going to make the cut for
> > >>>> 0.17.0, please do
> > >>>>
> > >>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>> I just took a first pass at reviewing the Java and Rust issues and
> > >>>> removed
> > >>>>> some from the 0.17.0 release. There are a few small Rust issues that
> > I
> > >>> am
> > >>>>> actively working on for this release.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> hi Neal,
> > >>>>>>
> > >>>>>> Thanks for helping coordinate. I agree we should be in a position to
> > >>>>>> release sometime next week.
> > >>>>>>
> > >>>>>> Can folks from the Rust and Java side review issues in the backlog?
> > >>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
> > >>>>>> issues.
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>>
> > >>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > >>>>>> <ne...@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>> Hi all,
> > >>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
> > >>> a
> > >>>> 0.17
> > >>>>>>> release at the end of the month. Judging from
> > >>>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > >>>>>> it
> > >>>>>>> looks like we're getting closer.
> > >>>>>>>
> > >>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
> > >>>> 0.17
> > >>>>>>> scope any tickets they don't plan to finish this week, and (2) if
> > >>>> there
> > >>>>>> are
> > >>>>>>> any issues that should block release, make sure they are flagged as
> > >>>>>>> "blockers".
> > >>>>>>>
> > >>>>>>> Neal
> > >>>>>>>
> > >>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
> > >>>>>>>> sooner rather than doing a patch release, since releases in
> > >>> general
> > >>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
> > >>> has
> > >>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
> > >>>> think
> > >>>>>>>> we can be in a position to release by the week of March 23 or the
> > >>>> week
> > >>>>>>>> of March 30?
> > >>>>>>>>
> > >>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> > >>>>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> If people are generally on board with accelerating a 0.17.0
> > >>> major
> > >>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > >>>>>>>>> beginning to do issue gardening to whittle things down to
> > >>>>>>>>> critical-looking bugs and high probability patches for the next
> > >>>>>> couple
> > >>>>>>>>> of weeks.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > >>>> wesmckinn@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> I recall there are some other issues that have been reported
> > >>> or
> > >>>>>> fixed
> > >>>>>>>>>> that are critical and not yet marked with 0.16.1.
> > >>>>>>>>>>
> > >>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > >>>>>>>>>> <ne...@gmail.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
> > >>>> instead of
> > >>>>>> a
> > >>>>>>>> patch
> > >>>>>>>>>>> release.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Neal
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > >>>>>>>> neal.p.richardson@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
> > >>>> it's
> > >>>>>> not
> > >>>>>>>> clear
> > >>>>>>>>>>>> to me that it's worth the diversion from other priorities
> > >>>> to
> > >>>>>> make
> > >>>>>>>> a release
> > >>>>>>>>>>>> right now. Nothing on
> > >>>>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > >>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
> > >>>>>> critical?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If we did decide to go forward, would it be possible to
> > >>> do
> > >>>> a
> > >>>>>>>> release that
> > >>>>>>>>>>>> is limited to the affected implementations (say, do a
> > >>>>>> Python-only
> > >>>>>>>> release)?
> > >>>>>>>>>>>> That might reduce the cost of building and verifying
> > >>>> enough to
> > >>>>>>>> make it
> > >>>>>>>>>>>> reasonable to consider.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Neal
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > >>>>>>>> szucs.krisztian@gmail.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > >>>>>> wesmckinn@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> hi folks,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> There have been a number of critical issues reported
> > >>>> (many
> > >>>>>> of
> > >>>>>>>> them
> > >>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
> > >>>> interest
> > >>>>>> in
> > >>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
> > >>>> patches
> > >>>>>> onto a
> > >>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
> > >>> major
> > >>>>>>>> release is a
> > >>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
> > >>>> Krisztian
> > >>>>>> wrote
> > >>>>>>>> get
> > >>>>>>>>>>>>>> contributed as a PR?
> > >>>>>>>>>>>>> Not yet, but it is available at
> > >>>>>>>>>>>>>
> > >>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>> Wes
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
Re: Preparing for 0.17.0 Arrow release
Posted by Andy Grove <an...@gmail.com>.
There are two trivial Rust PRs pending that I would like to see merged for
the release.
ARROW-7794: [Rust] Support releasing arrow-flight
https://github.com/apache/arrow/pull/6858
ARROW-8357: [Rust] [DataFusion] Dockerfile for CLI is missing format dir
https://github.com/apache/arrow/pull/6860
Thanks,
Andy.
On Mon, Apr 6, 2020 at 6:55 AM Antoine Pitrou <an...@python.org> wrote:
>
> Also nice to have perhaps (PR available and several back-and-forths
> already):
>
> * ARROW-7610: [Java] Finish support for 64 bit int allocations
>
> Needs a Java committer to decide...
>
> Regards
>
> Antoine.
>
>
> Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> > We are getting close to the 0.17.0 endgame.
> >
> > Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> > issues without patches yet so we should decide quickly whether they
> > need to be included. Are they any blocking issues not accounted for in
> > the milestone?
> >
> > * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
> >
> > Patch available
> >
> > * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> > relative path to Flight.proto
> >
> > No patch yet
> >
> > * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> > documentation when updating website
> >
> > This issue needs to be addressed by the release manager and the
> > Confluence instructions must be updated.
> >
> > * ARROW-7891 [C++] RecordBatch->Equals should also have a
> > check_metadata argument
> >
> > Patch available that needs to be reviewed and approved
> >
> > * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> schema
> >
> > Patch available, but failures to be resolved
> >
> > * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> reuse
> >
> > Depends on ARROW-8164, will require rebase
> >
> > * ARROW-8039: [Python][Dataset] Support using dataset API in
> > pyarrow.parquet with a minimal ParquetDataset shim
> >
> > Patch pending
> >
> > * ARROW-8047: [Python][Documentation] Document migration from
> > ParquetDataset to pyarrow.datasets
> >
> > May be tackled beyond 0.17.0
> >
> > * ARROW-8063: [Python] Add user guide documentation for Datasets API
> >
> > May be tackled beyond 0.17.0
> >
> > * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
> >
> > Does not seem strictly necessary for release, since a packaging issue
> >
> > * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
> >
> > Patch available, but needs review. May
> >
> > * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> > incorrect path gives confusing error message
> >
> > Nice to have, but not essential
> >
> > * ARROW-8266: [C++] Add backup mirrors for external project source
> downloads
> >
> > Patch available, nice to have
> >
> > * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> > per "Feather V2" changes
> >
> > Patch available
> >
> > * ARROW-8300 [R] Documentation and changelog updates for 0.17
> >
> > Patch available
> >
> > * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> > requirements in C data interface
> >
> > Patch available
> >
> > * ARROW-8330: [Documentation] The post release script generates the
> > documentation with a development version
> >
> > Patch available
> >
> > * ARROW-8335: [Release] Add crossbow jobs to run release verification
> >
> > Patch in progress
> >
> > On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> >>
> >> I see ARROW-6871 in the list.
> >> It seems it has some bugs, which are being fixed by ARROW-8239.
> >> So I have added ARROW-8239 to the list.
> >>
> >> The PR for ARROW-8239 is already approved, so it is expected to be
> resolved
> >> soon.
> >>
> >> Best,
> >> Liya Fan
> >>
> >> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> >> wrote:
> >>
> >>> I moved the Java issues out of 0.17.0, they seem complex enough or not
> of
> >>> enough significance to make them blockers for 0.17.0 release. If
> owners of
> >>> the issues disagree please move them back int.
> >>>
> >>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> wrote:
> >>>
> >>>> We've made good progress, but there are still 35 issues in the
> >>>> backlog. Some of them are documentation related, but there are some
> >>>> functionality-related patches that could be at risk. If all could
> >>>> review again to trim out anything that isn't going to make the cut for
> >>>> 0.17.0, please do
> >>>>
> >>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> I just took a first pass at reviewing the Java and Rust issues and
> >>>> removed
> >>>>> some from the 0.17.0 release. There are a few small Rust issues that
> I
> >>> am
> >>>>> actively working on for this release.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> hi Neal,
> >>>>>>
> >>>>>> Thanks for helping coordinate. I agree we should be in a position to
> >>>>>> release sometime next week.
> >>>>>>
> >>>>>> Can folks from the Rust and Java side review issues in the backlog?
> >>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
> >>>>>> issues.
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> >>>>>> <ne...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
> >>> a
> >>>> 0.17
> >>>>>>> release at the end of the month. Judging from
> >>>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> >>>>>> it
> >>>>>>> looks like we're getting closer.
> >>>>>>>
> >>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
> >>>> 0.17
> >>>>>>> scope any tickets they don't plan to finish this week, and (2) if
> >>>> there
> >>>>>> are
> >>>>>>> any issues that should block release, make sure they are flagged as
> >>>>>>> "blockers".
> >>>>>>>
> >>>>>>> Neal
> >>>>>>>
> >>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
> >>>>>>>> sooner rather than doing a patch release, since releases in
> >>> general
> >>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
> >>> has
> >>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
> >>>> think
> >>>>>>>> we can be in a position to release by the week of March 23 or the
> >>>> week
> >>>>>>>> of March 30?
> >>>>>>>>
> >>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> If people are generally on board with accelerating a 0.17.0
> >>> major
> >>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
> >>>>>>>>> beginning to do issue gardening to whittle things down to
> >>>>>>>>> critical-looking bugs and high probability patches for the next
> >>>>>> couple
> >>>>>>>>> of weeks.
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> >>>> wesmckinn@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I recall there are some other issues that have been reported
> >>> or
> >>>>>> fixed
> >>>>>>>>>> that are critical and not yet marked with 0.16.1.
> >>>>>>>>>>
> >>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> >>>>>>>>>> <ne...@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
> >>>> instead of
> >>>>>> a
> >>>>>>>> patch
> >>>>>>>>>>> release.
> >>>>>>>>>>>
> >>>>>>>>>>> Neal
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> >>>>>>>> neal.p.richardson@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
> >>>> it's
> >>>>>> not
> >>>>>>>> clear
> >>>>>>>>>>>> to me that it's worth the diversion from other priorities
> >>>> to
> >>>>>> make
> >>>>>>>> a release
> >>>>>>>>>>>> right now. Nothing on
> >>>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> >>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
> >>>>>> critical?
> >>>>>>>>>>>>
> >>>>>>>>>>>> If we did decide to go forward, would it be possible to
> >>> do
> >>>> a
> >>>>>>>> release that
> >>>>>>>>>>>> is limited to the affected implementations (say, do a
> >>>>>> Python-only
> >>>>>>>> release)?
> >>>>>>>>>>>> That might reduce the cost of building and verifying
> >>>> enough to
> >>>>>>>> make it
> >>>>>>>>>>>> reasonable to consider.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Neal
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> >>>>>>>> szucs.krisztian@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> >>>>>> wesmckinn@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> hi folks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There have been a number of critical issues reported
> >>>> (many
> >>>>>> of
> >>>>>>>> them
> >>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
> >>>> interest
> >>>>>> in
> >>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
> >>>> patches
> >>>>>> onto a
> >>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
> >>> major
> >>>>>>>> release is a
> >>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
> >>>> Krisztian
> >>>>>> wrote
> >>>>>>>> get
> >>>>>>>>>>>>>> contributed as a PR?
> >>>>>>>>>>>>> Not yet, but it is available at
> >>>>>>>>>>>>>
> >>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>> Wes
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
>
Re: Preparing for 0.17.0 Arrow release
Posted by Antoine Pitrou <an...@python.org>.
Also nice to have perhaps (PR available and several back-and-forths
already):
* ARROW-7610: [Java] Finish support for 64 bit int allocations
Needs a Java committer to decide...
Regards
Antoine.
Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> We are getting close to the 0.17.0 endgame.
>
> Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> issues without patches yet so we should decide quickly whether they
> need to be included. Are they any blocking issues not accounted for in
> the milestone?
>
> * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
>
> Patch available
>
> * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> relative path to Flight.proto
>
> No patch yet
>
> * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> documentation when updating website
>
> This issue needs to be addressed by the release manager and the
> Confluence instructions must be updated.
>
> * ARROW-7891 [C++] RecordBatch->Equals should also have a
> check_metadata argument
>
> Patch available that needs to be reviewed and approved
>
> * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical schema
>
> Patch available, but failures to be resolved
>
> * ARROW-7965: [Python] Hold a reference to the dataset factory for later reuse
>
> Depends on ARROW-8164, will require rebase
>
> * ARROW-8039: [Python][Dataset] Support using dataset API in
> pyarrow.parquet with a minimal ParquetDataset shim
>
> Patch pending
>
> * ARROW-8047: [Python][Documentation] Document migration from
> ParquetDataset to pyarrow.datasets
>
> May be tackled beyond 0.17.0
>
> * ARROW-8063: [Python] Add user guide documentation for Datasets API
>
> May be tackled beyond 0.17.0
>
> * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
>
> Does not seem strictly necessary for release, since a packaging issue
>
> * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
>
> Patch available, but needs review. May
>
> * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> incorrect path gives confusing error message
>
> Nice to have, but not essential
>
> * ARROW-8266: [C++] Add backup mirrors for external project source downloads
>
> Patch available, nice to have
>
> * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> per "Feather V2" changes
>
> Patch available
>
> * ARROW-8300 [R] Documentation and changelog updates for 0.17
>
> Patch available
>
> * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> requirements in C data interface
>
> Patch available
>
> * ARROW-8330: [Documentation] The post release script generates the
> documentation with a development version
>
> Patch available
>
> * ARROW-8335: [Release] Add crossbow jobs to run release verification
>
> Patch in progress
>
> On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
>>
>> I see ARROW-6871 in the list.
>> It seems it has some bugs, which are being fixed by ARROW-8239.
>> So I have added ARROW-8239 to the list.
>>
>> The PR for ARROW-8239 is already approved, so it is expected to be resolved
>> soon.
>>
>> Best,
>> Liya Fan
>>
>> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> I moved the Java issues out of 0.17.0, they seem complex enough or not of
>>> enough significance to make them blockers for 0.17.0 release. If owners of
>>> the issues disagree please move them back int.
>>>
>>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com> wrote:
>>>
>>>> We've made good progress, but there are still 35 issues in the
>>>> backlog. Some of them are documentation related, but there are some
>>>> functionality-related patches that could be at risk. If all could
>>>> review again to trim out anything that isn't going to make the cut for
>>>> 0.17.0, please do
>>>>
>>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
>>> wrote:
>>>>>
>>>>> I just took a first pass at reviewing the Java and Rust issues and
>>>> removed
>>>>> some from the 0.17.0 release. There are a few small Rust issues that I
>>> am
>>>>> actively working on for this release.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> hi Neal,
>>>>>>
>>>>>> Thanks for helping coordinate. I agree we should be in a position to
>>>>>> release sometime next week.
>>>>>>
>>>>>> Can folks from the Rust and Java side review issues in the backlog?
>>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
>>>>>> issues.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
>>>>>> <ne...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
>>> a
>>>> 0.17
>>>>>>> release at the end of the month. Judging from
>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
>>>>>> it
>>>>>>> looks like we're getting closer.
>>>>>>>
>>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
>>>> 0.17
>>>>>>> scope any tickets they don't plan to finish this week, and (2) if
>>>> there
>>>>>> are
>>>>>>> any issues that should block release, make sure they are flagged as
>>>>>>> "blockers".
>>>>>>>
>>>>>>> Neal
>>>>>>>
>>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
>>>>>>>> sooner rather than doing a patch release, since releases in
>>> general
>>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
>>> has
>>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
>>>> think
>>>>>>>> we can be in a position to release by the week of March 23 or the
>>>> week
>>>>>>>> of March 30?
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> If people are generally on board with accelerating a 0.17.0
>>> major
>>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
>>>>>>>>> beginning to do issue gardening to whittle things down to
>>>>>>>>> critical-looking bugs and high probability patches for the next
>>>>>> couple
>>>>>>>>> of weeks.
>>>>>>>>>
>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
>>>> wesmckinn@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I recall there are some other issues that have been reported
>>> or
>>>>>> fixed
>>>>>>>>>> that are critical and not yet marked with 0.16.1.
>>>>>>>>>>
>>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
>>>>>>>>>> <ne...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
>>>> instead of
>>>>>> a
>>>>>>>> patch
>>>>>>>>>>> release.
>>>>>>>>>>>
>>>>>>>>>>> Neal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
>>>>>>>> neal.p.richardson@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
>>>> it's
>>>>>> not
>>>>>>>> clear
>>>>>>>>>>>> to me that it's worth the diversion from other priorities
>>>> to
>>>>>> make
>>>>>>>> a release
>>>>>>>>>>>> right now. Nothing on
>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
>>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
>>>>>> critical?
>>>>>>>>>>>>
>>>>>>>>>>>> If we did decide to go forward, would it be possible to
>>> do
>>>> a
>>>>>>>> release that
>>>>>>>>>>>> is limited to the affected implementations (say, do a
>>>>>> Python-only
>>>>>>>> release)?
>>>>>>>>>>>> That might reduce the cost of building and verifying
>>>> enough to
>>>>>>>> make it
>>>>>>>>>>>> reasonable to consider.
>>>>>>>>>>>>
>>>>>>>>>>>> Neal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
>>>>>>>> szucs.krisztian@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
>>>>>> wesmckinn@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There have been a number of critical issues reported
>>>> (many
>>>>>> of
>>>>>>>> them
>>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
>>>> interest
>>>>>> in
>>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
>>>> patches
>>>>>> onto a
>>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
>>> major
>>>>>>>> release is a
>>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
>>>> Krisztian
>>>>>> wrote
>>>>>>>> get
>>>>>>>>>>>>>> contributed as a PR?
>>>>>>>>>>>>> Not yet, but it is available at
>>>>>>>>>>>>>
>>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
That may be so. If we do partially revert it (the dict return value is the
only thing probably that needs to be changed), we need to get the
downstream libraries to make changes to allow us to make this change.
Another option is returning the KV wrapper via another attribute.
On Mon, Apr 6, 2020, 3:06 AM Antoine Pitrou <an...@python.org> wrote:
>
> Hmm, if downstream libraries were expecting a dict, perhaps we'll need
> to revert that change...
>
> Regards
>
> Antoine.
>
>
> Le 06/04/2020 à 08:50, Joris Van den Bossche a écrit :
> > We also have a recent regression related to the KeyValueMetadata wrapping
> > python that is causing failures in downstream libraries, that seems a
> > blocker for the release:
> https://issues.apache.org/jira/browse/ARROW-8342
> >
> > On Mon, 6 Apr 2020 at 00:25, Wes McKinney <we...@gmail.com> wrote:
> >
> >> We are getting close to the 0.17.0 endgame.
> >>
> >> Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> >> issues without patches yet so we should decide quickly whether they
> >> need to be included. Are they any blocking issues not accounted for in
> >> the milestone?
> >>
> >> * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
> >>
> >> Patch available
> >>
> >> * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> >> relative path to Flight.proto
> >>
> >> No patch yet
> >>
> >> * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> >> documentation when updating website
> >>
> >> This issue needs to be addressed by the release manager and the
> >> Confluence instructions must be updated.
> >>
> >> * ARROW-7891 [C++] RecordBatch->Equals should also have a
> >> check_metadata argument
> >>
> >> Patch available that needs to be reviewed and approved
> >>
> >> * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> >> schema
> >>
> >> Patch available, but failures to be resolved
> >>
> >> * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> >> reuse
> >>
> >> Depends on ARROW-8164, will require rebase
> >>
> >> * ARROW-8039: [Python][Dataset] Support using dataset API in
> >> pyarrow.parquet with a minimal ParquetDataset shim
> >>
> >> Patch pending
> >>
> >> * ARROW-8047: [Python][Documentation] Document migration from
> >> ParquetDataset to pyarrow.datasets
> >>
> >> May be tackled beyond 0.17.0
> >>
> >> * ARROW-8063: [Python] Add user guide documentation for Datasets API
> >>
> >> May be tackled beyond 0.17.0
> >>
> >> * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
> >>
> >> Does not seem strictly necessary for release, since a packaging issue
> >>
> >> * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
> >>
> >> Patch available, but needs review. May
> >>
> >> * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> >> incorrect path gives confusing error message
> >>
> >> Nice to have, but not essential
> >>
> >> * ARROW-8266: [C++] Add backup mirrors for external project source
> >> downloads
> >>
> >> Patch available, nice to have
> >>
> >> * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> >> per "Feather V2" changes
> >>
> >> Patch available
> >>
> >> * ARROW-8300 [R] Documentation and changelog updates for 0.17
> >>
> >> Patch available
> >>
> >> * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> >> requirements in C data interface
> >>
> >> Patch available
> >>
> >> * ARROW-8330: [Documentation] The post release script generates the
> >> documentation with a development version
> >>
> >> Patch available
> >>
> >> * ARROW-8335: [Release] Add crossbow jobs to run release verification
> >>
> >> Patch in progress
> >>
> >> On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> >>>
> >>> I see ARROW-6871 in the list.
> >>> It seems it has some bugs, which are being fixed by ARROW-8239.
> >>> So I have added ARROW-8239 to the list.
> >>>
> >>> The PR for ARROW-8239 is already approved, so it is expected to be
> >> resolved
> >>> soon.
> >>>
> >>> Best,
> >>> Liya Fan
> >>>
> >>> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <emkornfield@gmail.com
> >
> >>> wrote:
> >>>
> >>>> I moved the Java issues out of 0.17.0, they seem complex enough or not
> >> of
> >>>> enough significance to make them blockers for 0.17.0 release. If
> >> owners of
> >>>> the issues disagree please move them back int.
> >>>>
> >>>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> >> wrote:
> >>>>
> >>>>> We've made good progress, but there are still 35 issues in the
> >>>>> backlog. Some of them are documentation related, but there are some
> >>>>> functionality-related patches that could be at risk. If all could
> >>>>> review again to trim out anything that isn't going to make the cut
> >> for
> >>>>> 0.17.0, please do
> >>>>>
> >>>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> I just took a first pass at reviewing the Java and Rust issues and
> >>>>> removed
> >>>>>> some from the 0.17.0 release. There are a few small Rust issues
> >> that I
> >>>> am
> >>>>>> actively working on for this release.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> hi Neal,
> >>>>>>>
> >>>>>>> Thanks for helping coordinate. I agree we should be in a
> >> position to
> >>>>>>> release sometime next week.
> >>>>>>>
> >>>>>>> Can folks from the Rust and Java side review issues in the
> >> backlog?
> >>>>>>> According to the dashboard there are 19 Rust issues open and 7
> >> Java
> >>>>>>> issues.
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> >>>>>>> <ne...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least)
> >> for
> >>>> a
> >>>>> 0.17
> >>>>>>>> release at the end of the month. Judging from
> >>>>>>>>
> >>>>>
> >> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> >>>>>>> it
> >>>>>>>> looks like we're getting closer.
> >>>>>>>>
> >>>>>>>> I'd encourage everyone to review their backlogs and (1) bump
> >> from
> >>>>> 0.17
> >>>>>>>> scope any tickets they don't plan to finish this week, and (2)
> >> if
> >>>>> there
> >>>>>>> are
> >>>>>>>> any issues that should block release, make sure they are
> >> flagged as
> >>>>>>>> "blockers".
> >>>>>>>>
> >>>>>>>> Neal
> >>>>>>>>
> >>>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <
> >> wesmckinn@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> It seems like the consensus is to push for a 0.17.0 major
> >> release
> >>>>>>>>> sooner rather than doing a patch release, since releases in
> >>>> general
> >>>>>>>>> are costly. This is fine with me. I see that a 0.17.0
> >> milestone
> >>>> has
> >>>>>>>>> been created in JIRA and some JIRA gardening has begun. Do
> >> you
> >>>>> think
> >>>>>>>>> we can be in a position to release by the week of March 23
> >> or the
> >>>>> week
> >>>>>>>>> of March 30?
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <
> >> wesmckinn@gmail.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> If people are generally on board with accelerating a 0.17.0
> >>>> major
> >>>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0"
> >> and
> >>>>>>>>>> beginning to do issue gardening to whittle things down to
> >>>>>>>>>> critical-looking bugs and high probability patches for the
> >> next
> >>>>>>> couple
> >>>>>>>>>> of weeks.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> >>>>> wesmckinn@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> I recall there are some other issues that have been
> >> reported
> >>>> or
> >>>>>>> fixed
> >>>>>>>>>>> that are critical and not yet marked with 0.16.1.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> >>>>>>>>>>> <ne...@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
> >>>>> instead of
> >>>>>>> a
> >>>>>>>>> patch
> >>>>>>>>>>>> release.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Neal
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> >>>>>>>>> neal.p.richardson@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> If releases were costless to make, I'd be all for
> >> it, but
> >>>>> it's
> >>>>>>> not
> >>>>>>>>> clear
> >>>>>>>>>>>>> to me that it's worth the diversion from other
> >> priorities
> >>>>> to
> >>>>>>> make
> >>>>>>>>> a release
> >>>>>>>>>>>>> right now. Nothing on
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> >>>>>>>>>>>>> jumps out to me as super urgent--what are you seeing
> >> as
> >>>>>>> critical?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If we did decide to go forward, would it be possible
> >> to
> >>>> do
> >>>>> a
> >>>>>>>>> release that
> >>>>>>>>>>>>> is limited to the affected implementations (say, do a
> >>>>>>> Python-only
> >>>>>>>>> release)?
> >>>>>>>>>>>>> That might reduce the cost of building and verifying
> >>>>> enough to
> >>>>>>>>> make it
> >>>>>>>>>>>>> reasonable to consider.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Neal
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> >>>>>>>>> szucs.krisztian@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> >>>>>>> wesmckinn@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> hi folks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There have been a number of critical issues
> >> reported
> >>>>> (many
> >>>>>>> of
> >>>>>>>>> them
> >>>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
> >>>>> interest
> >>>>>>> in
> >>>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
> >>>>> patches
> >>>>>>> onto a
> >>>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
> >>>> major
> >>>>>>>>> release is a
> >>>>>>>>>>>>>>> minimum of 6-8 weeks away from general
> >> availability?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
> >>>>> Krisztian
> >>>>>>> wrote
> >>>>>>>>> get
> >>>>>>>>>>>>>>> contributed as a PR?
> >>>>>>>>>>>>>> Not yet, but it is available at
> >>>>>>>>>>>>>>
> >>>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Wes
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Antoine Pitrou <an...@python.org>.
Hmm, if downstream libraries were expecting a dict, perhaps we'll need
to revert that change...
Regards
Antoine.
Le 06/04/2020 à 08:50, Joris Van den Bossche a écrit :
> We also have a recent regression related to the KeyValueMetadata wrapping
> python that is causing failures in downstream libraries, that seems a
> blocker for the release: https://issues.apache.org/jira/browse/ARROW-8342
>
> On Mon, 6 Apr 2020 at 00:25, Wes McKinney <we...@gmail.com> wrote:
>
>> We are getting close to the 0.17.0 endgame.
>>
>> Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
>> issues without patches yet so we should decide quickly whether they
>> need to be included. Are they any blocking issues not accounted for in
>> the milestone?
>>
>> * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
>>
>> Patch available
>>
>> * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
>> relative path to Flight.proto
>>
>> No patch yet
>>
>> * ARROW-7222 [Python][Release] Wipe any existing generated Python API
>> documentation when updating website
>>
>> This issue needs to be addressed by the release manager and the
>> Confluence instructions must be updated.
>>
>> * ARROW-7891 [C++] RecordBatch->Equals should also have a
>> check_metadata argument
>>
>> Patch available that needs to be reviewed and approved
>>
>> * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
>> schema
>>
>> Patch available, but failures to be resolved
>>
>> * ARROW-7965: [Python] Hold a reference to the dataset factory for later
>> reuse
>>
>> Depends on ARROW-8164, will require rebase
>>
>> * ARROW-8039: [Python][Dataset] Support using dataset API in
>> pyarrow.parquet with a minimal ParquetDataset shim
>>
>> Patch pending
>>
>> * ARROW-8047: [Python][Documentation] Document migration from
>> ParquetDataset to pyarrow.datasets
>>
>> May be tackled beyond 0.17.0
>>
>> * ARROW-8063: [Python] Add user guide documentation for Datasets API
>>
>> May be tackled beyond 0.17.0
>>
>> * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
>>
>> Does not seem strictly necessary for release, since a packaging issue
>>
>> * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
>>
>> Patch available, but needs review. May
>>
>> * ARROW-8213: [Python][Dataset] Opening a dataset with a local
>> incorrect path gives confusing error message
>>
>> Nice to have, but not essential
>>
>> * ARROW-8266: [C++] Add backup mirrors for external project source
>> downloads
>>
>> Patch available, nice to have
>>
>> * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
>> per "Feather V2" changes
>>
>> Patch available
>>
>> * ARROW-8300 [R] Documentation and changelog updates for 0.17
>>
>> Patch available
>>
>> * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
>> requirements in C data interface
>>
>> Patch available
>>
>> * ARROW-8330: [Documentation] The post release script generates the
>> documentation with a development version
>>
>> Patch available
>>
>> * ARROW-8335: [Release] Add crossbow jobs to run release verification
>>
>> Patch in progress
>>
>> On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
>>>
>>> I see ARROW-6871 in the list.
>>> It seems it has some bugs, which are being fixed by ARROW-8239.
>>> So I have added ARROW-8239 to the list.
>>>
>>> The PR for ARROW-8239 is already approved, so it is expected to be
>> resolved
>>> soon.
>>>
>>> Best,
>>> Liya Fan
>>>
>>> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
>>> wrote:
>>>
>>>> I moved the Java issues out of 0.17.0, they seem complex enough or not
>> of
>>>> enough significance to make them blockers for 0.17.0 release. If
>> owners of
>>>> the issues disagree please move them back int.
>>>>
>>>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
>> wrote:
>>>>
>>>>> We've made good progress, but there are still 35 issues in the
>>>>> backlog. Some of them are documentation related, but there are some
>>>>> functionality-related patches that could be at risk. If all could
>>>>> review again to trim out anything that isn't going to make the cut
>> for
>>>>> 0.17.0, please do
>>>>>
>>>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> I just took a first pass at reviewing the Java and Rust issues and
>>>>> removed
>>>>>> some from the 0.17.0 release. There are a few small Rust issues
>> that I
>>>> am
>>>>>> actively working on for this release.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> hi Neal,
>>>>>>>
>>>>>>> Thanks for helping coordinate. I agree we should be in a
>> position to
>>>>>>> release sometime next week.
>>>>>>>
>>>>>>> Can folks from the Rust and Java side review issues in the
>> backlog?
>>>>>>> According to the dashboard there are 19 Rust issues open and 7
>> Java
>>>>>>> issues.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
>>>>>>> <ne...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least)
>> for
>>>> a
>>>>> 0.17
>>>>>>>> release at the end of the month. Judging from
>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
>>>>>>> it
>>>>>>>> looks like we're getting closer.
>>>>>>>>
>>>>>>>> I'd encourage everyone to review their backlogs and (1) bump
>> from
>>>>> 0.17
>>>>>>>> scope any tickets they don't plan to finish this week, and (2)
>> if
>>>>> there
>>>>>>> are
>>>>>>>> any issues that should block release, make sure they are
>> flagged as
>>>>>>>> "blockers".
>>>>>>>>
>>>>>>>> Neal
>>>>>>>>
>>>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <
>> wesmckinn@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It seems like the consensus is to push for a 0.17.0 major
>> release
>>>>>>>>> sooner rather than doing a patch release, since releases in
>>>> general
>>>>>>>>> are costly. This is fine with me. I see that a 0.17.0
>> milestone
>>>> has
>>>>>>>>> been created in JIRA and some JIRA gardening has begun. Do
>> you
>>>>> think
>>>>>>>>> we can be in a position to release by the week of March 23
>> or the
>>>>> week
>>>>>>>>> of March 30?
>>>>>>>>>
>>>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <
>> wesmckinn@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> If people are generally on board with accelerating a 0.17.0
>>>> major
>>>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0"
>> and
>>>>>>>>>> beginning to do issue gardening to whittle things down to
>>>>>>>>>> critical-looking bugs and high probability patches for the
>> next
>>>>>>> couple
>>>>>>>>>> of weeks.
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
>>>>> wesmckinn@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I recall there are some other issues that have been
>> reported
>>>> or
>>>>>>> fixed
>>>>>>>>>>> that are critical and not yet marked with 0.16.1.
>>>>>>>>>>>
>>>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
>>>>>>>>>>> <ne...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
>>>>> instead of
>>>>>>> a
>>>>>>>>> patch
>>>>>>>>>>>> release.
>>>>>>>>>>>>
>>>>>>>>>>>> Neal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
>>>>>>>>> neal.p.richardson@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> If releases were costless to make, I'd be all for
>> it, but
>>>>> it's
>>>>>>> not
>>>>>>>>> clear
>>>>>>>>>>>>> to me that it's worth the diversion from other
>> priorities
>>>>> to
>>>>>>> make
>>>>>>>>> a release
>>>>>>>>>>>>> right now. Nothing on
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
>>>>>>>>>>>>> jumps out to me as super urgent--what are you seeing
>> as
>>>>>>> critical?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If we did decide to go forward, would it be possible
>> to
>>>> do
>>>>> a
>>>>>>>>> release that
>>>>>>>>>>>>> is limited to the affected implementations (say, do a
>>>>>>> Python-only
>>>>>>>>> release)?
>>>>>>>>>>>>> That might reduce the cost of building and verifying
>>>>> enough to
>>>>>>>>> make it
>>>>>>>>>>>>> reasonable to consider.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Neal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
>>>>>>>>> szucs.krisztian@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
>>>>>>> wesmckinn@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> hi folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There have been a number of critical issues
>> reported
>>>>> (many
>>>>>>> of
>>>>>>>>> them
>>>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
>>>>> interest
>>>>>>> in
>>>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
>>>>> patches
>>>>>>> onto a
>>>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
>>>> major
>>>>>>>>> release is a
>>>>>>>>>>>>>>> minimum of 6-8 weeks away from general
>> availability?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
>>>>> Krisztian
>>>>>>> wrote
>>>>>>>>> get
>>>>>>>>>>>>>>> contributed as a PR?
>>>>>>>>>>>>>> Not yet, but it is available at
>>>>>>>>>>>>>>
>>>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>
Re: Preparing for 0.17.0 Arrow release
Posted by Joris Van den Bossche <jo...@gmail.com>.
We also have a recent regression related to the KeyValueMetadata wrapping
python that is causing failures in downstream libraries, that seems a
blocker for the release: https://issues.apache.org/jira/browse/ARROW-8342
On Mon, 6 Apr 2020 at 00:25, Wes McKinney <we...@gmail.com> wrote:
> We are getting close to the 0.17.0 endgame.
>
> Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> issues without patches yet so we should decide quickly whether they
> need to be included. Are they any blocking issues not accounted for in
> the milestone?
>
> * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
>
> Patch available
>
> * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> relative path to Flight.proto
>
> No patch yet
>
> * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> documentation when updating website
>
> This issue needs to be addressed by the release manager and the
> Confluence instructions must be updated.
>
> * ARROW-7891 [C++] RecordBatch->Equals should also have a
> check_metadata argument
>
> Patch available that needs to be reviewed and approved
>
> * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical
> schema
>
> Patch available, but failures to be resolved
>
> * ARROW-7965: [Python] Hold a reference to the dataset factory for later
> reuse
>
> Depends on ARROW-8164, will require rebase
>
> * ARROW-8039: [Python][Dataset] Support using dataset API in
> pyarrow.parquet with a minimal ParquetDataset shim
>
> Patch pending
>
> * ARROW-8047: [Python][Documentation] Document migration from
> ParquetDataset to pyarrow.datasets
>
> May be tackled beyond 0.17.0
>
> * ARROW-8063: [Python] Add user guide documentation for Datasets API
>
> May be tackled beyond 0.17.0
>
> * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
>
> Does not seem strictly necessary for release, since a packaging issue
>
> * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
>
> Patch available, but needs review. May
>
> * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> incorrect path gives confusing error message
>
> Nice to have, but not essential
>
> * ARROW-8266: [C++] Add backup mirrors for external project source
> downloads
>
> Patch available, nice to have
>
> * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> per "Feather V2" changes
>
> Patch available
>
> * ARROW-8300 [R] Documentation and changelog updates for 0.17
>
> Patch available
>
> * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> requirements in C data interface
>
> Patch available
>
> * ARROW-8330: [Documentation] The post release script generates the
> documentation with a development version
>
> Patch available
>
> * ARROW-8335: [Release] Add crossbow jobs to run release verification
>
> Patch in progress
>
> On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
> >
> > I see ARROW-6871 in the list.
> > It seems it has some bugs, which are being fixed by ARROW-8239.
> > So I have added ARROW-8239 to the list.
> >
> > The PR for ARROW-8239 is already approved, so it is expected to be
> resolved
> > soon.
> >
> > Best,
> > Liya Fan
> >
> > On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > I moved the Java issues out of 0.17.0, they seem complex enough or not
> of
> > > enough significance to make them blockers for 0.17.0 release. If
> owners of
> > > the issues disagree please move them back int.
> > >
> > > On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > We've made good progress, but there are still 35 issues in the
> > > > backlog. Some of them are documentation related, but there are some
> > > > functionality-related patches that could be at risk. If all could
> > > > review again to trim out anything that isn't going to make the cut
> for
> > > > 0.17.0, please do
> > > >
> > > > On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> > > wrote:
> > > > >
> > > > > I just took a first pass at reviewing the Java and Rust issues and
> > > > removed
> > > > > some from the 0.17.0 release. There are a few small Rust issues
> that I
> > > am
> > > > > actively working on for this release.
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > hi Neal,
> > > > > >
> > > > > > Thanks for helping coordinate. I agree we should be in a
> position to
> > > > > > release sometime next week.
> > > > > >
> > > > > > Can folks from the Rust and Java side review issues in the
> backlog?
> > > > > > According to the dashboard there are 19 Rust issues open and 7
> Java
> > > > > > issues.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > > > > <ne...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > > A few weeks ago, there seemed to be consensus (lazy, at least)
> for
> > > a
> > > > 0.17
> > > > > > > release at the end of the month. Judging from
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > > > > it
> > > > > > > looks like we're getting closer.
> > > > > > >
> > > > > > > I'd encourage everyone to review their backlogs and (1) bump
> from
> > > > 0.17
> > > > > > > scope any tickets they don't plan to finish this week, and (2)
> if
> > > > there
> > > > > > are
> > > > > > > any issues that should block release, make sure they are
> flagged as
> > > > > > > "blockers".
> > > > > > >
> > > > > > > Neal
> > > > > > >
> > > > > > > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <
> wesmckinn@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > It seems like the consensus is to push for a 0.17.0 major
> release
> > > > > > > > sooner rather than doing a patch release, since releases in
> > > general
> > > > > > > > are costly. This is fine with me. I see that a 0.17.0
> milestone
> > > has
> > > > > > > > been created in JIRA and some JIRA gardening has begun. Do
> you
> > > > think
> > > > > > > > we can be in a position to release by the week of March 23
> or the
> > > > week
> > > > > > > > of March 30?
> > > > > > > >
> > > > > > > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <
> wesmckinn@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > If people are generally on board with accelerating a 0.17.0
> > > major
> > > > > > > > > release, then I would suggest renaming "1.0.0" to "0.17.0"
> and
> > > > > > > > > beginning to do issue gardening to whittle things down to
> > > > > > > > > critical-looking bugs and high probability patches for the
> next
> > > > > > couple
> > > > > > > > > of weeks.
> > > > > > > > >
> > > > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I recall there are some other issues that have been
> reported
> > > or
> > > > > > fixed
> > > > > > > > > > that are critical and not yet marked with 0.16.1.
> > > > > > > > > >
> > > > > > > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > > > > > > <ne...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > I would also be more supportive of doing 0.17 earlier
> > > > instead of
> > > > > > a
> > > > > > > > patch
> > > > > > > > > > > release.
> > > > > > > > > > >
> > > > > > > > > > > Neal
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > > > > > neal.p.richardson@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > If releases were costless to make, I'd be all for
> it, but
> > > > it's
> > > > > > not
> > > > > > > > clear
> > > > > > > > > > > > to me that it's worth the diversion from other
> priorities
> > > > to
> > > > > > make
> > > > > > > > a release
> > > > > > > > > > > > right now. Nothing on
> > > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > > > > > > jumps out to me as super urgent--what are you seeing
> as
> > > > > > critical?
> > > > > > > > > > > >
> > > > > > > > > > > > If we did decide to go forward, would it be possible
> to
> > > do
> > > > a
> > > > > > > > release that
> > > > > > > > > > > > is limited to the affected implementations (say, do a
> > > > > > Python-only
> > > > > > > > release)?
> > > > > > > > > > > > That might reduce the cost of building and verifying
> > > > enough to
> > > > > > > > make it
> > > > > > > > > > > > reasonable to consider.
> > > > > > > > > > > >
> > > > > > > > > > > > Neal
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > > > > > szucs.krisztian@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > > > > wesmckinn@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > hi folks,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > There have been a number of critical issues
> reported
> > > > (many
> > > > > > of
> > > > > > > > them
> > > > > > > > > > > >> > fixed already) since 0.16.0 was released. Is there
> > > > interest
> > > > > > in
> > > > > > > > > > > >> > preparing a patch 0.16.1 release (with backported
> > > > patches
> > > > > > onto a
> > > > > > > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next
> > > major
> > > > > > > > release is a
> > > > > > > > > > > >> > minimum of 6-8 weeks away from general
> availability?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Did the 0.15.1 patch release helper script that
> > > > Krisztian
> > > > > > wrote
> > > > > > > > get
> > > > > > > > > > > >> > contributed as a PR?
> > > > > > > > > > > >> Not yet, but it is available at
> > > > > > > > > > > >>
> > > > > > https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks
> > > > > > > > > > > >> > Wes
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Antoine Pitrou <an...@python.org>.
Hi,
I added the following issue to the cpp-1.6.0 milestone:
* PARQUET-1835 [C++] Fix crashes on invalid input (OSS-Fuzz)
There's a PR up for it and it's simple enough to be validated quickly, IMHO.
Regards
Antoine.
Le 06/04/2020 à 00:24, Wes McKinney a écrit :
> We are getting close to the 0.17.0 endgame.
>
> Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
> issues without patches yet so we should decide quickly whether they
> need to be included. Are they any blocking issues not accounted for in
> the milestone?
>
> * ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
>
> Patch available
>
> * ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
> relative path to Flight.proto
>
> No patch yet
>
> * ARROW-7222 [Python][Release] Wipe any existing generated Python API
> documentation when updating website
>
> This issue needs to be addressed by the release manager and the
> Confluence instructions must be updated.
>
> * ARROW-7891 [C++] RecordBatch->Equals should also have a
> check_metadata argument
>
> Patch available that needs to be reviewed and approved
>
> * ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical schema
>
> Patch available, but failures to be resolved
>
> * ARROW-7965: [Python] Hold a reference to the dataset factory for later reuse
>
> Depends on ARROW-8164, will require rebase
>
> * ARROW-8039: [Python][Dataset] Support using dataset API in
> pyarrow.parquet with a minimal ParquetDataset shim
>
> Patch pending
>
> * ARROW-8047: [Python][Documentation] Document migration from
> ParquetDataset to pyarrow.datasets
>
> May be tackled beyond 0.17.0
>
> * ARROW-8063: [Python] Add user guide documentation for Datasets API
>
> May be tackled beyond 0.17.0
>
> * ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
>
> Does not seem strictly necessary for release, since a packaging issue
>
> * ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
>
> Patch available, but needs review. May
>
> * ARROW-8213: [Python][Dataset] Opening a dataset with a local
> incorrect path gives confusing error message
>
> Nice to have, but not essential
>
> * ARROW-8266: [C++] Add backup mirrors for external project source downloads
>
> Patch available, nice to have
>
> * ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
> per "Feather V2" changes
>
> Patch available
>
> * ARROW-8300 [R] Documentation and changelog updates for 0.17
>
> Patch available
>
> * ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
> requirements in C data interface
>
> Patch available
>
> * ARROW-8330: [Documentation] The post release script generates the
> documentation with a development version
>
> Patch available
>
> * ARROW-8335: [Release] Add crossbow jobs to run release verification
>
> Patch in progress
>
> On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
>>
>> I see ARROW-6871 in the list.
>> It seems it has some bugs, which are being fixed by ARROW-8239.
>> So I have added ARROW-8239 to the list.
>>
>> The PR for ARROW-8239 is already approved, so it is expected to be resolved
>> soon.
>>
>> Best,
>> Liya Fan
>>
>> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> I moved the Java issues out of 0.17.0, they seem complex enough or not of
>>> enough significance to make them blockers for 0.17.0 release. If owners of
>>> the issues disagree please move them back int.
>>>
>>> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com> wrote:
>>>
>>>> We've made good progress, but there are still 35 issues in the
>>>> backlog. Some of them are documentation related, but there are some
>>>> functionality-related patches that could be at risk. If all could
>>>> review again to trim out anything that isn't going to make the cut for
>>>> 0.17.0, please do
>>>>
>>>> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
>>> wrote:
>>>>>
>>>>> I just took a first pass at reviewing the Java and Rust issues and
>>>> removed
>>>>> some from the 0.17.0 release. There are a few small Rust issues that I
>>> am
>>>>> actively working on for this release.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> hi Neal,
>>>>>>
>>>>>> Thanks for helping coordinate. I agree we should be in a position to
>>>>>> release sometime next week.
>>>>>>
>>>>>> Can folks from the Rust and Java side review issues in the backlog?
>>>>>> According to the dashboard there are 19 Rust issues open and 7 Java
>>>>>> issues.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
>>>>>> <ne...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>> A few weeks ago, there seemed to be consensus (lazy, at least) for
>>> a
>>>> 0.17
>>>>>>> release at the end of the month. Judging from
>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
>>>>>> it
>>>>>>> looks like we're getting closer.
>>>>>>>
>>>>>>> I'd encourage everyone to review their backlogs and (1) bump from
>>>> 0.17
>>>>>>> scope any tickets they don't plan to finish this week, and (2) if
>>>> there
>>>>>> are
>>>>>>> any issues that should block release, make sure they are flagged as
>>>>>>> "blockers".
>>>>>>>
>>>>>>> Neal
>>>>>>>
>>>>>>> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> It seems like the consensus is to push for a 0.17.0 major release
>>>>>>>> sooner rather than doing a patch release, since releases in
>>> general
>>>>>>>> are costly. This is fine with me. I see that a 0.17.0 milestone
>>> has
>>>>>>>> been created in JIRA and some JIRA gardening has begun. Do you
>>>> think
>>>>>>>> we can be in a position to release by the week of March 23 or the
>>>> week
>>>>>>>> of March 30?
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> If people are generally on board with accelerating a 0.17.0
>>> major
>>>>>>>>> release, then I would suggest renaming "1.0.0" to "0.17.0" and
>>>>>>>>> beginning to do issue gardening to whittle things down to
>>>>>>>>> critical-looking bugs and high probability patches for the next
>>>>>> couple
>>>>>>>>> of weeks.
>>>>>>>>>
>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
>>>> wesmckinn@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I recall there are some other issues that have been reported
>>> or
>>>>>> fixed
>>>>>>>>>> that are critical and not yet marked with 0.16.1.
>>>>>>>>>>
>>>>>>>>>> I'm also OK with doing a 0.17.0 release sooner
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
>>>>>>>>>> <ne...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I would also be more supportive of doing 0.17 earlier
>>>> instead of
>>>>>> a
>>>>>>>> patch
>>>>>>>>>>> release.
>>>>>>>>>>>
>>>>>>>>>>> Neal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
>>>>>>>> neal.p.richardson@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If releases were costless to make, I'd be all for it, but
>>>> it's
>>>>>> not
>>>>>>>> clear
>>>>>>>>>>>> to me that it's worth the diversion from other priorities
>>>> to
>>>>>> make
>>>>>>>> a release
>>>>>>>>>>>> right now. Nothing on
>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
>>>>>>>>>>>> jumps out to me as super urgent--what are you seeing as
>>>>>> critical?
>>>>>>>>>>>>
>>>>>>>>>>>> If we did decide to go forward, would it be possible to
>>> do
>>>> a
>>>>>>>> release that
>>>>>>>>>>>> is limited to the affected implementations (say, do a
>>>>>> Python-only
>>>>>>>> release)?
>>>>>>>>>>>> That might reduce the cost of building and verifying
>>>> enough to
>>>>>>>> make it
>>>>>>>>>>>> reasonable to consider.
>>>>>>>>>>>>
>>>>>>>>>>>> Neal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
>>>>>>>> szucs.krisztian@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
>>>>>> wesmckinn@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There have been a number of critical issues reported
>>>> (many
>>>>>> of
>>>>>>>> them
>>>>>>>>>>>>>> fixed already) since 0.16.0 was released. Is there
>>>> interest
>>>>>> in
>>>>>>>>>>>>>> preparing a patch 0.16.1 release (with backported
>>>> patches
>>>>>> onto a
>>>>>>>>>>>>>> maint-0.16.x branch as with 0.15.1) since the next
>>> major
>>>>>>>> release is a
>>>>>>>>>>>>>> minimum of 6-8 weeks away from general availability?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Did the 0.15.1 patch release helper script that
>>>> Krisztian
>>>>>> wrote
>>>>>>>> get
>>>>>>>>>>>>>> contributed as a PR?
>>>>>>>>>>>>> Not yet, but it is available at
>>>>>>>>>>>>>
>>>>>> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
We are getting close to the 0.17.0 endgame.
Here are the 18 JIRAs still in the 0.17.0 milestone. There are a few
issues without patches yet so we should decide quickly whether they
need to be included. Are they any blocking issues not accounted for in
the milestone?
* ARROW-6947 [Rust] [DataFusion] Add support for scalar UDFs
Patch available
* ARROW-7794 [Rust] cargo publish fails for arrow-flight due to
relative path to Flight.proto
No patch yet
* ARROW-7222 [Python][Release] Wipe any existing generated Python API
documentation when updating website
This issue needs to be addressed by the release manager and the
Confluence instructions must be updated.
* ARROW-7891 [C++] RecordBatch->Equals should also have a
check_metadata argument
Patch available that needs to be reviewed and approved
* ARROW-8164: [C++][Dataset] Let datasets be viewable with non-identical schema
Patch available, but failures to be resolved
* ARROW-7965: [Python] Hold a reference to the dataset factory for later reuse
Depends on ARROW-8164, will require rebase
* ARROW-8039: [Python][Dataset] Support using dataset API in
pyarrow.parquet with a minimal ParquetDataset shim
Patch pending
* ARROW-8047: [Python][Documentation] Document migration from
ParquetDataset to pyarrow.datasets
May be tackled beyond 0.17.0
* ARROW-8063: [Python] Add user guide documentation for Datasets API
May be tackled beyond 0.17.0
* ARROW-8149 [C++/Python] Enable CUDA Support in conda recipes
Does not seem strictly necessary for release, since a packaging issue
* ARROW-8162: [Format][Python] Add serialization for CSF sparse tensors
Patch available, but needs review. May
* ARROW-8213: [Python][Dataset] Opening a dataset with a local
incorrect path gives confusing error message
Nice to have, but not essential
* ARROW-8266: [C++] Add backup mirrors for external project source downloads
Patch available, nice to have
* ARROW-8275 [Python][Docs] Review Feather + IPC file documentation
per "Feather V2" changes
Patch available
* ARROW-8300 [R] Documentation and changelog updates for 0.17
Patch available
* ARROW-8320 [Documentation][Format] Clarify (lack of) alignment
requirements in C data interface
Patch available
* ARROW-8330: [Documentation] The post release script generates the
documentation with a development version
Patch available
* ARROW-8335: [Release] Add crossbow jobs to run release verification
Patch in progress
On Tue, Mar 31, 2020 at 11:23 PM Fan Liya <li...@gmail.com> wrote:
>
> I see ARROW-6871 in the list.
> It seems it has some bugs, which are being fixed by ARROW-8239.
> So I have added ARROW-8239 to the list.
>
> The PR for ARROW-8239 is already approved, so it is expected to be resolved
> soon.
>
> Best,
> Liya Fan
>
> On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > I moved the Java issues out of 0.17.0, they seem complex enough or not of
> > enough significance to make them blockers for 0.17.0 release. If owners of
> > the issues disagree please move them back int.
> >
> > On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > We've made good progress, but there are still 35 issues in the
> > > backlog. Some of them are documentation related, but there are some
> > > functionality-related patches that could be at risk. If all could
> > > review again to trim out anything that isn't going to make the cut for
> > > 0.17.0, please do
> > >
> > > On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> > wrote:
> > > >
> > > > I just took a first pass at reviewing the Java and Rust issues and
> > > removed
> > > > some from the 0.17.0 release. There are a few small Rust issues that I
> > am
> > > > actively working on for this release.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >
> > > > > hi Neal,
> > > > >
> > > > > Thanks for helping coordinate. I agree we should be in a position to
> > > > > release sometime next week.
> > > > >
> > > > > Can folks from the Rust and Java side review issues in the backlog?
> > > > > According to the dashboard there are 19 Rust issues open and 7 Java
> > > > > issues.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > > > <ne...@gmail.com> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > > A few weeks ago, there seemed to be consensus (lazy, at least) for
> > a
> > > 0.17
> > > > > > release at the end of the month. Judging from
> > > > > >
> > > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > > > it
> > > > > > looks like we're getting closer.
> > > > > >
> > > > > > I'd encourage everyone to review their backlogs and (1) bump from
> > > 0.17
> > > > > > scope any tickets they don't plan to finish this week, and (2) if
> > > there
> > > > > are
> > > > > > any issues that should block release, make sure they are flagged as
> > > > > > "blockers".
> > > > > >
> > > > > > Neal
> > > > > >
> > > > > > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > It seems like the consensus is to push for a 0.17.0 major release
> > > > > > > sooner rather than doing a patch release, since releases in
> > general
> > > > > > > are costly. This is fine with me. I see that a 0.17.0 milestone
> > has
> > > > > > > been created in JIRA and some JIRA gardening has begun. Do you
> > > think
> > > > > > > we can be in a position to release by the week of March 23 or the
> > > week
> > > > > > > of March 30?
> > > > > > >
> > > > > > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > If people are generally on board with accelerating a 0.17.0
> > major
> > > > > > > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > > > > > beginning to do issue gardening to whittle things down to
> > > > > > > > critical-looking bugs and high probability patches for the next
> > > > > couple
> > > > > > > > of weeks.
> > > > > > > >
> > > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I recall there are some other issues that have been reported
> > or
> > > > > fixed
> > > > > > > > > that are critical and not yet marked with 0.16.1.
> > > > > > > > >
> > > > > > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > > > > > >
> > > > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > > > > > <ne...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I would also be more supportive of doing 0.17 earlier
> > > instead of
> > > > > a
> > > > > > > patch
> > > > > > > > > > release.
> > > > > > > > > >
> > > > > > > > > > Neal
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > > > > neal.p.richardson@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > If releases were costless to make, I'd be all for it, but
> > > it's
> > > > > not
> > > > > > > clear
> > > > > > > > > > > to me that it's worth the diversion from other priorities
> > > to
> > > > > make
> > > > > > > a release
> > > > > > > > > > > right now. Nothing on
> > > > > > > > > > >
> > > > > > >
> > > > >
> > >
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > > > > > jumps out to me as super urgent--what are you seeing as
> > > > > critical?
> > > > > > > > > > >
> > > > > > > > > > > If we did decide to go forward, would it be possible to
> > do
> > > a
> > > > > > > release that
> > > > > > > > > > > is limited to the affected implementations (say, do a
> > > > > Python-only
> > > > > > > release)?
> > > > > > > > > > > That might reduce the cost of building and verifying
> > > enough to
> > > > > > > make it
> > > > > > > > > > > reasonable to consider.
> > > > > > > > > > >
> > > > > > > > > > > Neal
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > > > > szucs.krisztian@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > > > wesmckinn@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > hi folks,
> > > > > > > > > > >> >
> > > > > > > > > > >> > There have been a number of critical issues reported
> > > (many
> > > > > of
> > > > > > > them
> > > > > > > > > > >> > fixed already) since 0.16.0 was released. Is there
> > > interest
> > > > > in
> > > > > > > > > > >> > preparing a patch 0.16.1 release (with backported
> > > patches
> > > > > onto a
> > > > > > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next
> > major
> > > > > > > release is a
> > > > > > > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > > > > > > >> >
> > > > > > > > > > >> > Did the 0.15.1 patch release helper script that
> > > Krisztian
> > > > > wrote
> > > > > > > get
> > > > > > > > > > >> > contributed as a PR?
> > > > > > > > > > >> Not yet, but it is available at
> > > > > > > > > > >>
> > > > > https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks
> > > > > > > > > > >> > Wes
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
Re: Preparing for 0.17.0 Arrow release
Posted by Fan Liya <li...@gmail.com>.
I see ARROW-6871 in the list.
It seems it has some bugs, which are being fixed by ARROW-8239.
So I have added ARROW-8239 to the list.
The PR for ARROW-8239 is already approved, so it is expected to be resolved
soon.
Best,
Liya Fan
On Wed, Apr 1, 2020 at 12:01 PM Micah Kornfield <em...@gmail.com>
wrote:
> I moved the Java issues out of 0.17.0, they seem complex enough or not of
> enough significance to make them blockers for 0.17.0 release. If owners of
> the issues disagree please move them back int.
>
> On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com> wrote:
>
> > We've made good progress, but there are still 35 issues in the
> > backlog. Some of them are documentation related, but there are some
> > functionality-related patches that could be at risk. If all could
> > review again to trim out anything that isn't going to make the cut for
> > 0.17.0, please do
> >
> > On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > I just took a first pass at reviewing the Java and Rust issues and
> > removed
> > > some from the 0.17.0 release. There are a few small Rust issues that I
> am
> > > actively working on for this release.
> > >
> > > Thanks.
> > >
> > >
> > > On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > > > hi Neal,
> > > >
> > > > Thanks for helping coordinate. I agree we should be in a position to
> > > > release sometime next week.
> > > >
> > > > Can folks from the Rust and Java side review issues in the backlog?
> > > > According to the dashboard there are 19 Rust issues open and 7 Java
> > > > issues.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > > <ne...@gmail.com> wrote:
> > > > >
> > > > > Hi all,
> > > > > A few weeks ago, there seemed to be consensus (lazy, at least) for
> a
> > 0.17
> > > > > release at the end of the month. Judging from
> > > > >
> > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > > it
> > > > > looks like we're getting closer.
> > > > >
> > > > > I'd encourage everyone to review their backlogs and (1) bump from
> > 0.17
> > > > > scope any tickets they don't plan to finish this week, and (2) if
> > there
> > > > are
> > > > > any issues that should block release, make sure they are flagged as
> > > > > "blockers".
> > > > >
> > > > > Neal
> > > > >
> > > > > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > It seems like the consensus is to push for a 0.17.0 major release
> > > > > > sooner rather than doing a patch release, since releases in
> general
> > > > > > are costly. This is fine with me. I see that a 0.17.0 milestone
> has
> > > > > > been created in JIRA and some JIRA gardening has begun. Do you
> > think
> > > > > > we can be in a position to release by the week of March 23 or the
> > week
> > > > > > of March 30?
> > > > > >
> > > > > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <wesmckinn@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > If people are generally on board with accelerating a 0.17.0
> major
> > > > > > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > > > > beginning to do issue gardening to whittle things down to
> > > > > > > critical-looking bugs and high probability patches for the next
> > > > couple
> > > > > > > of weeks.
> > > > > > >
> > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> > wesmckinn@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > I recall there are some other issues that have been reported
> or
> > > > fixed
> > > > > > > > that are critical and not yet marked with 0.16.1.
> > > > > > > >
> > > > > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > > > > >
> > > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > > > > <ne...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I would also be more supportive of doing 0.17 earlier
> > instead of
> > > > a
> > > > > > patch
> > > > > > > > > release.
> > > > > > > > >
> > > > > > > > > Neal
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > > > neal.p.richardson@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If releases were costless to make, I'd be all for it, but
> > it's
> > > > not
> > > > > > clear
> > > > > > > > > > to me that it's worth the diversion from other priorities
> > to
> > > > make
> > > > > > a release
> > > > > > > > > > right now. Nothing on
> > > > > > > > > >
> > > > > >
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > > > > jumps out to me as super urgent--what are you seeing as
> > > > critical?
> > > > > > > > > >
> > > > > > > > > > If we did decide to go forward, would it be possible to
> do
> > a
> > > > > > release that
> > > > > > > > > > is limited to the affected implementations (say, do a
> > > > Python-only
> > > > > > release)?
> > > > > > > > > > That might reduce the cost of building and verifying
> > enough to
> > > > > > make it
> > > > > > > > > > reasonable to consider.
> > > > > > > > > >
> > > > > > > > > > Neal
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > > > szucs.krisztian@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > hi folks,
> > > > > > > > > >> >
> > > > > > > > > >> > There have been a number of critical issues reported
> > (many
> > > > of
> > > > > > them
> > > > > > > > > >> > fixed already) since 0.16.0 was released. Is there
> > interest
> > > > in
> > > > > > > > > >> > preparing a patch 0.16.1 release (with backported
> > patches
> > > > onto a
> > > > > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next
> major
> > > > > > release is a
> > > > > > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > > > > > >> >
> > > > > > > > > >> > Did the 0.15.1 patch release helper script that
> > Krisztian
> > > > wrote
> > > > > > get
> > > > > > > > > >> > contributed as a PR?
> > > > > > > > > >> Not yet, but it is available at
> > > > > > > > > >>
> > > > https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks
> > > > > > > > > >> > Wes
> > > > > > > > > >>
> > > > > > > > > >
> > > > > >
> > > >
> >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Micah Kornfield <em...@gmail.com>.
I moved the Java issues out of 0.17.0, they seem complex enough or not of
enough significance to make them blockers for 0.17.0 release. If owners of
the issues disagree please move them back int.
On Tue, Mar 31, 2020 at 6:05 PM Wes McKinney <we...@gmail.com> wrote:
> We've made good progress, but there are still 35 issues in the
> backlog. Some of them are documentation related, but there are some
> functionality-related patches that could be at risk. If all could
> review again to trim out anything that isn't going to make the cut for
> 0.17.0, please do
>
> On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com> wrote:
> >
> > I just took a first pass at reviewing the Java and Rust issues and
> removed
> > some from the 0.17.0 release. There are a few small Rust issues that I am
> > actively working on for this release.
> >
> > Thanks.
> >
> >
> > On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > hi Neal,
> > >
> > > Thanks for helping coordinate. I agree we should be in a position to
> > > release sometime next week.
> > >
> > > Can folks from the Rust and Java side review issues in the backlog?
> > > According to the dashboard there are 19 Rust issues open and 7 Java
> > > issues.
> > >
> > > Thanks
> > >
> > > On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > > <ne...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > > A few weeks ago, there seemed to be consensus (lazy, at least) for a
> 0.17
> > > > release at the end of the month. Judging from
> > > >
> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > > it
> > > > looks like we're getting closer.
> > > >
> > > > I'd encourage everyone to review their backlogs and (1) bump from
> 0.17
> > > > scope any tickets they don't plan to finish this week, and (2) if
> there
> > > are
> > > > any issues that should block release, make sure they are flagged as
> > > > "blockers".
> > > >
> > > > Neal
> > > >
> > > > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >
> > > > > It seems like the consensus is to push for a 0.17.0 major release
> > > > > sooner rather than doing a patch release, since releases in general
> > > > > are costly. This is fine with me. I see that a 0.17.0 milestone has
> > > > > been created in JIRA and some JIRA gardening has begun. Do you
> think
> > > > > we can be in a position to release by the week of March 23 or the
> week
> > > > > of March 30?
> > > > >
> > > > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > If people are generally on board with accelerating a 0.17.0 major
> > > > > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > > > beginning to do issue gardening to whittle things down to
> > > > > > critical-looking bugs and high probability patches for the next
> > > couple
> > > > > > of weeks.
> > > > > >
> > > > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <
> wesmckinn@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > I recall there are some other issues that have been reported or
> > > fixed
> > > > > > > that are critical and not yet marked with 0.16.1.
> > > > > > >
> > > > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > > > >
> > > > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > > > <ne...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I would also be more supportive of doing 0.17 earlier
> instead of
> > > a
> > > > > patch
> > > > > > > > release.
> > > > > > > >
> > > > > > > > Neal
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > > neal.p.richardson@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If releases were costless to make, I'd be all for it, but
> it's
> > > not
> > > > > clear
> > > > > > > > > to me that it's worth the diversion from other priorities
> to
> > > make
> > > > > a release
> > > > > > > > > right now. Nothing on
> > > > > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > > > jumps out to me as super urgent--what are you seeing as
> > > critical?
> > > > > > > > >
> > > > > > > > > If we did decide to go forward, would it be possible to do
> a
> > > > > release that
> > > > > > > > > is limited to the affected implementations (say, do a
> > > Python-only
> > > > > release)?
> > > > > > > > > That might reduce the cost of building and verifying
> enough to
> > > > > make it
> > > > > > > > > reasonable to consider.
> > > > > > > > >
> > > > > > > > > Neal
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > > szucs.krisztian@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > hi folks,
> > > > > > > > >> >
> > > > > > > > >> > There have been a number of critical issues reported
> (many
> > > of
> > > > > them
> > > > > > > > >> > fixed already) since 0.16.0 was released. Is there
> interest
> > > in
> > > > > > > > >> > preparing a patch 0.16.1 release (with backported
> patches
> > > onto a
> > > > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next major
> > > > > release is a
> > > > > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > > > > >> >
> > > > > > > > >> > Did the 0.15.1 patch release helper script that
> Krisztian
> > > wrote
> > > > > get
> > > > > > > > >> > contributed as a PR?
> > > > > > > > >> Not yet, but it is available at
> > > > > > > > >>
> > > https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > > > >> >
> > > > > > > > >> > Thanks
> > > > > > > > >> > Wes
> > > > > > > > >>
> > > > > > > > >
> > > > >
> > >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
We've made good progress, but there are still 35 issues in the
backlog. Some of them are documentation related, but there are some
functionality-related patches that could be at risk. If all could
review again to trim out anything that isn't going to make the cut for
0.17.0, please do
On Wed, Mar 25, 2020 at 2:39 PM Andy Grove <an...@gmail.com> wrote:
>
> I just took a first pass at reviewing the Java and Rust issues and removed
> some from the 0.17.0 release. There are a few small Rust issues that I am
> actively working on for this release.
>
> Thanks.
>
>
> On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Neal,
> >
> > Thanks for helping coordinate. I agree we should be in a position to
> > release sometime next week.
> >
> > Can folks from the Rust and Java side review issues in the backlog?
> > According to the dashboard there are 19 Rust issues open and 7 Java
> > issues.
> >
> > Thanks
> >
> > On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> > <ne...@gmail.com> wrote:
> > >
> > > Hi all,
> > > A few weeks ago, there seemed to be consensus (lazy, at least) for a 0.17
> > > release at the end of the month. Judging from
> > > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> > it
> > > looks like we're getting closer.
> > >
> > > I'd encourage everyone to review their backlogs and (1) bump from 0.17
> > > scope any tickets they don't plan to finish this week, and (2) if there
> > are
> > > any issues that should block release, make sure they are flagged as
> > > "blockers".
> > >
> > > Neal
> > >
> > > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > > > It seems like the consensus is to push for a 0.17.0 major release
> > > > sooner rather than doing a patch release, since releases in general
> > > > are costly. This is fine with me. I see that a 0.17.0 milestone has
> > > > been created in JIRA and some JIRA gardening has begun. Do you think
> > > > we can be in a position to release by the week of March 23 or the week
> > > > of March 30?
> > > >
> > > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > > >
> > > > > If people are generally on board with accelerating a 0.17.0 major
> > > > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > > beginning to do issue gardening to whittle things down to
> > > > > critical-looking bugs and high probability patches for the next
> > couple
> > > > > of weeks.
> > > > >
> > > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > I recall there are some other issues that have been reported or
> > fixed
> > > > > > that are critical and not yet marked with 0.16.1.
> > > > > >
> > > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > > >
> > > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > > <ne...@gmail.com> wrote:
> > > > > > >
> > > > > > > I would also be more supportive of doing 0.17 earlier instead of
> > a
> > > > patch
> > > > > > > release.
> > > > > > >
> > > > > > > Neal
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > > neal.p.richardson@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > If releases were costless to make, I'd be all for it, but it's
> > not
> > > > clear
> > > > > > > > to me that it's worth the diversion from other priorities to
> > make
> > > > a release
> > > > > > > > right now. Nothing on
> > > > > > > >
> > > >
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > > jumps out to me as super urgent--what are you seeing as
> > critical?
> > > > > > > >
> > > > > > > > If we did decide to go forward, would it be possible to do a
> > > > release that
> > > > > > > > is limited to the affected implementations (say, do a
> > Python-only
> > > > release)?
> > > > > > > > That might reduce the cost of building and verifying enough to
> > > > make it
> > > > > > > > reasonable to consider.
> > > > > > > >
> > > > > > > > Neal
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > > szucs.krisztian@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > > > wrote:
> > > > > > > >> >
> > > > > > > >> > hi folks,
> > > > > > > >> >
> > > > > > > >> > There have been a number of critical issues reported (many
> > of
> > > > them
> > > > > > > >> > fixed already) since 0.16.0 was released. Is there interest
> > in
> > > > > > > >> > preparing a patch 0.16.1 release (with backported patches
> > onto a
> > > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next major
> > > > release is a
> > > > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > > > >> >
> > > > > > > >> > Did the 0.15.1 patch release helper script that Krisztian
> > wrote
> > > > get
> > > > > > > >> > contributed as a PR?
> > > > > > > >> Not yet, but it is available at
> > > > > > > >>
> > https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > > >> >
> > > > > > > >> > Thanks
> > > > > > > >> > Wes
> > > > > > > >>
> > > > > > > >
> > > >
> >
Re: Preparing for 0.17.0 Arrow release
Posted by Andy Grove <an...@gmail.com>.
I just took a first pass at reviewing the Java and Rust issues and removed
some from the 0.17.0 release. There are a few small Rust issues that I am
actively working on for this release.
Thanks.
On Wed, Mar 25, 2020 at 1:13 PM Wes McKinney <we...@gmail.com> wrote:
> hi Neal,
>
> Thanks for helping coordinate. I agree we should be in a position to
> release sometime next week.
>
> Can folks from the Rust and Java side review issues in the backlog?
> According to the dashboard there are 19 Rust issues open and 7 Java
> issues.
>
> Thanks
>
> On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
> <ne...@gmail.com> wrote:
> >
> > Hi all,
> > A few weeks ago, there seemed to be consensus (lazy, at least) for a 0.17
> > release at the end of the month. Judging from
> > https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release,
> it
> > looks like we're getting closer.
> >
> > I'd encourage everyone to review their backlogs and (1) bump from 0.17
> > scope any tickets they don't plan to finish this week, and (2) if there
> are
> > any issues that should block release, make sure they are flagged as
> > "blockers".
> >
> > Neal
> >
> > On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > It seems like the consensus is to push for a 0.17.0 major release
> > > sooner rather than doing a patch release, since releases in general
> > > are costly. This is fine with me. I see that a 0.17.0 milestone has
> > > been created in JIRA and some JIRA gardening has begun. Do you think
> > > we can be in a position to release by the week of March 23 or the week
> > > of March 30?
> > >
> > > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > >
> > > > If people are generally on board with accelerating a 0.17.0 major
> > > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > > beginning to do issue gardening to whittle things down to
> > > > critical-looking bugs and high probability patches for the next
> couple
> > > > of weeks.
> > > >
> > > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > I recall there are some other issues that have been reported or
> fixed
> > > > > that are critical and not yet marked with 0.16.1.
> > > > >
> > > > > I'm also OK with doing a 0.17.0 release sooner
> > > > >
> > > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > > <ne...@gmail.com> wrote:
> > > > > >
> > > > > > I would also be more supportive of doing 0.17 earlier instead of
> a
> > > patch
> > > > > > release.
> > > > > >
> > > > > > Neal
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > > neal.p.richardson@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > If releases were costless to make, I'd be all for it, but it's
> not
> > > clear
> > > > > > > to me that it's worth the diversion from other priorities to
> make
> > > a release
> > > > > > > right now. Nothing on
> > > > > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > > jumps out to me as super urgent--what are you seeing as
> critical?
> > > > > > >
> > > > > > > If we did decide to go forward, would it be possible to do a
> > > release that
> > > > > > > is limited to the affected implementations (say, do a
> Python-only
> > > release)?
> > > > > > > That might reduce the cost of building and verifying enough to
> > > make it
> > > > > > > reasonable to consider.
> > > > > > >
> > > > > > > Neal
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > > szucs.krisztian@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > wrote:
> > > > > > >> >
> > > > > > >> > hi folks,
> > > > > > >> >
> > > > > > >> > There have been a number of critical issues reported (many
> of
> > > them
> > > > > > >> > fixed already) since 0.16.0 was released. Is there interest
> in
> > > > > > >> > preparing a patch 0.16.1 release (with backported patches
> onto a
> > > > > > >> > maint-0.16.x branch as with 0.15.1) since the next major
> > > release is a
> > > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > > >> >
> > > > > > >> > Did the 0.15.1 patch release helper script that Krisztian
> wrote
> > > get
> > > > > > >> > contributed as a PR?
> > > > > > >> Not yet, but it is available at
> > > > > > >>
> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > > >> >
> > > > > > >> > Thanks
> > > > > > >> > Wes
> > > > > > >>
> > > > > > >
> > >
>
Re: Preparing for 0.17.0 Arrow release
Posted by Wes McKinney <we...@gmail.com>.
hi Neal,
Thanks for helping coordinate. I agree we should be in a position to
release sometime next week.
Can folks from the Rust and Java side review issues in the backlog?
According to the dashboard there are 19 Rust issues open and 7 Java
issues.
Thanks
On Tue, Mar 24, 2020 at 10:01 AM Neal Richardson
<ne...@gmail.com> wrote:
>
> Hi all,
> A few weeks ago, there seemed to be consensus (lazy, at least) for a 0.17
> release at the end of the month. Judging from
> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.17.0+Release, it
> looks like we're getting closer.
>
> I'd encourage everyone to review their backlogs and (1) bump from 0.17
> scope any tickets they don't plan to finish this week, and (2) if there are
> any issues that should block release, make sure they are flagged as
> "blockers".
>
> Neal
>
> On Tue, Mar 10, 2020 at 7:39 AM Wes McKinney <we...@gmail.com> wrote:
>
> > It seems like the consensus is to push for a 0.17.0 major release
> > sooner rather than doing a patch release, since releases in general
> > are costly. This is fine with me. I see that a 0.17.0 milestone has
> > been created in JIRA and some JIRA gardening has begun. Do you think
> > we can be in a position to release by the week of March 23 or the week
> > of March 30?
> >
> > On Thu, Mar 5, 2020 at 8:39 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > If people are generally on board with accelerating a 0.17.0 major
> > > release, then I would suggest renaming "1.0.0" to "0.17.0" and
> > > beginning to do issue gardening to whittle things down to
> > > critical-looking bugs and high probability patches for the next couple
> > > of weeks.
> > >
> > > On Thu, Mar 5, 2020 at 11:31 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > I recall there are some other issues that have been reported or fixed
> > > > that are critical and not yet marked with 0.16.1.
> > > >
> > > > I'm also OK with doing a 0.17.0 release sooner
> > > >
> > > > On Thu, Mar 5, 2020 at 11:31 AM Neal Richardson
> > > > <ne...@gmail.com> wrote:
> > > > >
> > > > > I would also be more supportive of doing 0.17 earlier instead of a
> > patch
> > > > > release.
> > > > >
> > > > > Neal
> > > > >
> > > > >
> > > > > On Thu, Mar 5, 2020 at 9:29 AM Neal Richardson <
> > neal.p.richardson@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > If releases were costless to make, I'd be all for it, but it's not
> > clear
> > > > > > to me that it's worth the diversion from other priorities to make
> > a release
> > > > > > right now. Nothing on
> > > > > >
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.16.1
> > > > > > jumps out to me as super urgent--what are you seeing as critical?
> > > > > >
> > > > > > If we did decide to go forward, would it be possible to do a
> > release that
> > > > > > is limited to the affected implementations (say, do a Python-only
> > release)?
> > > > > > That might reduce the cost of building and verifying enough to
> > make it
> > > > > > reasonable to consider.
> > > > > >
> > > > > > Neal
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 5, 2020 at 8:19 AM Krisztián Szűcs <
> > szucs.krisztian@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> On Thu, Mar 5, 2020 at 5:07 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > > > >> >
> > > > > >> > hi folks,
> > > > > >> >
> > > > > >> > There have been a number of critical issues reported (many of
> > them
> > > > > >> > fixed already) since 0.16.0 was released. Is there interest in
> > > > > >> > preparing a patch 0.16.1 release (with backported patches onto a
> > > > > >> > maint-0.16.x branch as with 0.15.1) since the next major
> > release is a
> > > > > >> > minimum of 6-8 weeks away from general availability?
> > > > > >> >
> > > > > >> > Did the 0.15.1 patch release helper script that Krisztian wrote
> > get
> > > > > >> > contributed as a PR?
> > > > > >> Not yet, but it is available at
> > > > > >> https://gist.github.com/kszucs/b2743546044ccd3215e5bb34fa0d76a0
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> > Wes
> > > > > >>
> > > > > >
> >