You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Wes McKinney <we...@gmail.com> on 2019/08/14 03:26:23 UTC
Timeline for 0.15.0 release
hi folks,
Since there have been a number of fairly serious issues (e.g.
ARROW-6060) since 0.14.1 that have been fixed I think we should start
planning of the next major release. Note that we still have some
format-related work (the Flatbuffers alignment issue) that ought to be
resolved (not a small task since it affects 4 or 5 implementations),
but aside from that, is there anything else that has come up that
definitely needs to happen before we can release again?
I would say cutting a release somewhere around the US Labor Day
holiday (~the week after or so) would be called for.
Thanks,
Wes
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
Is there a JIRA for the issue that caused us to pull the 0.14.1
Windows Python wheel installers? If we want to have working wheels for
0.15.0 we'll need a volunteer to help address whatever was wrong with
0.14.1.
On Tue, Aug 13, 2019 at 10:26 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi folks,
>
> Since there have been a number of fairly serious issues (e.g.
> ARROW-6060) since 0.14.1 that have been fixed I think we should start
> planning of the next major release. Note that we still have some
> format-related work (the Flatbuffers alignment issue) that ought to be
> resolved (not a small task since it affects 4 or 5 implementations),
> but aside from that, is there anything else that has come up that
> definitely needs to happen before we can release again?
>
> I would say cutting a release somewhere around the US Labor Day
> holiday (~the week after or so) would be called for.
>
> Thanks,
> Wes
Re: Timeline for 0.15.0 release
Posted by Krisztián Szűcs <sz...@gmail.com>.
There are still missing linux artifacts [1]:
- for amd64 debug symbol packages
- for arm64 optional CUDA, plasma and Gandiva modules
I think we can safely ignore them for the release, crossbow will
report them as missing but the artifact downloading step finish.
Let me know Micah if you have any issues.
[1]: https://github.com/apache/arrow/pull/5506#issuecomment-535495351
On Thu, Sep 26, 2019 at 3:38 PM Micah Kornfield <em...@gmail.com>
wrote:
> Yes, I merged it and it will be included. I needed to start over due to a
> cross-bow issue...
>
> On Thu, Sep 26, 2019 at 7:18 AM Ji Liu <ni...@aliyun.com> wrote:
>
>> Hi Micah,
>> Hmm, unfortunately, I just found a bug in JDBC adapter and open a
>> PR, could this change catch up with 0.15?
>> See https://github.com/apache/arrow/pull/5511
>>
>>
>> Thanks,
>> Ji Liu
>>
>>
>> ------------------------------------------------------------------
>> From:Micah Kornfield <em...@gmail.com>
>> Send Time:2019年9月26日(星期四) 14:23
>> To:Neal Richardson <ne...@gmail.com>
>> Cc:"Krisztián Szűcs" <sz...@gmail.com>; Wes McKinney <
>> wesmckinn@gmail.com>; dev <de...@arrow.apache.org>
>> Subject:Re: Timeline for 0.15.0 release
>>
>> Just an I've started the RC generation process off, the last commit from
>> master is [1]
>>
>> I am currently waiting the crossbow builds (build-690 on
>> ursa-labs/crossbow). I think this will take a little while so I will pick
>> it up tomorrow (Thursday).
>>
>> Thanks,
>> Micah
>>
>> [1]
>>
>> https://github.com/apache/arrow/commit/07ab5083d5a2925ced6f8168b60b8fa336f4eccc
>>
>> On Wed, Sep 25, 2019 at 2:07 PM Neal Richardson <
>> neal.p.richardson@gmail.com>
>> wrote:
>>
>> > IMO it's too risky to add something that adds a dependency
>> > (aws-sdk-cpp) on the day of cutting a release.
>> >
>> > Neal
>> >
>> > On Wed, Sep 25, 2019 at 12:54 PM Krisztián Szűcs
>> > <sz...@gmail.com> wrote:
>> > >
>> > > We don't have a comprehensive documentation yet, so let's postpone it.
>> > >
>> > >
>> > > On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <
>> > szucs.krisztian@gmail.com> wrote:
>> > >>
>> > >> The S3 python bindings would be a nice addition to the release.
>> > >> I don't think we should block on this but the PR is ready. Opinions?
>> > >> https://github.com/apache/arrow/pull/5423
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <
>> emkornfield@gmail.com>
>> > wrote:
>> > >>>
>> > >>> OK, I'll start the process today. I'll send up e-mail updates as I
>> > make progress.
>> > >>>
>> > >>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com>
>> > wrote:
>> > >>>>
>> > >>>> Yes, all systems go as far as I'm concerned.
>> > >>>>
>> > >>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
>> > >>>> <ne...@gmail.com> wrote:
>> > >>>> >
>>
>> > >>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
>>
>> > >>>> > and it looks like the LICENSE issue is being resolved as I type. So
>> > >>>> > are we good to go now?
>> > >>>> >
>> > >>>> > Neal
>> > >>>> >
>> > >>>> >
>> > >>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <
>> andygrove73@gmail.com>
>> > wrote:
>> > >>>> > >
>> > >>>> > > I found a last minute issue with DataFusion (Rust) and would
>> > appreciate it
>> > >>>> > > if we could merge ARROW-6086 (PR is
>> > >>>> > > https://github.com/apache/arrow/pull/5494
>> ) before cutting the RC.
>> > >>>> > >
>> > >>>> > > Thanks,
>> > >>>> > >
>> > >>>> > > Andy.
>> > >>>> > >
>> > >>>> > >
>> > >>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >>>> > > wrote:
>> > >>>> > >
>> > >>>> > > > OK, I'm going to postpone cutting a release until tomorrow
>> > (hoping we can
>> > >>>> > > > issues resolved by then).. I'll also try to review the
>> > third-party
>> > >>>> > > > additions since 14.x.
>> > >>>> > > >
>> > >>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <
>> > wesmckinn@gmail.com> wrote:
>> > >>>> > > >
>> > >>>> > > > > I found a licensing issue
>> > >>>> > > > >
>> > >>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
>> > >>>> > > > >
>> > >>>> > > > > It might be worth examining third party code added to the
>> > project
>> > >>>> > > > > since 0.14.x to make sure there are no other such issues.
>> > >>>> > > > >
>> > >>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <
>> > wesmckinn@gmail.com>
>> > >>>> > > > wrote:
>> > >>>> > > > > >
>>
>> > >>>> > > > > > I have diagnosed the problem (Thrift "string" data must be
>> > UTF-8,
>>
>> > >>>> > > > > > cannot be arbitrary binary) and am working on a patch right
>> > now
>> > >>>> > > > > >
>> > >>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
>> > wesmckinn@gmail.com>
>> > >>>> > > > > wrote:
>> > >>>> > > > > > >
>> > >>>> > > > > > > I just opened
>> > >>>> > > > > > >
>> > >>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
>> > >>>> > > > > > >
>> > >>>> > > > > > > Please don't cut an RC until I have an opportunity to
>> > diagnose this,
>> > >>>> > > > > > > will report back.
>> > >>>> > > > > > >
>> > >>>> > > > > > >
>> > >>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
>> > wesmckinn@gmail.com>
>> > >>>> > > > > wrote:
>> > >>>> > > > > > > >
>> > >>>> > > > > > > > I'm investigating a possible Parquet-related
>> > compatibility bug
>> > >>>> > > > that I
>> > >>>> > > > > > > > encountered through some routine testing /
>> > benchmarking. I'll
>> > >>>> > > > report
>> > >>>> > > > > > > > back once I figure out what is going on (if anything)
>> > >>>> > > > > > > >
>> > >>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
>> > >>>> > > > > emkornfield@gmail.com> wrote:
>> > >>>> > > > > > > > >>
>> > >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
>> > (i.e. you can
>> > >>>> > > > > get it
>> > >>>> > > > > > > > >> signed by another PMC member), but is not 100%
>> > essential.
>> > >>>> > > > > > > > >
>> > >>>> > > > > > > > > That won't be an option for me this week (it seems
>> > like I would
>> > >>>> > > > > need to meet one face-to-face). I'll try to get the GPG
>> > checked in and
>> > >>>> > > > the
>> > >>>> > > > > rest of the pre-requisites done tomorrow (Monday) to
>> > hopefully start the
>> > >>>> > > > > release on Tuesday (hopefully we can solve the last
>> > blocker/integration
>> > >>>> > > > > tests by then).
>> > >>>> > > > > > > > >
>> > >>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
>> > >>>> > > > wesmckinn@gmail.com>
>> > >>>> > > > > wrote:
>> > >>>> > > > > > > > >>
>> > >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
>> > (i.e. you can
>> > >>>> > > > > get it
>> > >>>> > > > > > > > >> signed by another PMC member), but is not 100%
>> > essential.
>> > >>>> > > > > > > > >>
>> > >>>> > > > > > > > >> Speaking of the release, there are at least 2 code
>> > changes I
>> > >>>> > > > still
>> > >>>> > > > > > > > >> want to get in
>> > >>>> > > > > > > > >>
>> > >>>> > > > > > > > >> ARROW-5717
>> > >>>> > > > > > > > >> ARROW-6353
>> > >>>> > > > > > > > >>
>>
>> > >>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once
>> > the build
>> > >>>> > > > is
>> > >>>> > > > > green.
>> > >>>> > > > > > > > >>
>>
>> > >>>> > > > > > > > >> There are a couple of Rust patches still marked for
>> > 0.15. The
>> > >>>> > > > rest
>> > >>>> > > > > > > > >> seems to be documentation and a couple of
>> > integration test
>> > >>>> > > > > failures we
>> > >>>> > > > > > > > >> should see about fixing in time.
>> > >>>> > > > > > > > >>
>> > >>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
>> > >>>> > > > > emkornfield@gmail.com> wrote:
>> > >>>> > > > > > > > >> >
>> > >>>> > > > > > > > >> > Thanks Krisztián and Wes,
>>
>> > >>>> > > > > > > > >> > I've gone ahead and started registering myself on
>> > all the
>> > >>>> > > > > packaging sites.
>> > >>>> > > > > > > > >> >
>>
>> > >>>> > > > > > > > >> > Is there any review process when adding my GPG key
>> > to the SVN
>> > >>>> > > > > file? [1]
>> > >>>> > > > > > > > >> > doesn't seem to mention explicitly.
>> > >>>> > > > > > > > >> >
>> > >>>> > > > > > > > >> > Thanks,
>> > >>>> > > > > > > > >> > Micah
>> > >>>> > > > > > > > >> >
>> > >>>> > > > > > > > >> > [1]
>> > https://www.apache.org/dev/version-control.html#https-svn
>> > >>>> > > > > > > > >> >
>>
>> > >>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
>> > >>>> > > > > szucs.krisztian@gmail.com>
>> > >>>> > > > > > > > >> > wrote:
>> > >>>> > > > > > > > >> >
>> > >>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
>> > >>>> > > > > wesmckinn@gmail.com> wrote:
>> > >>>> > > > > > > > >> > >
>> > >>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah
>> > Kornfield <
>> > >>>> > > > > emkornfield@gmail.com>
>> > >>>> > > > > > > > >> > >> wrote:
>> > >>>> > > > > > > > >> > >> >>
>> > >>>> > > > > > > > >> > >> >> The process should be well documented at
>> > this point but
>> > >>>> > > > > there are a
>> > >>>> > > > > > > > >> > >> >> number of steps.
>> > >>>> > > > > > > > >> > >> >
>> > >>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
>> > release?
>> > >>>> > > > Are
>> > >>>> > > > > there
>> > >>>> > > > > > > > >> > >> instructions for the adding the code signing
>> > Key to SVN?
>> > >>>> > > > > > > > >> > >> >
>> > >>>> > > > > > > > >> > >> > I will make a go of it. i will try to
>> > mitigate any
>> > >>>> > > > > internet issues by
>> > >>>> > > > > > > > >> > >> doing the process for a cloud instance (I
>> > assume that isn't
>> > >>>> > > > > a problem?).
>> > >>>> > > > > > > > >> > >> >
>> > >>>> > > > > > > > >> > >>
>>
>> > >>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
>> > producing
>> > >>>> > > > an
>> > >>>> > > > > RC may be
>> > >>>> > > > > > > > >> > >> time consuming, but you are welcome to try.
>> > Krisztian --
>> > >>>> > > > are
>> > >>>> > > > > you
>> > >>>> > > > > > > > >> > >> available next week to help Micah and
>> > potentially take over
>> > >>>> > > > > producing
>> > >>>> > > > > > > > >> > >> the RC if there are issues?
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > > > > > >> > > Sure, I'll be available next week. We can also
>> > grant access
>> > >>>> > > > to
>> > >>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
>> > configuring
>> > >>>> > > > all
>> > >>>> > > > > > > > >> > > the CI backends can be time consuming.
>> > >>>> > > > > > > > >> > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > > > > > >> > >> > Thanks,
>> > >>>> > > > > > > > >> > >> > Micah
>> > >>>> > > > > > > > >> > >> >
>> > >>>> > > > > > > > >> > >> > [1]
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > >>>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>> > >>>> > > > > > > > >> > >> >
>>
>> > >>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
>> > >>>> > > > > wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> wrote:
>> > >>>> > > > > > > > >> > >> >>
>> > >>>> > > > > > > > >> > >> >> The process should be well documented at
>> > this point but
>> > >>>> > > > > there are a
>> > >>>> > > > > > > > >> > >> >> number of steps. Note that you need to add
>> > your code
>> > >>>> > > > > signing key to
>> > >>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard
>> > to do). I
>> > >>>> > > > > think it's fine
>>
>> > >>>> > > > > > > > >> > >> >> to hand off the process to others after the
>> > VOTE but it
>> > >>>> > > > > would be
>> > >>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
>> > producing the
>> > >>>> > > > > source and
>> > >>>> > > > > > > > >> > >> >> binary artifacts for the vote
>> > >>>> > > > > > > > >> > >> >>
>> > >>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah
>> > Kornfield <
>> > >>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > SGTM, as well.
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > I should have a little bit of time next
>> > week if I can
>> > >>>> > > > > help as RM but
>> > >>>> > > > > > > > >> > >> I have
>> > >>>> > > > > > > > >> > >> >> > a couple of concerns:
>> > >>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble
>> > downloading and
>> > >>>> > > > > validating
>> > >>>> > > > > > > > >> > >> releases. I'm a
>> > >>>> > > > > > > > >> > >> >> > bit worried, that I might have similar
>> > problems doing
>> > >>>> > > > > the necessary
>> > >>>> > > > > > > > >> > >> uploads.
>>
>> > >>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be
>> > not great, I
>> > >>>> > > > > don't know if
>> > >>>> > > > > > > > >> > >> this
>> > >>>> > > > > > > > >> > >> >> > would make it even less likely to be
>> > successful.
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I
>> > would have to
>> > >>>> > > > > abandon the
>> > >>>> > > > > > > > >> > >> process
>> > >>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could
>> > serve as a
>> > >>>> > > > > backup? Are the
>> > >>>> > > > > > > > >> > >> steps
>> > >>>> > > > > > > > >> > >> >> > well documented?
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > Thanks,
>> > >>>> > > > > > > > >> > >> >> > Micah
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
>> > Richardson <
>> > >>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > wrote:
>> > >>>> > > > > > > > >> > >> >> >
>> > >>>> > > > > > > > >> > >> >> > > Sounds good to me.
>> > >>>> > > > > > > > >> > >> >> > >
>> > >>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
>> > volunteers?
>> > >>>> > > > > > > > >> > >> >> > >
>> > >>>> > > > > > > > >> > >> >> > > Neal
>> > >>>> > > > > > > > >> > >> >> > >
>> > >>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes
>> > McKinney <
>> > >>>> > > > > wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> wrote:
>> > >>>> > > > > > > > >> > >> >> > >
>> > >>>> > > > > > > > >> > >> >> > > > hi all,
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to
>> > be able to
>> > >>>> > > > > make the 0.15.0
>> > >>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils
>> > down" at the end
>> > >>>> > > > > of this week
>> > >>>> > > > > > > > >> > >> and
>> > >>>> > > > > > > > >> > >> >> > > > see if a release candidate can be
>> > produced next
>> > >>>> > > > > Monday September
>> > >>>> > > > > > > > >> > >> 23.
>> > >>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >> >> > > > Thanks,
>> > >>>> > > > > > > > >> > >> >> > > > Wes
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
>> > McKinney <
>> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
>> > planning to
>> > >>>> > > > > amend the
>> > >>>> > > > > > > > >> > >> Format docs
>> > >>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and
>> > also update
>> > >>>> > > > the
>> > >>>> > > > > C++ library
>> > >>>> > > > > > > > >> > >> >> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM
>> > Eric Erhardt
>> > >>>> > > > > > > > >> > >> >> > > > > <Eric.Erhardt@microsoft.com
>> > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
>> > >>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
>> > >>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15
>> > release,
>> > >>>> > > > correct?
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
>> > changes are
>> > >>>> > > > > ready to be
>> > >>>> > > > > > > > >> > >> merged into
>> > >>>> > > > > > > > >> > >> >> > > > the alignment branch -
>> > >>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > Eric
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
>> > >>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
>> > emkornfield@gmail.com>
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
>> > 10:24 PM
>> > >>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <
>> > wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
>> > niki.lj <
>> > >>>> > > > > niki.lj@aliyun.com>
>> > >>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
>> > release
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > I should have a little more
>> > bandwidth to help
>> > >>>> > > > > with some of
>> > >>>> > > > > > > > >> > >> the
>>
>> > >>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going
>> > into the
>> > >>>> > > > > weekend.
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019,
>> > Wes McKinney <
>> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
>> > >>>> > > > > > > > >> > >> >> > > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly
>> > packaging and
>> > >>>> > > > > integration builds
>> > >>>> > > > > > > > >> > >> things
>> > >>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for
>> > being in release
>> > >>>> > > > > readiness by
>> > >>>> > > > > > > > >> > >> the end
>> > >>>> > > > > > > > >> > >> >> > > of
>> > >>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong.
>> > I'm planning
>> > >>>> > > > > to be working
>> > >>>> > > > > > > > >> > >> to close
>> > >>>> > > > > > > > >> > >> >> > > as
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
>> > help with
>> > >>>> > > > > the ongoing
>> > >>>> > > > > > > > >> > >> alignment
>> > >>>> > > > > > > > >> > >> >> > > > fixes.
>> > >>>> > > > > > > > >> > >> >> > > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > > Wes
>> > >>>> > > > > > > > >> > >> >> > > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM
>> > Micah
>> > >>>> > > > Kornfield
>> > >>>> > > > > <
>> > >>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
>> > >>>> > > > > > > > >> > >> >> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
>> > dashboard of
>> > >>>> > > > > the current
>> > >>>> > > > > > > > >> > >> issues:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> > >>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > se&data=02%7C01%7CEric.Erhardt%
>> > >>>> > > > > 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> > >>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM
>> > Wes
>> > >>>> > > > McKinney <
>> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're
>> > going to be in
>> > >>>> > > > a
>> > >>>> > > > > position to
>> > >>>> > > > > > > > >> > >> release
>> > >>>> > > > > > > > >> > >> >> > > at
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I
>> > hope that
>> > >>>> > > > one
>> > >>>> > > > > more week of
>> > >>>> > > > > > > > >> > >> work (or
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
>> > there.
>> > >>>> > > > Aside
>> > >>>> > > > > from merging
>> > >>>> > > > > > > > >> > >> the
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to
>> > make sure
>> > >>>> > > > > that our
>> > >>>> > > > > > > > >> > >> packaging jobs
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> required for the release
>> > candidate are all
>> > >>>> > > > > working.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues
>> > from the
>> > >>>> > > > > 0.15.0 backlog
>> > >>>> > > > > > > > >> > >> that they
>> > >>>> > > > > > > > >> > >> >> > > > don't
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end
>> > of next week
>> > >>>> > > > > that would
>> > >>>> > > > > > > > >> > >> help focus
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently
>> > 78 issues in
>> > >>>> > > > > 0.15.0 still).
>> > >>>> > > > > > > > >> > >> I am
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
>> > features
>> > >>>> > > > > related to
>> > >>>> > > > > > > > >> > >> dictionaries
>> > >>>> > > > > > > > >> > >> >> > > > while
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still
>> > open.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48
>> > PM Wes
>> > >>>> > > > > McKinney <
>> > >>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to
>> > release the
>> > >>>> > > > week
>> > >>>> > > > > of September
>> > >>>> > > > > > > > >> > >> 9, so
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
>> > completed by
>> > >>>> > > > > end of next
>> > >>>> > > > > > > > >> > >> week.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for
>> > the
>> > >>>> > > > protocol
>> > >>>> > > > > alignment
>> > >>>> > > > > > > > >> > >> changes for
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of
>> > days -- I
>> > >>>> > > > think
>> > >>>> > > > > that getting
>> > >>>> > > > > > > > >> > >> the
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the
>> > main barrier
>> > >>>> > > > > to releasing.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at
>> > 12:25 PM Ji Liu
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > <niki.lj@aliyun.com.invalid
>> >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side,
>> > I can think
>> > >>>> > > > > of several
>> > >>>> > > > > > > > >> > >> bugs that
>> > >>>> > > > > > > > >> > >> >> > > > need
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > to
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
>> > entries are
>> > >>>> > > > > required in
>> > >>>> > > > > > > > >> > >> IPC streams
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > even
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review
>> > now, however
>> > >>>> > > > > through this
>> > >>>> > > > > > > > >> > >> PR we find
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > that
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java
>> > reading and
>> > >>>> > > > > writing
>> > >>>> > > > > > > > >> > >> dictionaries in IPC
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with
>> > spec[2] since
>> > >>>> > > > it
>> > >>>> > > > > assumes all
>> > >>>> > > > > > > > >> > >> >> > > > dictionaries
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream
>> > (see details in
>> > >>>> > > > > PR comments,
>> > >>>> > > > > > > > >> > >> and this
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with
>> > version 0.15).
>> > >>>> > > > > @Micah Kornfield
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write
>> > 64-bit ints as
>> > >>>> > > > > strings in
>> > >>>> > > > > > > > >> > >> integration
>> > >>>> > > > > > > > >> > >> >> > > > test
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already
>> > checked in,
>> > >>>> > > > other
>> > >>>> > > > > > > > >> > >> implementations
>> > >>>> > > > > > > > >> > >> >> > > seems
>> > >>>> > > > > > > > >> > >> >> > > > not.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202:
>> > OutOfMemory in
>> > >>>> > > > > JdbcAdapter[4]
>> > >>>> > > > > > > > >> > >> Caused by
>> > >>>> > > > > > > > >> > >> >> > > trying
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
>> > contiguous
>> > >>>> > > > > batch, fixed
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
>> > iteratively
>> > >>>> > > > > reading in
>> > >>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%
>> > 40microsoft.com
>>
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
>> > 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
>> > 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > >
>> > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > ----------------------------------------------------------------
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
>> > >>>> > > > > wesmckinn@gmail.com> Send
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
>> > To:dev <
>> > >>>> > > > > > > > >> > >> dev@arrow.apache.org>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for
>> > 0.15.0
>> > >>>> > > > release
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
>> > organizing
>> > >>>> > > > > the 0.15.0
>> > >>>> > > > > > > > >> > >> backlog some
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants
>> > to help
>> > >>>> > > > with
>> > >>>> > > > > grooming
>> > >>>> > > > > > > > >> > >> >> > > (particularly
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
>> > C++/Python
>> > >>>> > > > > where I'm
>> > >>>> > > > > > > > >> > >> focusing) that
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There
>> > have been
>> > >>>> > > > > almost 500 JIRA
>> > >>>> > > > > > > > >> > >> issues
>> > >>>> > > > > > > > >> > >> >> > > opened
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we
>> > should make sure
>> > >>>> > > > > to check
>> > >>>> > > > > > > > >> > >> whether
>> > >>>> > > > > > > > >> > >> >> > > there's
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
>> > serious bugs
>> > >>>> > > > > that we should
>> > >>>> > > > > > > > >> > >> try to
>> > >>>> > > > > > > > >> > >> >> > > fix
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at
>> > 6:23 PM Wes
>> > >>>> > > > > McKinney
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue
>> > in 0.14.1
>> > >>>> > > > > seems to be
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
>> > >>>> > > > > > > > >> > >> >> > > >
>> > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
>> > >>>> > > > 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause
>> > could be the
>> > >>>> > > > > Windows
>> > >>>> > > > > > > > >> > >> changes in
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
>> > 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > 223ae744cc2a12c60cecb5db593263a03c13f85a
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative
>> > if a
>> > >>>> > > > > volunteer would look
>> > >>>> > > > > > > > >> > >> into what
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrong
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels
>> > on Windows.
>> > >>>> > > > > Otherwise
>> > >>>> > > > > > > > >> > >> 0.15.0 Windows
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken,
>> > too
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be
>> > found at
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > >
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>> > >>>> > > > > > > > >> > >> 40microsoft.com
>> > >>>> > > > > > > > >> > >> >> > > > %7Ccbea
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > > > > >> > >> >> > > >
>>
>> > >>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > >>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at
>> > 1:28 PM
>> > >>>> > > > > Antoine Pitrou <
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
>> > 11:17:07 -0700
>> > >>>> > > > > Micah
>> > >>>> > > > > > > > >> > >> Kornfield
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <
>> emkornfield@gmail.com>
>> > wrote:
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we
>> > could have
>> > >>>> > > > > 32-bit array
>> > >>>> > > > > > > > >> > >> lengths and
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
>> > offsets if
>> > >>>> > > > we
>> > >>>> > > > > wanted (we
>> > >>>> > > > > > > > >> > >> just
>> > >>>> > > > > > > > >> > >> >> > > > wouldn't
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> able to
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child
>> > with more
>> > >>>> > > > > than INT32_MAX
>> > >>>> > > > > > > > >> > >> elements).
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is
>> > we could do
>> > >>>> > > > > this in C++
>> > >>>> > > > > > > > >> > >> but we
>> > >>>> > > > > > > > >> > >> >> > > > don't.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have
>> > introduced the
>> > >>>> > > > "Large"
>> > >>>> > > > > types if we
>> > >>>> > > > > > > > >> > >> did.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take
>> > twice as much
>> > >>>> > > > > space as 32-bit
>> > >>>> > > > > > > > >> > >> >> > > offsets,
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> you're
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of
>> > small-ish lists or
>> > >>>> > > > > strings,
>> > >>>> > > > > > > > >> > >> 32-bit
>> > >>>> > > > > > > > >> > >> >> > > offsets
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So
>> > even with
>> > >>>> > > > > 64-bit array
>> > >>>> > > > > > > > >> > >> lengths from
>> > >>>> > > > > > > > >> > >> >> > > > the
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> it would
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
>> > have types
>> > >>>> > > > > with 32-bit
>> > >>>> > > > > > > > >> > >> offsets.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the
>> > limited address
>> > >>>> > > > > space in Java
>> > >>>> > > > > > > > >> > >> and
>> > >>>> > > > > > > > >> > >> >> > > calling
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> reference
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
>> > suboptimal.
>> > >>>> > > > > If a consumer
>> > >>>> > > > > > > > >> > >> uses a
>> > >>>> > > > > > > > >> > >> >> > > > "Large"
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> type
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is
>> > because they
>> > >>>> > > > > need the ability
>> > >>>> > > > > > > > >> > >> to store
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
>> > column,
>> > >>>> > > > > otherwise it is
>> > >>>> > > > > > > > >> > >> just
>> > >>>> > > > > > > > >> > >> >> > > wasting
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> [1].
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if
>> > the individual
>> > >>>> > > > > elements
>> > >>>> > > > > > > > >> > >> (lists or
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> are
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space
>> > is wasted in
>> > >>>> > > > > proportion,
>> > >>>> > > > > > > > >> > >> so it may
>> > >>>> > > > > > > > >> > >> >> > > be
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
>> > create a
>> > >>>> > > > > "Large" type
>> > >>>> > > > > > > > >> > >> array.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
>> > theoretically
>> > >>>> > > > there
>> > >>>> > > > > might be some
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures
>> > to using
>> > >>>> > > > the
>> > >>>> > > > > native word
>> > >>>> > > > > > > > >> > >> sizes.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common
>> > 64-bit
>> > >>>> > > > > architectures don't do
>> > >>>> > > > > > > > >> > >> that, as
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> is an
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common
>> > integer size even
>> > >>>> > > > > in
>> > >>>> > > > > > > > >> > >> high-performance
>> > >>>> > > > > > > > >> > >> >> > > > code.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > >>>> > > > > > > > >> > >> >> > > > > > >>>
>> > >>>> > > > > > > > >> > >> >> > > > > > >>
>> > >>>> > > > > > > > >> > >> >> > > >
>> > >>>> > > > > > > > >> > >> >> > >
>> > >>>> > > > > > > > >> > >>
>> > >>>> > > > > > > > >> > >
>> > >>>> > > > >
>> > >>>> > > >
>> >
>>
>>
>>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
Yes, I merged it and it will be included. I needed to start over due to a
cross-bow issue...
On Thu, Sep 26, 2019 at 7:18 AM Ji Liu <ni...@aliyun.com> wrote:
> Hi Micah,
> Hmm, unfortunately, I just found a bug in JDBC adapter and open a
> PR, could this change catch up with 0.15?
> See https://github.com/apache/arrow/pull/5511
>
>
> Thanks,
> Ji Liu
>
>
> ------------------------------------------------------------------
> From:Micah Kornfield <em...@gmail.com>
> Send Time:2019年9月26日(星期四) 14:23
> To:Neal Richardson <ne...@gmail.com>
> Cc:"Krisztián Szűcs" <sz...@gmail.com>; Wes McKinney <
> wesmckinn@gmail.com>; dev <de...@arrow.apache.org>
> Subject:Re: Timeline for 0.15.0 release
>
> Just an I've started the RC generation process off, the last commit from
> master is [1]
>
> I am currently waiting the crossbow builds (build-690 on
> ursa-labs/crossbow). I think this will take a little while so I will pick
> it up tomorrow (Thursday).
>
> Thanks,
> Micah
>
> [1]
>
> https://github.com/apache/arrow/commit/07ab5083d5a2925ced6f8168b60b8fa336f4eccc
>
> On Wed, Sep 25, 2019 at 2:07 PM Neal Richardson <
> neal.p.richardson@gmail.com>
> wrote:
>
> > IMO it's too risky to add something that adds a dependency
> > (aws-sdk-cpp) on the day of cutting a release.
> >
> > Neal
> >
> > On Wed, Sep 25, 2019 at 12:54 PM Krisztián Szűcs
> > <sz...@gmail.com> wrote:
> > >
> > > We don't have a comprehensive documentation yet, so let's postpone it.
> > >
> > >
> > > On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <
> > szucs.krisztian@gmail.com> wrote:
> > >>
> > >> The S3 python bindings would be a nice addition to the release.
> > >> I don't think we should block on this but the PR is ready. Opinions?
> > >> https://github.com/apache/arrow/pull/5423
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <
> emkornfield@gmail.com>
> > wrote:
> > >>>
> > >>> OK, I'll start the process today. I'll send up e-mail updates as I
> > make progress.
> > >>>
> > >>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>>>
> > >>>> Yes, all systems go as far as I'm concerned.
> > >>>>
> > >>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
> > >>>> <ne...@gmail.com> wrote:
> > >>>> >
>
> > >>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
>
> > >>>> > and it looks like the LICENSE issue is being resolved as I type. So
> > >>>> > are we good to go now?
> > >>>> >
> > >>>> > Neal
> > >>>> >
> > >>>> >
> > >>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <
> andygrove73@gmail.com>
> > wrote:
> > >>>> > >
> > >>>> > > I found a last minute issue with DataFusion (Rust) and would
> > appreciate it
> > >>>> > > if we could merge ARROW-6086 (PR is
> > >>>> > > https://github.com/apache/arrow/pull/5494
> ) before cutting the RC.
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > >
> > >>>> > > Andy.
> > >>>> > >
> > >>>> > >
> > >>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > > > OK, I'm going to postpone cutting a release until tomorrow
> > (hoping we can
> > >>>> > > > issues resolved by then).. I'll also try to review the
> > third-party
> > >>>> > > > additions since 14.x.
> > >>>> > > >
> > >>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <
> > wesmckinn@gmail.com> wrote:
> > >>>> > > >
> > >>>> > > > > I found a licensing issue
> > >>>> > > > >
> > >>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
> > >>>> > > > >
> > >>>> > > > > It might be worth examining third party code added to the
> > project
> > >>>> > > > > since 0.14.x to make sure there are no other such issues.
> > >>>> > > > >
> > >>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > >>>> > > > wrote:
> > >>>> > > > > >
> > >>>> > > > > > I have diagnosed the problem (Thrift "string" data must be
> > UTF-8,
>
> > >>>> > > > > > cannot be arbitrary binary) and am working on a patch right
> > now
> > >>>> > > > > >
> > >>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > >>>> > > > > wrote:
> > >>>> > > > > > >
> > >>>> > > > > > > I just opened
> > >>>> > > > > > >
> > >>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
> > >>>> > > > > > >
> > >>>> > > > > > > Please don't cut an RC until I have an opportunity to
> > diagnose this,
> > >>>> > > > > > > will report back.
> > >>>> > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > >>>> > > > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > I'm investigating a possible Parquet-related
> > compatibility bug
> > >>>> > > > that I
> > >>>> > > > > > > > encountered through some routine testing /
> > benchmarking. I'll
> > >>>> > > > report
> > >>>> > > > > > > > back once I figure out what is going on (if anything)
> > >>>> > > > > > > >
> > >>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> > >>>> > > > > emkornfield@gmail.com> wrote:
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> > (i.e. you can
> > >>>> > > > > get it
> > >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> > essential.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > That won't be an option for me this week (it seems
> > like I would
> > >>>> > > > > need to meet one face-to-face). I'll try to get the GPG
> > checked in and
> > >>>> > > > the
> > >>>> > > > > rest of the pre-requisites done tomorrow (Monday) to
> > hopefully start the
> > >>>> > > > > release on Tuesday (hopefully we can solve the last
> > blocker/integration
> > >>>> > > > > tests by then).
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> > >>>> > > > wesmckinn@gmail.com>
> > >>>> > > > > wrote:
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> > (i.e. you can
> > >>>> > > > > get it
> > >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> > essential.
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> Speaking of the release, there are at least 2 code
> > changes I
> > >>>> > > > still
> > >>>> > > > > > > > >> want to get in
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> ARROW-5717
> > >>>> > > > > > > > >> ARROW-6353
> > >>>> > > > > > > > >>
>
> > >>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once
> > the build
> > >>>> > > > is
> > >>>> > > > > green.
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> There are a couple of Rust patches still marked for
> > 0.15. The
> > >>>> > > > rest
> > >>>> > > > > > > > >> seems to be documentation and a couple of
> > integration test
> > >>>> > > > > failures we
> > >>>> > > > > > > > >> should see about fixing in time.
> > >>>> > > > > > > > >>
> > >>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> > >>>> > > > > emkornfield@gmail.com> wrote:
> > >>>> > > > > > > > >> >
> > >>>> > > > > > > > >> > Thanks Krisztián and Wes,
> > >>>> > > > > > > > >> > I've gone ahead and started registering myself on
> > all the
> > >>>> > > > > packaging sites.
> > >>>> > > > > > > > >> >
>
> > >>>> > > > > > > > >> > Is there any review process when adding my GPG key
> > to the SVN
> > >>>> > > > > file? [1]
> > >>>> > > > > > > > >> > doesn't seem to mention explicitly.
> > >>>> > > > > > > > >> >
> > >>>> > > > > > > > >> > Thanks,
> > >>>> > > > > > > > >> > Micah
> > >>>> > > > > > > > >> >
> > >>>> > > > > > > > >> > [1]
> > https://www.apache.org/dev/version-control.html#https-svn
> > >>>> > > > > > > > >> >
> > >>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> > >>>> > > > > szucs.krisztian@gmail.com>
> > >>>> > > > > > > > >> > wrote:
> > >>>> > > > > > > > >> >
> > >>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> > >>>> > > > > wesmckinn@gmail.com> wrote:
> > >>>> > > > > > > > >> > >
> > >>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah
> > Kornfield <
> > >>>> > > > > emkornfield@gmail.com>
> > >>>> > > > > > > > >> > >> wrote:
> > >>>> > > > > > > > >> > >> >>
> > >>>> > > > > > > > >> > >> >> The process should be well documented at
> > this point but
> > >>>> > > > > there are a
> > >>>> > > > > > > > >> > >> >> number of steps.
> > >>>> > > > > > > > >> > >> >
> > >>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
> > release?
> > >>>> > > > Are
> > >>>> > > > > there
> > >>>> > > > > > > > >> > >> instructions for the adding the code signing
> > Key to SVN?
> > >>>> > > > > > > > >> > >> >
> > >>>> > > > > > > > >> > >> > I will make a go of it. i will try to
> > mitigate any
> > >>>> > > > > internet issues by
> > >>>> > > > > > > > >> > >> doing the process for a cloud instance (I
> > assume that isn't
> > >>>> > > > > a problem?).
> > >>>> > > > > > > > >> > >> >
> > >>>> > > > > > > > >> > >>
>
> > >>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
> > producing
> > >>>> > > > an
> > >>>> > > > > RC may be
> > >>>> > > > > > > > >> > >> time consuming, but you are welcome to try.
> > Krisztian --
> > >>>> > > > are
> > >>>> > > > > you
> > >>>> > > > > > > > >> > >> available next week to help Micah and
> > potentially take over
> > >>>> > > > > producing
> > >>>> > > > > > > > >> > >> the RC if there are issues?
> > >>>> > > > > > > > >> > >>
> > >>>> > > > > > > > >> > > Sure, I'll be available next week. We can also
> > grant access
> > >>>> > > > to
> > >>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
> > configuring
> > >>>> > > > all
> > >>>> > > > > > > > >> > > the CI backends can be time consuming.
> > >>>> > > > > > > > >> > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > > > > > >> > >> > Thanks,
> > >>>> > > > > > > > >> > >> > Micah
> > >>>> > > > > > > > >> > >> >
> > >>>> > > > > > > > >> > >> > [1]
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > >>>> > > >
> >
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > >>>> > > > > > > > >> > >> >
>
> > >>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> > >>>> > > > > wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> wrote:
> > >>>> > > > > > > > >> > >> >>
> > >>>> > > > > > > > >> > >> >> The process should be well documented at
> > this point but
> > >>>> > > > > there are a
> > >>>> > > > > > > > >> > >> >> number of steps. Note that you need to add
> > your code
> > >>>> > > > > signing key to
> > >>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard
> > to do). I
> > >>>> > > > > think it's fine
> > >>>> > > > > > > > >> > >> >> to hand off the process to others after the
> > VOTE but it
> > >>>> > > > > would be
> > >>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
> > producing the
> > >>>> > > > > source and
> > >>>> > > > > > > > >> > >> >> binary artifacts for the vote
> > >>>> > > > > > > > >> > >> >>
> > >>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah
> > Kornfield <
> > >>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > SGTM, as well.
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > I should have a little bit of time next
> > week if I can
> > >>>> > > > > help as RM but
> > >>>> > > > > > > > >> > >> I have
> > >>>> > > > > > > > >> > >> >> > a couple of concerns:
> > >>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble
> > downloading and
> > >>>> > > > > validating
> > >>>> > > > > > > > >> > >> releases. I'm a
> > >>>> > > > > > > > >> > >> >> > bit worried, that I might have similar
> > problems doing
> > >>>> > > > > the necessary
> > >>>> > > > > > > > >> > >> uploads.
> > >>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be
> > not great, I
> > >>>> > > > > don't know if
> > >>>> > > > > > > > >> > >> this
> > >>>> > > > > > > > >> > >> >> > would make it even less likely to be
> > successful.
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I
> > would have to
> > >>>> > > > > abandon the
> > >>>> > > > > > > > >> > >> process
> > >>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could
> > serve as a
> > >>>> > > > > backup? Are the
> > >>>> > > > > > > > >> > >> steps
> > >>>> > > > > > > > >> > >> >> > well documented?
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > Thanks,
> > >>>> > > > > > > > >> > >> >> > Micah
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
> > Richardson <
> > >>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
> > >>>> > > > > > > > >> > >> >> > wrote:
> > >>>> > > > > > > > >> > >> >> >
> > >>>> > > > > > > > >> > >> >> > > Sounds good to me.
> > >>>> > > > > > > > >> > >> >> > >
> > >>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
> > volunteers?
> > >>>> > > > > > > > >> > >> >> > >
> > >>>> > > > > > > > >> > >> >> > > Neal
> > >>>> > > > > > > > >> > >> >> > >
> > >>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes
> > McKinney <
> > >>>> > > > > wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> wrote:
> > >>>> > > > > > > > >> > >> >> > >
> > >>>> > > > > > > > >> > >> >> > > > hi all,
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to
> > be able to
> > >>>> > > > > make the 0.15.0
> > >>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils
> > down" at the end
> > >>>> > > > > of this week
> > >>>> > > > > > > > >> > >> and
> > >>>> > > > > > > > >> > >> >> > > > see if a release candidate can be
> > produced next
> > >>>> > > > > Monday September
> > >>>> > > > > > > > >> > >> 23.
> > >>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >> >> > > > Thanks,
> > >>>> > > > > > > > >> > >> >> > > > Wes
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
> > McKinney <
> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > wrote:
> > >>>> > > > > > > > >> > >> >> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
> > planning to
> > >>>> > > > > amend the
> > >>>> > > > > > > > >> > >> Format docs
> > >>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and
> > also update
> > >>>> > > > the
> > >>>> > > > > C++ library
> > >>>> > > > > > > > >> > >> >> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM
> > Eric Erhardt
> > >>>> > > > > > > > >> > >> >> > > > > <Eric.Erhardt@microsoft.com
> > wrote:
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> > >>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> > >>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15
> > release,
> > >>>> > > > correct?
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
> > changes are
> > >>>> > > > > ready to be
> > >>>> > > > > > > > >> > >> merged into
> > >>>> > > > > > > > >> > >> >> > > > the alignment branch -
> > >>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > Eric
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
> > >>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
> > emkornfield@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
> > 10:24 PM
> > >>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <
> > wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
> > niki.lj <
> > >>>> > > > > niki.lj@aliyun.com>
> > >>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
> > release
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > I should have a little more
> > bandwidth to help
> > >>>> > > > > with some of
> > >>>> > > > > > > > >> > >> the
> > >>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going
> > into the
> > >>>> > > > > weekend.
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019,
> > Wes McKinney <
> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > wrote:
> > >>>> > > > > > > > >> > >> >> > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
> > >>>> > > > > > > > >> > >> >> > > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly
> > packaging and
> > >>>> > > > > integration builds
> > >>>> > > > > > > > >> > >> things
> > >>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for
> > being in release
> > >>>> > > > > readiness by
> > >>>> > > > > > > > >> > >> the end
> > >>>> > > > > > > > >> > >> >> > > of
> > >>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong.
> > I'm planning
> > >>>> > > > > to be working
> > >>>> > > > > > > > >> > >> to close
> > >>>> > > > > > > > >> > >> >> > > as
>
> > >>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
> > help with
> > >>>> > > > > the ongoing
> > >>>> > > > > > > > >> > >> alignment
> > >>>> > > > > > > > >> > >> >> > > > fixes.
> > >>>> > > > > > > > >> > >> >> > > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > > Wes
> > >>>> > > > > > > > >> > >> >> > > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM
> > Micah
> > >>>> > > > Kornfield
> > >>>> > > > > <
> > >>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
> > >>>> > > > > > > > >> > >> >> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > > wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
> > dashboard of
> > >>>> > > > > the current
> > >>>> > > > > > > > >> > >> issues:
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > >>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > se&data=02%7C01%7CEric.Erhardt%
> > >>>> > > > > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > >>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM
> > Wes
> > >>>> > > > McKinney <
> > >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're
> > going to be in
> > >>>> > > > a
> > >>>> > > > > position to
> > >>>> > > > > > > > >> > >> release
> > >>>> > > > > > > > >> > >> >> > > at
> > >>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I
> > hope that
> > >>>> > > > one
> > >>>> > > > > more week of
> > >>>> > > > > > > > >> > >> work (or
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
> > there.
> > >>>> > > > Aside
> > >>>> > > > > from merging
> > >>>> > > > > > > > >> > >> the
> > >>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to
> > make sure
> > >>>> > > > > that our
> > >>>> > > > > > > > >> > >> packaging jobs
> > >>>> > > > > > > > >> > >> >> > > > > > >>> required for the release
> > candidate are all
> > >>>> > > > > working.
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues
> > from the
> > >>>> > > > > 0.15.0 backlog
> > >>>> > > > > > > > >> > >> that they
> > >>>> > > > > > > > >> > >> >> > > > don't
> > >>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end
> > of next week
> > >>>> > > > > that would
> > >>>> > > > > > > > >> > >> help focus
> > >>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently
> > 78 issues in
> > >>>> > > > > 0.15.0 still).
> > >>>> > > > > > > > >> > >> I am
> > >>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
> > features
> > >>>> > > > > related to
> > >>>> > > > > > > > >> > >> dictionaries
> > >>>> > > > > > > > >> > >> >> > > > while
> > >>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still
> > open.
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48
> > PM Wes
> > >>>> > > > > McKinney <
> > >>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to
> > release the
> > >>>> > > > week
> > >>>> > > > > of September
> > >>>> > > > > > > > >> > >> 9, so
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
> > completed by
> > >>>> > > > > end of next
> > >>>> > > > > > > > >> > >> week.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for
> > the
> > >>>> > > > protocol
> > >>>> > > > > alignment
> > >>>> > > > > > > > >> > >> changes for
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of
> > days -- I
> > >>>> > > > think
> > >>>> > > > > that getting
> > >>>> > > > > > > > >> > >> the
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the
> > main barrier
> > >>>> > > > > to releasing.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
> > >>>> > > > > > > > >> > >> >> > > > > > >>> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at
> > 12:25 PM Ji Liu
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > <niki.lj@aliyun.com.invalid
> >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side,
> > I can think
> > >>>> > > > > of several
> > >>>> > > > > > > > >> > >> bugs that
> > >>>> > > > > > > > >> > >> >> > > > need
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > to
> > >>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
> > entries are
> > >>>> > > > > required in
> > >>>> > > > > > > > >> > >> IPC streams
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > even
> > >>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review
> > now, however
> > >>>> > > > > through this
> > >>>> > > > > > > > >> > >> PR we find
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > that
> > >>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java
> > reading and
> > >>>> > > > > writing
> > >>>> > > > > > > > >> > >> dictionaries in IPC
> > >>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with
> > spec[2] since
> > >>>> > > > it
> > >>>> > > > > assumes all
> > >>>> > > > > > > > >> > >> >> > > > dictionaries
> > >>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream
> > (see details in
> > >>>> > > > > PR comments,
> > >>>> > > > > > > > >> > >> and this
> > >>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with
> > version 0.15).
> > >>>> > > > > @Micah Kornfield
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write
> > 64-bit ints as
> > >>>> > > > > strings in
> > >>>> > > > > > > > >> > >> integration
> > >>>> > > > > > > > >> > >> >> > > > test
> > >>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already
> > checked in,
> > >>>> > > > other
> > >>>> > > > > > > > >> > >> implementations
> > >>>> > > > > > > > >> > >> >> > > seems
> > >>>> > > > > > > > >> > >> >> > > > not.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202:
> > OutOfMemory in
> > >>>> > > > > JdbcAdapter[4]
> > >>>> > > > > > > > >> > >> Caused by
> > >>>> > > > > > > > >> > >> >> > > trying
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
> > contiguous
> > >>>> > > > > batch, fixed
> > >>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
> > iteratively
> > >>>> > > > > reading in
> > >>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%
> > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > >
> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
> > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
> > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > >>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > >>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > >
> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >>
> > >>>> > > > >
> > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > ----------------------------------------------------------------
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> > >>>> > > > > wesmckinn@gmail.com> Send
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
> > To:dev <
> > >>>> > > > > > > > >> > >> dev@arrow.apache.org>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for
> > 0.15.0
> > >>>> > > > release
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
> > organizing
> > >>>> > > > > the 0.15.0
> > >>>> > > > > > > > >> > >> backlog some
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants
> > to help
> > >>>> > > > with
> > >>>> > > > > grooming
> > >>>> > > > > > > > >> > >> >> > > (particularly
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
> > C++/Python
> > >>>> > > > > where I'm
> > >>>> > > > > > > > >> > >> focusing) that
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There
> > have been
> > >>>> > > > > almost 500 JIRA
> > >>>> > > > > > > > >> > >> issues
> > >>>> > > > > > > > >> > >> >> > > opened
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we
> > should make sure
> > >>>> > > > > to check
> > >>>> > > > > > > > >> > >> whether
> > >>>> > > > > > > > >> > >> >> > > there's
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
> > serious bugs
> > >>>> > > > > that we should
> > >>>> > > > > > > > >> > >> try to
> > >>>> > > > > > > > >> > >> >> > > fix
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at
> > 6:23 PM Wes
> > >>>> > > > > McKinney
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue
> > in 0.14.1
> > >>>> > > > > seems to be
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > >>>> > > > > > > > >> > >> >> > > >
> > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> > >>>> > > > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause
> > could be the
> > >>>> > > > > Windows
> > >>>> > > > > > > > >> > >> changes in
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
> > 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > 223ae744cc2a12c60cecb5db593263a03c13f85a
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative
> > if a
> > >>>> > > > > volunteer would look
> > >>>> > > > > > > > >> > >> into what
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
> > >>>> > > > > > > > >> > >> >> > > > > > >>> wrong
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels
> > on Windows.
> > >>>> > > > > Otherwise
> > >>>> > > > > > > > >> > >> 0.15.0 Windows
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken,
> > too
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be
> > found at
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > >>>> > > > > > > > >> > >> 40microsoft.com
> > >>>> > > > > > > > >> > >> >> > > > %7Ccbea
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > >
>
> > >>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at
> > 1:28 PM
> > >>>> > > > > Antoine Pitrou <
> > >>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
> > 11:17:07 -0700
> > >>>> > > > > Micah
> > >>>> > > > > > > > >> > >> Kornfield
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <
> emkornfield@gmail.com>
> > wrote:
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we
> > could have
> > >>>> > > > > 32-bit array
> > >>>> > > > > > > > >> > >> lengths and
> > >>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
> > offsets if
> > >>>> > > > we
> > >>>> > > > > wanted (we
> > >>>> > > > > > > > >> > >> just
> > >>>> > > > > > > > >> > >> >> > > > wouldn't
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
> > >>>> > > > > > > > >> > >> >> > > > > > >>> able to
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child
> > with more
> > >>>> > > > > than INT32_MAX
> > >>>> > > > > > > > >> > >> elements).
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is
> > we could do
> > >>>> > > > > this in C++
> > >>>> > > > > > > > >> > >> but we
> > >>>> > > > > > > > >> > >> >> > > > don't.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have
> > introduced the
> > >>>> > > > "Large"
> > >>>> > > > > types if we
> > >>>> > > > > > > > >> > >> did.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take
> > twice as much
> > >>>> > > > > space as 32-bit
> > >>>> > > > > > > > >> > >> >> > > offsets,
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
> > >>>> > > > > > > > >> > >> >> > > > > > >>> you're
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of
> > small-ish lists or
> > >>>> > > > > strings,
> > >>>> > > > > > > > >> > >> 32-bit
> > >>>> > > > > > > > >> > >> >> > > offsets
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So
> > even with
> > >>>> > > > > 64-bit array
> > >>>> > > > > > > > >> > >> lengths from
> > >>>> > > > > > > > >> > >> >> > > > the
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
> > >>>> > > > > > > > >> > >> >> > > > > > >>> it would
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
> > have types
> > >>>> > > > > with 32-bit
> > >>>> > > > > > > > >> > >> offsets.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the
> > limited address
> > >>>> > > > > space in Java
> > >>>> > > > > > > > >> > >> and
> > >>>> > > > > > > > >> > >> >> > > calling
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
> > >>>> > > > > > > > >> > >> >> > > > > > >>> reference
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
> > suboptimal.
> > >>>> > > > > If a consumer
> > >>>> > > > > > > > >> > >> uses a
> > >>>> > > > > > > > >> > >> >> > > > "Large"
> > >>>> > > > > > > > >> > >> >> > > > > > >>> type
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is
> > because they
> > >>>> > > > > need the ability
> > >>>> > > > > > > > >> > >> to store
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
> > >>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
> > column,
> > >>>> > > > > otherwise it is
> > >>>> > > > > > > > >> > >> just
> > >>>> > > > > > > > >> > >> >> > > wasting
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
> > >>>> > > > > > > > >> > >> >> > > > > > >>> [1].
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if
> > the individual
> > >>>> > > > > elements
> > >>>> > > > > > > > >> > >> (lists or
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
> > >>>> > > > > > > > >> > >> >> > > > > > >>> are
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space
> > is wasted in
> > >>>> > > > > proportion,
> > >>>> > > > > > > > >> > >> so it may
> > >>>> > > > > > > > >> > >> >> > > be
> > >>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
> > create a
> > >>>> > > > > "Large" type
> > >>>> > > > > > > > >> > >> array.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
> > theoretically
> > >>>> > > > there
> > >>>> > > > > might be some
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
> > >>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
>
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures
> > to using
> > >>>> > > > the
> > >>>> > > > > native word
> > >>>> > > > > > > > >> > >> sizes.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common
> > 64-bit
> > >>>> > > > > architectures don't do
> > >>>> > > > > > > > >> > >> that, as
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > >>>> > > > > > > > >> > >> >> > > > > > >>> is an
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common
> > integer size even
> > >>>> > > > > in
> > >>>> > > > > > > > >> > >> high-performance
> > >>>> > > > > > > > >> > >> >> > > > code.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> > >>>> > > > > > > > >> > >> >> > > > > > >>>
> > >>>> > > > > > > > >> > >> >> > > > > > >>
> > >>>> > > > > > > > >> > >> >> > > >
> > >>>> > > > > > > > >> > >> >> > >
> > >>>> > > > > > > > >> > >>
> > >>>> > > > > > > > >> > >
> > >>>> > > > >
> > >>>> > > >
> >
>
>
>
Re: Timeline for 0.15.0 release
Posted by Ji Liu <ni...@aliyun.com.INVALID>.
Hi Micah,
Hmm, unfortunately, I just found a bug in JDBC adapter and open a PR, could this change catch up with 0.15?
See https://github.com/apache/arrow/pull/5511
Thanks,
Ji Liu
------------------------------------------------------------------
From:Micah Kornfield <em...@gmail.com>
Send Time:2019年9月26日(星期四) 14:23
To:Neal Richardson <ne...@gmail.com>
Cc:"Krisztián Szűcs" <sz...@gmail.com>; Wes McKinney <we...@gmail.com>; dev <de...@arrow.apache.org>
Subject:Re: Timeline for 0.15.0 release
Just an I've started the RC generation process off, the last commit from
master is [1]
I am currently waiting the crossbow builds (build-690 on
ursa-labs/crossbow). I think this will take a little while so I will pick
it up tomorrow (Thursday).
Thanks,
Micah
[1]
https://github.com/apache/arrow/commit/07ab5083d5a2925ced6f8168b60b8fa336f4eccc
On Wed, Sep 25, 2019 at 2:07 PM Neal Richardson <ne...@gmail.com>
wrote:
> IMO it's too risky to add something that adds a dependency
> (aws-sdk-cpp) on the day of cutting a release.
>
> Neal
>
> On Wed, Sep 25, 2019 at 12:54 PM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > We don't have a comprehensive documentation yet, so let's postpone it.
> >
> >
> > On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com> wrote:
> >>
> >> The S3 python bindings would be a nice addition to the release.
> >> I don't think we should block on this but the PR is ready. Opinions?
> >> https://github.com/apache/arrow/pull/5423
> >>
> >>
> >>
> >>
> >> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >>>
> >>> OK, I'll start the process today. I'll send up e-mail updates as I
> make progress.
> >>>
> >>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com>
> wrote:
> >>>>
> >>>> Yes, all systems go as far as I'm concerned.
> >>>>
> >>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
> >>>> <ne...@gmail.com> wrote:
> >>>> >
> >>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
> >>>> > and it looks like the LICENSE issue is being resolved as I type. So
> >>>> > are we good to go now?
> >>>> >
> >>>> > Neal
> >>>> >
> >>>> >
> >>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com>
> wrote:
> >>>> > >
> >>>> > > I found a last minute issue with DataFusion (Rust) and would
> appreciate it
> >>>> > > if we could merge ARROW-6086 (PR is
> >>>> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
> >>>> > >
> >>>> > > Thanks,
> >>>> > >
> >>>> > > Andy.
> >>>> > >
> >>>> > >
> >>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > OK, I'm going to postpone cutting a release until tomorrow
> (hoping we can
> >>>> > > > issues resolved by then).. I'll also try to review the
> third-party
> >>>> > > > additions since 14.x.
> >>>> > > >
> >>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>> > > >
> >>>> > > > > I found a licensing issue
> >>>> > > > >
> >>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
> >>>> > > > >
> >>>> > > > > It might be worth examining third party code added to the
> project
> >>>> > > > > since 0.14.x to make sure there are no other such issues.
> >>>> > > > >
> >>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > wrote:
> >>>> > > > > >
> >>>> > > > > > I have diagnosed the problem (Thrift "string" data must be
> UTF-8,
> >>>> > > > > > cannot be arbitrary binary) and am working on a patch right
> now
> >>>> > > > > >
> >>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > I just opened
> >>>> > > > > > >
> >>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
> >>>> > > > > > >
> >>>> > > > > > > Please don't cut an RC until I have an opportunity to
> diagnose this,
> >>>> > > > > > > will report back.
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > I'm investigating a possible Parquet-related
> compatibility bug
> >>>> > > > that I
> >>>> > > > > > > > encountered through some routine testing /
> benchmarking. I'll
> >>>> > > > report
> >>>> > > > > > > > back once I figure out what is going on (if anything)
> >>>> > > > > > > >
> >>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> >>>> > > > > emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> (i.e. you can
> >>>> > > > > get it
> >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> essential.
> >>>> > > > > > > > >
> >>>> > > > > > > > > That won't be an option for me this week (it seems
> like I would
> >>>> > > > > need to meet one face-to-face). I'll try to get the GPG
> checked in and
> >>>> > > > the
> >>>> > > > > rest of the pre-requisites done tomorrow (Monday) to
> hopefully start the
> >>>> > > > > release on Tuesday (hopefully we can solve the last
> blocker/integration
> >>>> > > > > tests by then).
> >>>> > > > > > > > >
> >>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> >>>> > > > wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> (i.e. you can
> >>>> > > > > get it
> >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> essential.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> Speaking of the release, there are at least 2 code
> changes I
> >>>> > > > still
> >>>> > > > > > > > >> want to get in
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> ARROW-5717
> >>>> > > > > > > > >> ARROW-6353
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once
> the build
> >>>> > > > is
> >>>> > > > > green.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> There are a couple of Rust patches still marked for
> 0.15. The
> >>>> > > > rest
> >>>> > > > > > > > >> seems to be documentation and a couple of
> integration test
> >>>> > > > > failures we
> >>>> > > > > > > > >> should see about fixing in time.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> >>>> > > > > emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Thanks Krisztián and Wes,
> >>>> > > > > > > > >> > I've gone ahead and started registering myself on
> all the
> >>>> > > > > packaging sites.
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Is there any review process when adding my GPG key
> to the SVN
> >>>> > > > > file? [1]
> >>>> > > > > > > > >> > doesn't seem to mention explicitly.
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Thanks,
> >>>> > > > > > > > >> > Micah
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > [1]
> https://www.apache.org/dev/version-control.html#https-svn
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> >>>> > > > > szucs.krisztian@gmail.com>
> >>>> > > > > > > > >> > wrote:
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com> wrote:
> >>>> > > > > > > > >> > >
> >>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah
> Kornfield <
> >>>> > > > > emkornfield@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> The process should be well documented at
> this point but
> >>>> > > > > there are a
> >>>> > > > > > > > >> > >> >> number of steps.
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
> release?
> >>>> > > > Are
> >>>> > > > > there
> >>>> > > > > > > > >> > >> instructions for the adding the code signing
> Key to SVN?
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > I will make a go of it. i will try to
> mitigate any
> >>>> > > > > internet issues by
> >>>> > > > > > > > >> > >> doing the process for a cloud instance (I
> assume that isn't
> >>>> > > > > a problem?).
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
> producing
> >>>> > > > an
> >>>> > > > > RC may be
> >>>> > > > > > > > >> > >> time consuming, but you are welcome to try.
> Krisztian --
> >>>> > > > are
> >>>> > > > > you
> >>>> > > > > > > > >> > >> available next week to help Micah and
> potentially take over
> >>>> > > > > producing
> >>>> > > > > > > > >> > >> the RC if there are issues?
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > > Sure, I'll be available next week. We can also
> grant access
> >>>> > > > to
> >>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
> configuring
> >>>> > > > all
> >>>> > > > > > > > >> > > the CI backends can be time consuming.
> >>>> > > > > > > > >> > >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >> > Thanks,
> >>>> > > > > > > > >> > >> > Micah
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > [1]
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> >>>> > > >
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> The process should be well documented at
> this point but
> >>>> > > > > there are a
> >>>> > > > > > > > >> > >> >> number of steps. Note that you need to add
> your code
> >>>> > > > > signing key to
> >>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard
> to do). I
> >>>> > > > > think it's fine
> >>>> > > > > > > > >> > >> >> to hand off the process to others after the
> VOTE but it
> >>>> > > > > would be
> >>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
> producing the
> >>>> > > > > source and
> >>>> > > > > > > > >> > >> >> binary artifacts for the vote
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah
> Kornfield <
> >>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > SGTM, as well.
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > I should have a little bit of time next
> week if I can
> >>>> > > > > help as RM but
> >>>> > > > > > > > >> > >> I have
> >>>> > > > > > > > >> > >> >> > a couple of concerns:
> >>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble
> downloading and
> >>>> > > > > validating
> >>>> > > > > > > > >> > >> releases. I'm a
> >>>> > > > > > > > >> > >> >> > bit worried, that I might have similar
> problems doing
> >>>> > > > > the necessary
> >>>> > > > > > > > >> > >> uploads.
> >>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be
> not great, I
> >>>> > > > > don't know if
> >>>> > > > > > > > >> > >> this
> >>>> > > > > > > > >> > >> >> > would make it even less likely to be
> successful.
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I
> would have to
> >>>> > > > > abandon the
> >>>> > > > > > > > >> > >> process
> >>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could
> serve as a
> >>>> > > > > backup? Are the
> >>>> > > > > > > > >> > >> steps
> >>>> > > > > > > > >> > >> >> > well documented?
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > Thanks,
> >>>> > > > > > > > >> > >> >> > Micah
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
> Richardson <
> >>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
> >>>> > > > > > > > >> > >> >> > wrote:
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > > Sounds good to me.
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
> volunteers?
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > Neal
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes
> McKinney <
> >>>> > > > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > > hi all,
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to
> be able to
> >>>> > > > > make the 0.15.0
> >>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils
> down" at the end
> >>>> > > > > of this week
> >>>> > > > > > > > >> > >> and
> >>>> > > > > > > > >> > >> >> > > > see if a release candidate can be
> produced next
> >>>> > > > > Monday September
> >>>> > > > > > > > >> > >> 23.
> >>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > Thanks,
> >>>> > > > > > > > >> > >> >> > > > Wes
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
> McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > wrote:
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
> planning to
> >>>> > > > > amend the
> >>>> > > > > > > > >> > >> Format docs
> >>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and
> also update
> >>>> > > > the
> >>>> > > > > C++ library
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM
> Eric Erhardt
> >>>> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> >>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> >>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15
> release,
> >>>> > > > correct?
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
> changes are
> >>>> > > > > ready to be
> >>>> > > > > > > > >> > >> merged into
> >>>> > > > > > > > >> > >> >> > > > the alignment branch -
> >>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > Eric
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
> >>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
> emkornfield@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
> 10:24 PM
> >>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
> niki.lj <
> >>>> > > > > niki.lj@aliyun.com>
> >>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
> release
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > I should have a little more
> bandwidth to help
> >>>> > > > > with some of
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going
> into the
> >>>> > > > > weekend.
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019,
> Wes McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly
> packaging and
> >>>> > > > > integration builds
> >>>> > > > > > > > >> > >> things
> >>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for
> being in release
> >>>> > > > > readiness by
> >>>> > > > > > > > >> > >> the end
> >>>> > > > > > > > >> > >> >> > > of
> >>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong.
> I'm planning
> >>>> > > > > to be working
> >>>> > > > > > > > >> > >> to close
> >>>> > > > > > > > >> > >> >> > > as
> >>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
> help with
> >>>> > > > > the ongoing
> >>>> > > > > > > > >> > >> alignment
> >>>> > > > > > > > >> > >> >> > > > fixes.
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > Wes
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM
> Micah
> >>>> > > > Kornfield
> >>>> > > > > <
> >>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
> dashboard of
> >>>> > > > > the current
> >>>> > > > > > > > >> > >> issues:
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >>>> > > > > > > > >> > >> >> > > > > > >>
> se&data=02%7C01%7CEric.Erhardt%
> >>>> > > > > 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM
> Wes
> >>>> > > > McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're
> going to be in
> >>>> > > > a
> >>>> > > > > position to
> >>>> > > > > > > > >> > >> release
> >>>> > > > > > > > >> > >> >> > > at
> >>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I
> hope that
> >>>> > > > one
> >>>> > > > > more week of
> >>>> > > > > > > > >> > >> work (or
> >>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
> there.
> >>>> > > > Aside
> >>>> > > > > from merging
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to
> make sure
> >>>> > > > > that our
> >>>> > > > > > > > >> > >> packaging jobs
> >>>> > > > > > > > >> > >> >> > > > > > >>> required for the release
> candidate are all
> >>>> > > > > working.
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues
> from the
> >>>> > > > > 0.15.0 backlog
> >>>> > > > > > > > >> > >> that they
> >>>> > > > > > > > >> > >> >> > > > don't
> >>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end
> of next week
> >>>> > > > > that would
> >>>> > > > > > > > >> > >> help focus
> >>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently
> 78 issues in
> >>>> > > > > 0.15.0 still).
> >>>> > > > > > > > >> > >> I am
> >>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
> features
> >>>> > > > > related to
> >>>> > > > > > > > >> > >> dictionaries
> >>>> > > > > > > > >> > >> >> > > > while
> >>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still
> open.
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48
> PM Wes
> >>>> > > > > McKinney <
> >>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to
> release the
> >>>> > > > week
> >>>> > > > > of September
> >>>> > > > > > > > >> > >> 9, so
> >>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
> completed by
> >>>> > > > > end of next
> >>>> > > > > > > > >> > >> week.
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for
> the
> >>>> > > > protocol
> >>>> > > > > alignment
> >>>> > > > > > > > >> > >> changes for
> >>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of
> days -- I
> >>>> > > > think
> >>>> > > > > that getting
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the
> main barrier
> >>>> > > > > to releasing.
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at
> 12:25 PM Ji Liu
> >>>> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side,
> I can think
> >>>> > > > > of several
> >>>> > > > > > > > >> > >> bugs that
> >>>> > > > > > > > >> > >> >> > > > need
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > to
> >>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
> entries are
> >>>> > > > > required in
> >>>> > > > > > > > >> > >> IPC streams
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > even
> >>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review
> now, however
> >>>> > > > > through this
> >>>> > > > > > > > >> > >> PR we find
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > that
> >>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java
> reading and
> >>>> > > > > writing
> >>>> > > > > > > > >> > >> dictionaries in IPC
> >>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with
> spec[2] since
> >>>> > > > it
> >>>> > > > > assumes all
> >>>> > > > > > > > >> > >> >> > > > dictionaries
> >>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream
> (see details in
> >>>> > > > > PR comments,
> >>>> > > > > > > > >> > >> and this
> >>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with
> version 0.15).
> >>>> > > > > @Micah Kornfield
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write
> 64-bit ints as
> >>>> > > > > strings in
> >>>> > > > > > > > >> > >> integration
> >>>> > > > > > > > >> > >> >> > > > test
> >>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already
> checked in,
> >>>> > > > other
> >>>> > > > > > > > >> > >> implementations
> >>>> > > > > > > > >> > >> >> > > seems
> >>>> > > > > > > > >> > >> >> > > > not.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202:
> OutOfMemory in
> >>>> > > > > JdbcAdapter[4]
> >>>> > > > > > > > >> > >> Caused by
> >>>> > > > > > > > >> > >> >> > > trying
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
> contiguous
> >>>> > > > > batch, fixed
> >>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
> iteratively
> >>>> > > > > reading in
> >>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> >>>> > > > > > > > >> > >> >> > > >
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> >>>> > > > > > > > >> > >> >> > > >
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> ----------------------------------------------------------------
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com> Send
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
> To:dev <
> >>>> > > > > > > > >> > >> dev@arrow.apache.org>
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for
> 0.15.0
> >>>> > > > release
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
> organizing
> >>>> > > > > the 0.15.0
> >>>> > > > > > > > >> > >> backlog some
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants
> to help
> >>>> > > > with
> >>>> > > > > grooming
> >>>> > > > > > > > >> > >> >> > > (particularly
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
> C++/Python
> >>>> > > > > where I'm
> >>>> > > > > > > > >> > >> focusing) that
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There
> have been
> >>>> > > > > almost 500 JIRA
> >>>> > > > > > > > >> > >> issues
> >>>> > > > > > > > >> > >> >> > > opened
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we
> should make sure
> >>>> > > > > to check
> >>>> > > > > > > > >> > >> whether
> >>>> > > > > > > > >> > >> >> > > there's
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
> serious bugs
> >>>> > > > > that we should
> >>>> > > > > > > > >> > >> try to
> >>>> > > > > > > > >> > >> >> > > fix
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at
> 6:23 PM Wes
> >>>> > > > > McKinney
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue
> in 0.14.1
> >>>> > > > > seems to be
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6015&data=02
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> >>>> > > > 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause
> could be the
> >>>> > > > > Windows
> >>>> > > > > > > > >> > >> changes in
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative
> if a
> >>>> > > > > volunteer would look
> >>>> > > > > > > > >> > >> into what
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrong
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels
> on Windows.
> >>>> > > > > Otherwise
> >>>> > > > > > > > >> > >> 0.15.0 Windows
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken,
> too
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be
> found at
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> >>>> > > > > > > > >> > >> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbea
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at
> 1:28 PM
> >>>> > > > > Antoine Pitrou <
> >>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
> 11:17:07 -0700
> >>>> > > > > Micah
> >>>> > > > > > > > >> > >> Kornfield
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com>
> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we
> could have
> >>>> > > > > 32-bit array
> >>>> > > > > > > > >> > >> lengths and
> >>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
> offsets if
> >>>> > > > we
> >>>> > > > > wanted (we
> >>>> > > > > > > > >> > >> just
> >>>> > > > > > > > >> > >> >> > > > wouldn't
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
> >>>> > > > > > > > >> > >> >> > > > > > >>> able to
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child
> with more
> >>>> > > > > than INT32_MAX
> >>>> > > > > > > > >> > >> elements).
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is
> we could do
> >>>> > > > > this in C++
> >>>> > > > > > > > >> > >> but we
> >>>> > > > > > > > >> > >> >> > > > don't.
> >>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have
> introduced the
> >>>> > > > "Large"
> >>>> > > > > types if we
> >>>> > > > > > > > >> > >> did.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take
> twice as much
> >>>> > > > > space as 32-bit
> >>>> > > > > > > > >> > >> >> > > offsets,
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
> >>>> > > > > > > > >> > >> >> > > > > > >>> you're
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of
> small-ish lists or
> >>>> > > > > strings,
> >>>> > > > > > > > >> > >> 32-bit
> >>>> > > > > > > > >> > >> >> > > offsets
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So
> even with
> >>>> > > > > 64-bit array
> >>>> > > > > > > > >> > >> lengths from
> >>>> > > > > > > > >> > >> >> > > > the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
> >>>> > > > > > > > >> > >> >> > > > > > >>> it would
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
> have types
> >>>> > > > > with 32-bit
> >>>> > > > > > > > >> > >> offsets.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the
> limited address
> >>>> > > > > space in Java
> >>>> > > > > > > > >> > >> and
> >>>> > > > > > > > >> > >> >> > > calling
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
> >>>> > > > > > > > >> > >> >> > > > > > >>> reference
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
> suboptimal.
> >>>> > > > > If a consumer
> >>>> > > > > > > > >> > >> uses a
> >>>> > > > > > > > >> > >> >> > > > "Large"
> >>>> > > > > > > > >> > >> >> > > > > > >>> type
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is
> because they
> >>>> > > > > need the ability
> >>>> > > > > > > > >> > >> to store
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
> >>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
> column,
> >>>> > > > > otherwise it is
> >>>> > > > > > > > >> > >> just
> >>>> > > > > > > > >> > >> >> > > wasting
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
> >>>> > > > > > > > >> > >> >> > > > > > >>> [1].
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if
> the individual
> >>>> > > > > elements
> >>>> > > > > > > > >> > >> (lists or
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
> >>>> > > > > > > > >> > >> >> > > > > > >>> are
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space
> is wasted in
> >>>> > > > > proportion,
> >>>> > > > > > > > >> > >> so it may
> >>>> > > > > > > > >> > >> >> > > be
> >>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
> create a
> >>>> > > > > "Large" type
> >>>> > > > > > > > >> > >> array.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
> theoretically
> >>>> > > > there
> >>>> > > > > might be some
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
> >>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures
> to using
> >>>> > > > the
> >>>> > > > > native word
> >>>> > > > > > > > >> > >> sizes.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common
> 64-bit
> >>>> > > > > architectures don't do
> >>>> > > > > > > > >> > >> that, as
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> >>>> > > > > > > > >> > >> >> > > > > > >>> is an
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common
> integer size even
> >>>> > > > > in
> >>>> > > > > > > > >> > >> high-performance
> >>>> > > > > > > > >> > >> >> > > > code.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >
> >>>> > > > >
> >>>> > > >
>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
Just an I've started the RC generation process off, the last commit from
master is [1]
I am currently waiting the crossbow builds (build-690 on
ursa-labs/crossbow). I think this will take a little while so I will pick
it up tomorrow (Thursday).
Thanks,
Micah
[1]
https://github.com/apache/arrow/commit/07ab5083d5a2925ced6f8168b60b8fa336f4eccc
On Wed, Sep 25, 2019 at 2:07 PM Neal Richardson <ne...@gmail.com>
wrote:
> IMO it's too risky to add something that adds a dependency
> (aws-sdk-cpp) on the day of cutting a release.
>
> Neal
>
> On Wed, Sep 25, 2019 at 12:54 PM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > We don't have a comprehensive documentation yet, so let's postpone it.
> >
> >
> > On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com> wrote:
> >>
> >> The S3 python bindings would be a nice addition to the release.
> >> I don't think we should block on this but the PR is ready. Opinions?
> >> https://github.com/apache/arrow/pull/5423
> >>
> >>
> >>
> >>
> >> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >>>
> >>> OK, I'll start the process today. I'll send up e-mail updates as I
> make progress.
> >>>
> >>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com>
> wrote:
> >>>>
> >>>> Yes, all systems go as far as I'm concerned.
> >>>>
> >>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
> >>>> <ne...@gmail.com> wrote:
> >>>> >
> >>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
> >>>> > and it looks like the LICENSE issue is being resolved as I type. So
> >>>> > are we good to go now?
> >>>> >
> >>>> > Neal
> >>>> >
> >>>> >
> >>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com>
> wrote:
> >>>> > >
> >>>> > > I found a last minute issue with DataFusion (Rust) and would
> appreciate it
> >>>> > > if we could merge ARROW-6086 (PR is
> >>>> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
> >>>> > >
> >>>> > > Thanks,
> >>>> > >
> >>>> > > Andy.
> >>>> > >
> >>>> > >
> >>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > OK, I'm going to postpone cutting a release until tomorrow
> (hoping we can
> >>>> > > > issues resolved by then).. I'll also try to review the
> third-party
> >>>> > > > additions since 14.x.
> >>>> > > >
> >>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>> > > >
> >>>> > > > > I found a licensing issue
> >>>> > > > >
> >>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
> >>>> > > > >
> >>>> > > > > It might be worth examining third party code added to the
> project
> >>>> > > > > since 0.14.x to make sure there are no other such issues.
> >>>> > > > >
> >>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > wrote:
> >>>> > > > > >
> >>>> > > > > > I have diagnosed the problem (Thrift "string" data must be
> UTF-8,
> >>>> > > > > > cannot be arbitrary binary) and am working on a patch right
> now
> >>>> > > > > >
> >>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > I just opened
> >>>> > > > > > >
> >>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
> >>>> > > > > > >
> >>>> > > > > > > Please don't cut an RC until I have an opportunity to
> diagnose this,
> >>>> > > > > > > will report back.
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > I'm investigating a possible Parquet-related
> compatibility bug
> >>>> > > > that I
> >>>> > > > > > > > encountered through some routine testing /
> benchmarking. I'll
> >>>> > > > report
> >>>> > > > > > > > back once I figure out what is going on (if anything)
> >>>> > > > > > > >
> >>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> >>>> > > > > emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> (i.e. you can
> >>>> > > > > get it
> >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> essential.
> >>>> > > > > > > > >
> >>>> > > > > > > > > That won't be an option for me this week (it seems
> like I would
> >>>> > > > > need to meet one face-to-face). I'll try to get the GPG
> checked in and
> >>>> > > > the
> >>>> > > > > rest of the pre-requisites done tomorrow (Monday) to
> hopefully start the
> >>>> > > > > release on Tuesday (hopefully we can solve the last
> blocker/integration
> >>>> > > > > tests by then).
> >>>> > > > > > > > >
> >>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> >>>> > > > wesmckinn@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust
> (i.e. you can
> >>>> > > > > get it
> >>>> > > > > > > > >> signed by another PMC member), but is not 100%
> essential.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> Speaking of the release, there are at least 2 code
> changes I
> >>>> > > > still
> >>>> > > > > > > > >> want to get in
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> ARROW-5717
> >>>> > > > > > > > >> ARROW-6353
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once
> the build
> >>>> > > > is
> >>>> > > > > green.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> There are a couple of Rust patches still marked for
> 0.15. The
> >>>> > > > rest
> >>>> > > > > > > > >> seems to be documentation and a couple of
> integration test
> >>>> > > > > failures we
> >>>> > > > > > > > >> should see about fixing in time.
> >>>> > > > > > > > >>
> >>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> >>>> > > > > emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Thanks Krisztián and Wes,
> >>>> > > > > > > > >> > I've gone ahead and started registering myself on
> all the
> >>>> > > > > packaging sites.
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Is there any review process when adding my GPG key
> to the SVN
> >>>> > > > > file? [1]
> >>>> > > > > > > > >> > doesn't seem to mention explicitly.
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > Thanks,
> >>>> > > > > > > > >> > Micah
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > [1]
> https://www.apache.org/dev/version-control.html#https-svn
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> >>>> > > > > szucs.krisztian@gmail.com>
> >>>> > > > > > > > >> > wrote:
> >>>> > > > > > > > >> >
> >>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com> wrote:
> >>>> > > > > > > > >> > >
> >>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah
> Kornfield <
> >>>> > > > > emkornfield@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> The process should be well documented at
> this point but
> >>>> > > > > there are a
> >>>> > > > > > > > >> > >> >> number of steps.
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
> release?
> >>>> > > > Are
> >>>> > > > > there
> >>>> > > > > > > > >> > >> instructions for the adding the code signing
> Key to SVN?
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > I will make a go of it. i will try to
> mitigate any
> >>>> > > > > internet issues by
> >>>> > > > > > > > >> > >> doing the process for a cloud instance (I
> assume that isn't
> >>>> > > > > a problem?).
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
> producing
> >>>> > > > an
> >>>> > > > > RC may be
> >>>> > > > > > > > >> > >> time consuming, but you are welcome to try.
> Krisztian --
> >>>> > > > are
> >>>> > > > > you
> >>>> > > > > > > > >> > >> available next week to help Micah and
> potentially take over
> >>>> > > > > producing
> >>>> > > > > > > > >> > >> the RC if there are issues?
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > > Sure, I'll be available next week. We can also
> grant access
> >>>> > > > to
> >>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
> configuring
> >>>> > > > all
> >>>> > > > > > > > >> > > the CI backends can be time consuming.
> >>>> > > > > > > > >> > >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >> > Thanks,
> >>>> > > > > > > > >> > >> > Micah
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > [1]
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> >>>> > > >
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> >>>> > > > > > > > >> > >> >
> >>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> The process should be well documented at
> this point but
> >>>> > > > > there are a
> >>>> > > > > > > > >> > >> >> number of steps. Note that you need to add
> your code
> >>>> > > > > signing key to
> >>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard
> to do). I
> >>>> > > > > think it's fine
> >>>> > > > > > > > >> > >> >> to hand off the process to others after the
> VOTE but it
> >>>> > > > > would be
> >>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
> producing the
> >>>> > > > > source and
> >>>> > > > > > > > >> > >> >> binary artifacts for the vote
> >>>> > > > > > > > >> > >> >>
> >>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah
> Kornfield <
> >>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > SGTM, as well.
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > I should have a little bit of time next
> week if I can
> >>>> > > > > help as RM but
> >>>> > > > > > > > >> > >> I have
> >>>> > > > > > > > >> > >> >> > a couple of concerns:
> >>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble
> downloading and
> >>>> > > > > validating
> >>>> > > > > > > > >> > >> releases. I'm a
> >>>> > > > > > > > >> > >> >> > bit worried, that I might have similar
> problems doing
> >>>> > > > > the necessary
> >>>> > > > > > > > >> > >> uploads.
> >>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be
> not great, I
> >>>> > > > > don't know if
> >>>> > > > > > > > >> > >> this
> >>>> > > > > > > > >> > >> >> > would make it even less likely to be
> successful.
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I
> would have to
> >>>> > > > > abandon the
> >>>> > > > > > > > >> > >> process
> >>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could
> serve as a
> >>>> > > > > backup? Are the
> >>>> > > > > > > > >> > >> steps
> >>>> > > > > > > > >> > >> >> > well documented?
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > Thanks,
> >>>> > > > > > > > >> > >> >> > Micah
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
> Richardson <
> >>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
> >>>> > > > > > > > >> > >> >> > wrote:
> >>>> > > > > > > > >> > >> >> >
> >>>> > > > > > > > >> > >> >> > > Sounds good to me.
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
> volunteers?
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > Neal
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes
> McKinney <
> >>>> > > > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> wrote:
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >> >> > > > hi all,
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to
> be able to
> >>>> > > > > make the 0.15.0
> >>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils
> down" at the end
> >>>> > > > > of this week
> >>>> > > > > > > > >> > >> and
> >>>> > > > > > > > >> > >> >> > > > see if a release candidate can be
> produced next
> >>>> > > > > Monday September
> >>>> > > > > > > > >> > >> 23.
> >>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > Thanks,
> >>>> > > > > > > > >> > >> >> > > > Wes
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
> McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > wrote:
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
> planning to
> >>>> > > > > amend the
> >>>> > > > > > > > >> > >> Format docs
> >>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and
> also update
> >>>> > > > the
> >>>> > > > > C++ library
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM
> Eric Erhardt
> >>>> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> >>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> >>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15
> release,
> >>>> > > > correct?
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
> changes are
> >>>> > > > > ready to be
> >>>> > > > > > > > >> > >> merged into
> >>>> > > > > > > > >> > >> >> > > > the alignment branch -
> >>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > Eric
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
> >>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
> emkornfield@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
> 10:24 PM
> >>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <
> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
> niki.lj <
> >>>> > > > > niki.lj@aliyun.com>
> >>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
> release
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > I should have a little more
> bandwidth to help
> >>>> > > > > with some of
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going
> into the
> >>>> > > > > weekend.
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019,
> Wes McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly
> packaging and
> >>>> > > > > integration builds
> >>>> > > > > > > > >> > >> things
> >>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for
> being in release
> >>>> > > > > readiness by
> >>>> > > > > > > > >> > >> the end
> >>>> > > > > > > > >> > >> >> > > of
> >>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong.
> I'm planning
> >>>> > > > > to be working
> >>>> > > > > > > > >> > >> to close
> >>>> > > > > > > > >> > >> >> > > as
> >>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
> help with
> >>>> > > > > the ongoing
> >>>> > > > > > > > >> > >> alignment
> >>>> > > > > > > > >> > >> >> > > > fixes.
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > Wes
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM
> Micah
> >>>> > > > Kornfield
> >>>> > > > > <
> >>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
> >>>> > > > > > > > >> > >> >> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
> dashboard of
> >>>> > > > > the current
> >>>> > > > > > > > >> > >> issues:
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >>>> > > > > > > > >> > >> >> > > > > > >>
> se&data=02%7C01%7CEric.Erhardt%
> >>>> > > > > 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM
> Wes
> >>>> > > > McKinney <
> >>>> > > > > > > > >> > >> wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're
> going to be in
> >>>> > > > a
> >>>> > > > > position to
> >>>> > > > > > > > >> > >> release
> >>>> > > > > > > > >> > >> >> > > at
> >>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I
> hope that
> >>>> > > > one
> >>>> > > > > more week of
> >>>> > > > > > > > >> > >> work (or
> >>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
> there.
> >>>> > > > Aside
> >>>> > > > > from merging
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to
> make sure
> >>>> > > > > that our
> >>>> > > > > > > > >> > >> packaging jobs
> >>>> > > > > > > > >> > >> >> > > > > > >>> required for the release
> candidate are all
> >>>> > > > > working.
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues
> from the
> >>>> > > > > 0.15.0 backlog
> >>>> > > > > > > > >> > >> that they
> >>>> > > > > > > > >> > >> >> > > > don't
> >>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end
> of next week
> >>>> > > > > that would
> >>>> > > > > > > > >> > >> help focus
> >>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently
> 78 issues in
> >>>> > > > > 0.15.0 still).
> >>>> > > > > > > > >> > >> I am
> >>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
> features
> >>>> > > > > related to
> >>>> > > > > > > > >> > >> dictionaries
> >>>> > > > > > > > >> > >> >> > > > while
> >>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still
> open.
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48
> PM Wes
> >>>> > > > > McKinney <
> >>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to
> release the
> >>>> > > > week
> >>>> > > > > of September
> >>>> > > > > > > > >> > >> 9, so
> >>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
> completed by
> >>>> > > > > end of next
> >>>> > > > > > > > >> > >> week.
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for
> the
> >>>> > > > protocol
> >>>> > > > > alignment
> >>>> > > > > > > > >> > >> changes for
> >>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of
> days -- I
> >>>> > > > think
> >>>> > > > > that getting
> >>>> > > > > > > > >> > >> the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the
> main barrier
> >>>> > > > > to releasing.
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
> >>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
> >>>> > > > > > > > >> > >> >> > > > > > >>> >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at
> 12:25 PM Ji Liu
> >>>> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side,
> I can think
> >>>> > > > > of several
> >>>> > > > > > > > >> > >> bugs that
> >>>> > > > > > > > >> > >> >> > > > need
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > to
> >>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
> entries are
> >>>> > > > > required in
> >>>> > > > > > > > >> > >> IPC streams
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > even
> >>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review
> now, however
> >>>> > > > > through this
> >>>> > > > > > > > >> > >> PR we find
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > that
> >>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java
> reading and
> >>>> > > > > writing
> >>>> > > > > > > > >> > >> dictionaries in IPC
> >>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with
> spec[2] since
> >>>> > > > it
> >>>> > > > > assumes all
> >>>> > > > > > > > >> > >> >> > > > dictionaries
> >>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream
> (see details in
> >>>> > > > > PR comments,
> >>>> > > > > > > > >> > >> and this
> >>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with
> version 0.15).
> >>>> > > > > @Micah Kornfield
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write
> 64-bit ints as
> >>>> > > > > strings in
> >>>> > > > > > > > >> > >> integration
> >>>> > > > > > > > >> > >> >> > > > test
> >>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already
> checked in,
> >>>> > > > other
> >>>> > > > > > > > >> > >> implementations
> >>>> > > > > > > > >> > >> >> > > seems
> >>>> > > > > > > > >> > >> >> > > > not.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202:
> OutOfMemory in
> >>>> > > > > JdbcAdapter[4]
> >>>> > > > > > > > >> > >> Caused by
> >>>> > > > > > > > >> > >> >> > > trying
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
> contiguous
> >>>> > > > > batch, fixed
> >>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
> iteratively
> >>>> > > > > reading in
> >>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> >>>> > > > > > > > >> > >> >> > > >
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> >>>> > > > > > > > >> > >> >> > > >
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >>
> >>>> > > > >
> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> ----------------------------------------------------------------
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> >>>> > > > > wesmckinn@gmail.com> Send
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
> To:dev <
> >>>> > > > > > > > >> > >> dev@arrow.apache.org>
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for
> 0.15.0
> >>>> > > > release
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
> organizing
> >>>> > > > > the 0.15.0
> >>>> > > > > > > > >> > >> backlog some
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants
> to help
> >>>> > > > with
> >>>> > > > > grooming
> >>>> > > > > > > > >> > >> >> > > (particularly
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
> C++/Python
> >>>> > > > > where I'm
> >>>> > > > > > > > >> > >> focusing) that
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There
> have been
> >>>> > > > > almost 500 JIRA
> >>>> > > > > > > > >> > >> issues
> >>>> > > > > > > > >> > >> >> > > opened
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we
> should make sure
> >>>> > > > > to check
> >>>> > > > > > > > >> > >> whether
> >>>> > > > > > > > >> > >> >> > > there's
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
> serious bugs
> >>>> > > > > that we should
> >>>> > > > > > > > >> > >> try to
> >>>> > > > > > > > >> > >> >> > > fix
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at
> 6:23 PM Wes
> >>>> > > > > McKinney
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue
> in 0.14.1
> >>>> > > > > seems to be
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> >>>> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6015&data=02
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> >>>> > > > 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause
> could be the
> >>>> > > > > Windows
> >>>> > > > > > > > >> > >> changes in
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative
> if a
> >>>> > > > > volunteer would look
> >>>> > > > > > > > >> > >> into what
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
> >>>> > > > > > > > >> > >> >> > > > > > >>> wrong
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels
> on Windows.
> >>>> > > > > Otherwise
> >>>> > > > > > > > >> > >> 0.15.0 Windows
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken,
> too
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be
> found at
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> >>>> > > > > > > > >> > >> 40microsoft.com
> >>>> > > > > > > > >> > >> >> > > > %7Ccbea
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at
> 1:28 PM
> >>>> > > > > Antoine Pitrou <
> >>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
> 11:17:07 -0700
> >>>> > > > > Micah
> >>>> > > > > > > > >> > >> Kornfield
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com>
> wrote:
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we
> could have
> >>>> > > > > 32-bit array
> >>>> > > > > > > > >> > >> lengths and
> >>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
> offsets if
> >>>> > > > we
> >>>> > > > > wanted (we
> >>>> > > > > > > > >> > >> just
> >>>> > > > > > > > >> > >> >> > > > wouldn't
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
> >>>> > > > > > > > >> > >> >> > > > > > >>> able to
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child
> with more
> >>>> > > > > than INT32_MAX
> >>>> > > > > > > > >> > >> elements).
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is
> we could do
> >>>> > > > > this in C++
> >>>> > > > > > > > >> > >> but we
> >>>> > > > > > > > >> > >> >> > > > don't.
> >>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have
> introduced the
> >>>> > > > "Large"
> >>>> > > > > types if we
> >>>> > > > > > > > >> > >> did.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take
> twice as much
> >>>> > > > > space as 32-bit
> >>>> > > > > > > > >> > >> >> > > offsets,
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
> >>>> > > > > > > > >> > >> >> > > > > > >>> you're
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of
> small-ish lists or
> >>>> > > > > strings,
> >>>> > > > > > > > >> > >> 32-bit
> >>>> > > > > > > > >> > >> >> > > offsets
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So
> even with
> >>>> > > > > 64-bit array
> >>>> > > > > > > > >> > >> lengths from
> >>>> > > > > > > > >> > >> >> > > > the
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
> >>>> > > > > > > > >> > >> >> > > > > > >>> it would
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
> have types
> >>>> > > > > with 32-bit
> >>>> > > > > > > > >> > >> offsets.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the
> limited address
> >>>> > > > > space in Java
> >>>> > > > > > > > >> > >> and
> >>>> > > > > > > > >> > >> >> > > calling
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
> >>>> > > > > > > > >> > >> >> > > > > > >>> reference
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
> suboptimal.
> >>>> > > > > If a consumer
> >>>> > > > > > > > >> > >> uses a
> >>>> > > > > > > > >> > >> >> > > > "Large"
> >>>> > > > > > > > >> > >> >> > > > > > >>> type
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is
> because they
> >>>> > > > > need the ability
> >>>> > > > > > > > >> > >> to store
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
> >>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
> column,
> >>>> > > > > otherwise it is
> >>>> > > > > > > > >> > >> just
> >>>> > > > > > > > >> > >> >> > > wasting
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
> >>>> > > > > > > > >> > >> >> > > > > > >>> [1].
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if
> the individual
> >>>> > > > > elements
> >>>> > > > > > > > >> > >> (lists or
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
> >>>> > > > > > > > >> > >> >> > > > > > >>> are
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space
> is wasted in
> >>>> > > > > proportion,
> >>>> > > > > > > > >> > >> so it may
> >>>> > > > > > > > >> > >> >> > > be
> >>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
> create a
> >>>> > > > > "Large" type
> >>>> > > > > > > > >> > >> array.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
> theoretically
> >>>> > > > there
> >>>> > > > > might be some
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
> >>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures
> to using
> >>>> > > > the
> >>>> > > > > native word
> >>>> > > > > > > > >> > >> sizes.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common
> 64-bit
> >>>> > > > > architectures don't do
> >>>> > > > > > > > >> > >> that, as
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> >>>> > > > > > > > >> > >> >> > > > > > >>> is an
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common
> integer size even
> >>>> > > > > in
> >>>> > > > > > > > >> > >> high-performance
> >>>> > > > > > > > >> > >> >> > > > code.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
> >>>> > > > > > > > >> > >> >> > > > > > >>> > >
> >>>> > > > > > > > >> > >> >> > > > > > >>>
> >>>> > > > > > > > >> > >> >> > > > > > >>
> >>>> > > > > > > > >> > >> >> > > >
> >>>> > > > > > > > >> > >> >> > >
> >>>> > > > > > > > >> > >>
> >>>> > > > > > > > >> > >
> >>>> > > > >
> >>>> > > >
>
Re: Timeline for 0.15.0 release
Posted by Neal Richardson <ne...@gmail.com>.
IMO it's too risky to add something that adds a dependency
(aws-sdk-cpp) on the day of cutting a release.
Neal
On Wed, Sep 25, 2019 at 12:54 PM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> We don't have a comprehensive documentation yet, so let's postpone it.
>
>
> On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <sz...@gmail.com> wrote:
>>
>> The S3 python bindings would be a nice addition to the release.
>> I don't think we should block on this but the PR is ready. Opinions?
>> https://github.com/apache/arrow/pull/5423
>>
>>
>>
>>
>> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <em...@gmail.com> wrote:
>>>
>>> OK, I'll start the process today. I'll send up e-mail updates as I make progress.
>>>
>>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> Yes, all systems go as far as I'm concerned.
>>>>
>>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
>>>> <ne...@gmail.com> wrote:
>>>> >
>>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
>>>> > and it looks like the LICENSE issue is being resolved as I type. So
>>>> > are we good to go now?
>>>> >
>>>> > Neal
>>>> >
>>>> >
>>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com> wrote:
>>>> > >
>>>> > > I found a last minute issue with DataFusion (Rust) and would appreciate it
>>>> > > if we could merge ARROW-6086 (PR is
>>>> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
>>>> > >
>>>> > > Thanks,
>>>> > >
>>>> > > Andy.
>>>> > >
>>>> > >
>>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <em...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > OK, I'm going to postpone cutting a release until tomorrow (hoping we can
>>>> > > > issues resolved by then).. I'll also try to review the third-party
>>>> > > > additions since 14.x.
>>>> > > >
>>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
>>>> > > >
>>>> > > > > I found a licensing issue
>>>> > > > >
>>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
>>>> > > > >
>>>> > > > > It might be worth examining third party code added to the project
>>>> > > > > since 0.14.x to make sure there are no other such issues.
>>>> > > > >
>>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com>
>>>> > > > wrote:
>>>> > > > > >
>>>> > > > > > I have diagnosed the problem (Thrift "string" data must be UTF-8,
>>>> > > > > > cannot be arbitrary binary) and am working on a patch right now
>>>> > > > > >
>>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com>
>>>> > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > I just opened
>>>> > > > > > >
>>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
>>>> > > > > > >
>>>> > > > > > > Please don't cut an RC until I have an opportunity to diagnose this,
>>>> > > > > > > will report back.
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com>
>>>> > > > > wrote:
>>>> > > > > > > >
>>>> > > > > > > > I'm investigating a possible Parquet-related compatibility bug
>>>> > > > that I
>>>> > > > > > > > encountered through some routine testing / benchmarking. I'll
>>>> > > > report
>>>> > > > > > > > back once I figure out what is going on (if anything)
>>>> > > > > > > >
>>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
>>>> > > > > emkornfield@gmail.com> wrote:
>>>> > > > > > > > >>
>>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
>>>> > > > > get it
>>>> > > > > > > > >> signed by another PMC member), but is not 100% essential.
>>>> > > > > > > > >
>>>> > > > > > > > > That won't be an option for me this week (it seems like I would
>>>> > > > > need to meet one face-to-face). I'll try to get the GPG checked in and
>>>> > > > the
>>>> > > > > rest of the pre-requisites done tomorrow (Monday) to hopefully start the
>>>> > > > > release on Tuesday (hopefully we can solve the last blocker/integration
>>>> > > > > tests by then).
>>>> > > > > > > > >
>>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
>>>> > > > wesmckinn@gmail.com>
>>>> > > > > wrote:
>>>> > > > > > > > >>
>>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
>>>> > > > > get it
>>>> > > > > > > > >> signed by another PMC member), but is not 100% essential.
>>>> > > > > > > > >>
>>>> > > > > > > > >> Speaking of the release, there are at least 2 code changes I
>>>> > > > still
>>>> > > > > > > > >> want to get in
>>>> > > > > > > > >>
>>>> > > > > > > > >> ARROW-5717
>>>> > > > > > > > >> ARROW-6353
>>>> > > > > > > > >>
>>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once the build
>>>> > > > is
>>>> > > > > green.
>>>> > > > > > > > >>
>>>> > > > > > > > >> There are a couple of Rust patches still marked for 0.15. The
>>>> > > > rest
>>>> > > > > > > > >> seems to be documentation and a couple of integration test
>>>> > > > > failures we
>>>> > > > > > > > >> should see about fixing in time.
>>>> > > > > > > > >>
>>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
>>>> > > > > emkornfield@gmail.com> wrote:
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Thanks Krisztián and Wes,
>>>> > > > > > > > >> > I've gone ahead and started registering myself on all the
>>>> > > > > packaging sites.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Is there any review process when adding my GPG key to the SVN
>>>> > > > > file? [1]
>>>> > > > > > > > >> > doesn't seem to mention explicitly.
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > Thanks,
>>>> > > > > > > > >> > Micah
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
>>>> > > > > szucs.krisztian@gmail.com>
>>>> > > > > > > > >> > wrote:
>>>> > > > > > > > >> >
>>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
>>>> > > > > wesmckinn@gmail.com> wrote:
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
>>>> > > > > emkornfield@gmail.com>
>>>> > > > > > > > >> > >> wrote:
>>>> > > > > > > > >> > >> >>
>>>> > > > > > > > >> > >> >> The process should be well documented at this point but
>>>> > > > > there are a
>>>> > > > > > > > >> > >> >> number of steps.
>>>> > > > > > > > >> > >> >
>>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the release?
>>>> > > > Are
>>>> > > > > there
>>>> > > > > > > > >> > >> instructions for the adding the code signing Key to SVN?
>>>> > > > > > > > >> > >> >
>>>> > > > > > > > >> > >> > I will make a go of it. i will try to mitigate any
>>>> > > > > internet issues by
>>>> > > > > > > > >> > >> doing the process for a cloud instance (I assume that isn't
>>>> > > > > a problem?).
>>>> > > > > > > > >> > >> >
>>>> > > > > > > > >> > >>
>>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for producing
>>>> > > > an
>>>> > > > > RC may be
>>>> > > > > > > > >> > >> time consuming, but you are welcome to try. Krisztian --
>>>> > > > are
>>>> > > > > you
>>>> > > > > > > > >> > >> available next week to help Micah and potentially take over
>>>> > > > > producing
>>>> > > > > > > > >> > >> the RC if there are issues?
>>>> > > > > > > > >> > >>
>>>> > > > > > > > >> > > Sure, I'll be available next week. We can also grant access
>>>> > > > to
>>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because configuring
>>>> > > > all
>>>> > > > > > > > >> > > the CI backends can be time consuming.
>>>> > > > > > > > >> > >
>>>> > > > > > > > >> > >>
>>>> > > > > > > > >> > >> > Thanks,
>>>> > > > > > > > >> > >> > Micah
>>>> > > > > > > > >> > >> >
>>>> > > > > > > > >> > >> > [1]
>>>> > > > > > > > >> > >>
>>>> > > > >
>>>> > > > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>>>> > > > > > > > >> > >> >
>>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
>>>> > > > > wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> wrote:
>>>> > > > > > > > >> > >> >>
>>>> > > > > > > > >> > >> >> The process should be well documented at this point but
>>>> > > > > there are a
>>>> > > > > > > > >> > >> >> number of steps. Note that you need to add your code
>>>> > > > > signing key to
>>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I
>>>> > > > > think it's fine
>>>> > > > > > > > >> > >> >> to hand off the process to others after the VOTE but it
>>>> > > > > would be
>>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with producing the
>>>> > > > > source and
>>>> > > > > > > > >> > >> >> binary artifacts for the vote
>>>> > > > > > > > >> > >> >>
>>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
>>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > SGTM, as well.
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > I should have a little bit of time next week if I can
>>>> > > > > help as RM but
>>>> > > > > > > > >> > >> I have
>>>> > > > > > > > >> > >> >> > a couple of concerns:
>>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble downloading and
>>>> > > > > validating
>>>> > > > > > > > >> > >> releases. I'm a
>>>> > > > > > > > >> > >> >> > bit worried, that I might have similar problems doing
>>>> > > > > the necessary
>>>> > > > > > > > >> > >> uploads.
>>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be not great, I
>>>> > > > > don't know if
>>>> > > > > > > > >> > >> this
>>>> > > > > > > > >> > >> >> > would make it even less likely to be successful.
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I would have to
>>>> > > > > abandon the
>>>> > > > > > > > >> > >> process
>>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could serve as a
>>>> > > > > backup? Are the
>>>> > > > > > > > >> > >> steps
>>>> > > > > > > > >> > >> >> > well documented?
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > Thanks,
>>>> > > > > > > > >> > >> >> > Micah
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
>>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
>>>> > > > > > > > >> > >> >> > wrote:
>>>> > > > > > > > >> > >> >> >
>>>> > > > > > > > >> > >> >> > > Sounds good to me.
>>>> > > > > > > > >> > >> >> > >
>>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
>>>> > > > > > > > >> > >> >> > >
>>>> > > > > > > > >> > >> >> > > Neal
>>>> > > > > > > > >> > >> >> > >
>>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
>>>> > > > > wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> wrote:
>>>> > > > > > > > >> > >> >> > >
>>>> > > > > > > > >> > >> >> > > > hi all,
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to be able to
>>>> > > > > make the 0.15.0
>>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils down" at the end
>>>> > > > > of this week
>>>> > > > > > > > >> > >> and
>>>> > > > > > > > >> > >> >> > > > see if a release candidate can be produced next
>>>> > > > > Monday September
>>>> > > > > > > > >> > >> 23.
>>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >> >> > > > Thanks,
>>>> > > > > > > > >> > >> >> > > > Wes
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
>>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> >> > > wrote:
>>>> > > > > > > > >> > >> >> > > > >
>>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to
>>>> > > > > amend the
>>>> > > > > > > > >> > >> Format docs
>>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and also update
>>>> > > > the
>>>> > > > > C++ library
>>>> > > > > > > > >> > >> >> > > > >
>>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
>>>> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
>>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
>>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15 release,
>>>> > > > correct?
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are
>>>> > > > > ready to be
>>>> > > > > > > > >> > >> merged into
>>>> > > > > > > > >> > >> >> > > > the alignment branch -
>>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > Eric
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
>>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
>>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
>>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
>>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
>>>> > > > > niki.lj@aliyun.com>
>>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth to help
>>>> > > > > with some of
>>>> > > > > > > > >> > >> the
>>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going into the
>>>> > > > > weekend.
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
>>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> >> > > > wrote:
>>>> > > > > > > > >> > >> >> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
>>>> > > > > > > > >> > >> >> > > > > > >
>>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging and
>>>> > > > > integration builds
>>>> > > > > > > > >> > >> things
>>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for being in release
>>>> > > > > readiness by
>>>> > > > > > > > >> > >> the end
>>>> > > > > > > > >> > >> >> > > of
>>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning
>>>> > > > > to be working
>>>> > > > > > > > >> > >> to close
>>>> > > > > > > > >> > >> >> > > as
>>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to help with
>>>> > > > > the ongoing
>>>> > > > > > > > >> > >> alignment
>>>> > > > > > > > >> > >> >> > > > fixes.
>>>> > > > > > > > >> > >> >> > > > > > >
>>>> > > > > > > > >> > >> >> > > > > > > Wes
>>>> > > > > > > > >> > >> >> > > > > > >
>>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
>>>> > > > Kornfield
>>>> > > > > <
>>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
>>>> > > > > > > > >> > >> >> > > > >
>>>> > > > > > > > >> > >> >> > > > > > > wrote:
>>>> > > > > > > > >> > >> >> > > > > > >
>>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of
>>>> > > > > the current
>>>> > > > > > > > >> > >> issues:
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >>
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>>>> > > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
>>>> > > > > 40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >>
>>>> > > > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >>
>>>> > > > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
>>>> > > > McKinney <
>>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> >> > > > wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in
>>>> > > > a
>>>> > > > > position to
>>>> > > > > > > > >> > >> release
>>>> > > > > > > > >> > >> >> > > at
>>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope that
>>>> > > > one
>>>> > > > > more week of
>>>> > > > > > > > >> > >> work (or
>>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us there.
>>>> > > > Aside
>>>> > > > > from merging
>>>> > > > > > > > >> > >> the
>>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make sure
>>>> > > > > that our
>>>> > > > > > > > >> > >> packaging jobs
>>>> > > > > > > > >> > >> >> > > > > > >>> required for the release candidate are all
>>>> > > > > working.
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from the
>>>> > > > > 0.15.0 backlog
>>>> > > > > > > > >> > >> that they
>>>> > > > > > > > >> > >> >> > > > don't
>>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end of next week
>>>> > > > > that would
>>>> > > > > > > > >> > >> help focus
>>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in
>>>> > > > > 0.15.0 still).
>>>> > > > > > > > >> > >> I am
>>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small features
>>>> > > > > related to
>>>> > > > > > > > >> > >> dictionaries
>>>> > > > > > > > >> > >> >> > > > while
>>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still open.
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
>>>> > > > > McKinney <
>>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
>>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to release the
>>>> > > > week
>>>> > > > > of September
>>>> > > > > > > > >> > >> 9, so
>>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be completed by
>>>> > > > > end of next
>>>> > > > > > > > >> > >> week.
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
>>>> > > > protocol
>>>> > > > > alignment
>>>> > > > > > > > >> > >> changes for
>>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I
>>>> > > > think
>>>> > > > > that getting
>>>> > > > > > > > >> > >> the
>>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main barrier
>>>> > > > > to releasing.
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
>>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
>>>> > > > > > > > >> > >> >> > > > > > >>> >
>>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
>>>> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
>>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think
>>>> > > > > of several
>>>> > > > > > > > >> > >> bugs that
>>>> > > > > > > > >> > >> >> > > > need
>>>> > > > > > > > >> > >> >> > > > > > >>> > > to
>>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are
>>>> > > > > required in
>>>> > > > > > > > >> > >> IPC streams
>>>> > > > > > > > >> > >> >> > > > > > >>> > > even
>>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
>>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review now, however
>>>> > > > > through this
>>>> > > > > > > > >> > >> PR we find
>>>> > > > > > > > >> > >> >> > > > > > >>> > > that
>>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading and
>>>> > > > > writing
>>>> > > > > > > > >> > >> dictionaries in IPC
>>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since
>>>> > > > it
>>>> > > > > assumes all
>>>> > > > > > > > >> > >> >> > > > dictionaries
>>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see details in
>>>> > > > > PR comments,
>>>> > > > > > > > >> > >> and this
>>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15).
>>>> > > > > @Micah Kornfield
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as
>>>> > > > > strings in
>>>> > > > > > > > >> > >> integration
>>>> > > > > > > > >> > >> >> > > > test
>>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already checked in,
>>>> > > > other
>>>> > > > > > > > >> > >> implementations
>>>> > > > > > > > >> > >> >> > > seems
>>>> > > > > > > > >> > >> >> > > > not.
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
>>>> > > > > JdbcAdapter[4]
>>>> > > > > > > > >> > >> Caused by
>>>> > > > > > > > >> > >> >> > > trying
>>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one contiguous
>>>> > > > > batch, fixed
>>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for iteratively
>>>> > > > > reading in
>>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
>>>> > > > > > > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>>>> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>>>> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >>
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >>
>>>> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >>
>>>> > > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > ----------------------------------------------------------------
>>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
>>>> > > > > wesmckinn@gmail.com> Send
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
>>>> > > > > > > > >> > >> dev@arrow.apache.org>
>>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
>>>> > > > release
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing
>>>> > > > > the 0.15.0
>>>> > > > > > > > >> > >> backlog some
>>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help
>>>> > > > with
>>>> > > > > grooming
>>>> > > > > > > > >> > >> >> > > (particularly
>>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than C++/Python
>>>> > > > > where I'm
>>>> > > > > > > > >> > >> focusing) that
>>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have been
>>>> > > > > almost 500 JIRA
>>>> > > > > > > > >> > >> issues
>>>> > > > > > > > >> > >> >> > > opened
>>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
>>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure
>>>> > > > > to check
>>>> > > > > > > > >> > >> whether
>>>> > > > > > > > >> > >> >> > > there's
>>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs
>>>> > > > > that we should
>>>> > > > > > > > >> > >> try to
>>>> > > > > > > > >> > >> >> > > fix
>>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes
>>>> > > > > McKinney
>>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
>>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1
>>>> > > > > seems to be
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
>>>> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
>>>> > > > 40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the
>>>> > > > > Windows
>>>> > > > > > > > >> > >> changes in
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
>>>> > > > > > > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
>>>> > > > > volunteer would look
>>>> > > > > > > > >> > >> into what
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
>>>> > > > > > > > >> > >> >> > > > > > >>> wrong
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows.
>>>> > > > > Otherwise
>>>> > > > > > > > >> > >> 0.15.0 Windows
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>>>> > > > > > > > >> > >> 40microsoft.com
>>>> > > > > > > > >> > >> >> > > > %7Ccbea
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM
>>>> > > > > Antoine Pitrou <
>>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
>>>> > > > > Micah
>>>> > > > > > > > >> > >> Kornfield
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could have
>>>> > > > > 32-bit array
>>>> > > > > > > > >> > >> lengths and
>>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if
>>>> > > > we
>>>> > > > > wanted (we
>>>> > > > > > > > >> > >> just
>>>> > > > > > > > >> > >> >> > > > wouldn't
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
>>>> > > > > > > > >> > >> >> > > > > > >>> able to
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with more
>>>> > > > > than INT32_MAX
>>>> > > > > > > > >> > >> elements).
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do
>>>> > > > > this in C++
>>>> > > > > > > > >> > >> but we
>>>> > > > > > > > >> > >> >> > > > don't.
>>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
>>>> > > > "Large"
>>>> > > > > types if we
>>>> > > > > > > > >> > >> did.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much
>>>> > > > > space as 32-bit
>>>> > > > > > > > >> > >> >> > > offsets,
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
>>>> > > > > > > > >> > >> >> > > > > > >>> you're
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or
>>>> > > > > strings,
>>>> > > > > > > > >> > >> 32-bit
>>>> > > > > > > > >> > >> >> > > offsets
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with
>>>> > > > > 64-bit array
>>>> > > > > > > > >> > >> lengths from
>>>> > > > > > > > >> > >> >> > > > the
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
>>>> > > > > > > > >> > >> >> > > > > > >>> it would
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types
>>>> > > > > with 32-bit
>>>> > > > > > > > >> > >> offsets.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address
>>>> > > > > space in Java
>>>> > > > > > > > >> > >> and
>>>> > > > > > > > >> > >> >> > > calling
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
>>>> > > > > > > > >> > >> >> > > > > > >>> reference
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal.
>>>> > > > > If a consumer
>>>> > > > > > > > >> > >> uses a
>>>> > > > > > > > >> > >> >> > > > "Large"
>>>> > > > > > > > >> > >> >> > > > > > >>> type
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because they
>>>> > > > > need the ability
>>>> > > > > > > > >> > >> to store
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
>>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a column,
>>>> > > > > otherwise it is
>>>> > > > > > > > >> > >> just
>>>> > > > > > > > >> > >> >> > > wasting
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
>>>> > > > > > > > >> > >> >> > > > > > >>> [1].
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual
>>>> > > > > elements
>>>> > > > > > > > >> > >> (lists or
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
>>>> > > > > > > > >> > >> >> > > > > > >>> are
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in
>>>> > > > > proportion,
>>>> > > > > > > > >> > >> so it may
>>>> > > > > > > > >> > >> >> > > be
>>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always create a
>>>> > > > > "Large" type
>>>> > > > > > > > >> > >> array.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically
>>>> > > > there
>>>> > > > > might be some
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
>>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using
>>>> > > > the
>>>> > > > > native word
>>>> > > > > > > > >> > >> sizes.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
>>>> > > > > architectures don't do
>>>> > > > > > > > >> > >> that, as
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
>>>> > > > > > > > >> > >> >> > > > > > >>> is an
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer size even
>>>> > > > > in
>>>> > > > > > > > >> > >> high-performance
>>>> > > > > > > > >> > >> >> > > > code.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>>> > > > > > > > >> > >> >> > > > > > >>>
>>>> > > > > > > > >> > >> >> > > > > > >>
>>>> > > > > > > > >> > >> >> > > >
>>>> > > > > > > > >> > >> >> > >
>>>> > > > > > > > >> > >>
>>>> > > > > > > > >> > >
>>>> > > > >
>>>> > > >
Re: Timeline for 0.15.0 release
Posted by Krisztián Szűcs <sz...@gmail.com>.
We don't have a comprehensive documentation yet, so let's postpone it.
On Wed, Sep 25, 2019 at 9:48 PM Krisztián Szűcs <sz...@gmail.com>
wrote:
> The S3 python bindings would be a nice addition to the release.
> I don't think we should block on this but the PR is ready. Opinions?
> https://github.com/apache/arrow/pull/5423
>
>
>
>
> On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> OK, I'll start the process today. I'll send up e-mail updates as I make
>> progress.
>>
>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com> wrote:
>>
>>> Yes, all systems go as far as I'm concerned.
>>>
>>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
>>> <ne...@gmail.com> wrote:
>>> >
>>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
>>> > and it looks like the LICENSE issue is being resolved as I type. So
>>> > are we good to go now?
>>> >
>>> > Neal
>>> >
>>> >
>>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com>
>>> wrote:
>>> > >
>>> > > I found a last minute issue with DataFusion (Rust) and would
>>> appreciate it
>>> > > if we could merge ARROW-6086 (PR is
>>> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > Andy.
>>> > >
>>> > >
>>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > OK, I'm going to postpone cutting a release until tomorrow (hoping
>>> we can
>>> > > > issues resolved by then).. I'll also try to review the third-party
>>> > > > additions since 14.x.
>>> > > >
>>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > > I found a licensing issue
>>> > > > >
>>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
>>> > > > >
>>> > > > > It might be worth examining third party code added to the project
>>> > > > > since 0.14.x to make sure there are no other such issues.
>>> > > > >
>>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <
>>> wesmckinn@gmail.com>
>>> > > > wrote:
>>> > > > > >
>>> > > > > > I have diagnosed the problem (Thrift "string" data must be
>>> UTF-8,
>>> > > > > > cannot be arbitrary binary) and am working on a patch right now
>>> > > > > >
>>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
>>> wesmckinn@gmail.com>
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > I just opened
>>> > > > > > >
>>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
>>> > > > > > >
>>> > > > > > > Please don't cut an RC until I have an opportunity to
>>> diagnose this,
>>> > > > > > > will report back.
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
>>> wesmckinn@gmail.com>
>>> > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > I'm investigating a possible Parquet-related compatibility
>>> bug
>>> > > > that I
>>> > > > > > > > encountered through some routine testing / benchmarking.
>>> I'll
>>> > > > report
>>> > > > > > > > back once I figure out what is going on (if anything)
>>> > > > > > > >
>>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
>>> > > > > emkornfield@gmail.com> wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
>>> you can
>>> > > > > get it
>>> > > > > > > > >> signed by another PMC member), but is not 100%
>>> essential.
>>> > > > > > > > >
>>> > > > > > > > > That won't be an option for me this week (it seems like
>>> I would
>>> > > > > need to meet one face-to-face). I'll try to get the GPG checked
>>> in and
>>> > > > the
>>> > > > > rest of the pre-requisites done tomorrow (Monday) to hopefully
>>> start the
>>> > > > > release on Tuesday (hopefully we can solve the last
>>> blocker/integration
>>> > > > > tests by then).
>>> > > > > > > > >
>>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
>>> > > > wesmckinn@gmail.com>
>>> > > > > wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
>>> you can
>>> > > > > get it
>>> > > > > > > > >> signed by another PMC member), but is not 100%
>>> essential.
>>> > > > > > > > >>
>>> > > > > > > > >> Speaking of the release, there are at least 2 code
>>> changes I
>>> > > > still
>>> > > > > > > > >> want to get in
>>> > > > > > > > >>
>>> > > > > > > > >> ARROW-5717
>>> > > > > > > > >> ARROW-6353
>>> > > > > > > > >>
>>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once
>>> the build
>>> > > > is
>>> > > > > green.
>>> > > > > > > > >>
>>> > > > > > > > >> There are a couple of Rust patches still marked for
>>> 0.15. The
>>> > > > rest
>>> > > > > > > > >> seems to be documentation and a couple of integration
>>> test
>>> > > > > failures we
>>> > > > > > > > >> should see about fixing in time.
>>> > > > > > > > >>
>>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
>>> > > > > emkornfield@gmail.com> wrote:
>>> > > > > > > > >> >
>>> > > > > > > > >> > Thanks Krisztián and Wes,
>>> > > > > > > > >> > I've gone ahead and started registering myself on all
>>> the
>>> > > > > packaging sites.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Is there any review process when adding my GPG key to
>>> the SVN
>>> > > > > file? [1]
>>> > > > > > > > >> > doesn't seem to mention explicitly.
>>> > > > > > > > >> >
>>> > > > > > > > >> > Thanks,
>>> > > > > > > > >> > Micah
>>> > > > > > > > >> >
>>> > > > > > > > >> > [1]
>>> https://www.apache.org/dev/version-control.html#https-svn
>>> > > > > > > > >> >
>>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
>>> > > > > szucs.krisztian@gmail.com>
>>> > > > > > > > >> > wrote:
>>> > > > > > > > >> >
>>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
>>> > > > > wesmckinn@gmail.com> wrote:
>>> > > > > > > > >> > >
>>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
>>> > > > > emkornfield@gmail.com>
>>> > > > > > > > >> > >> wrote:
>>> > > > > > > > >> > >> >>
>>> > > > > > > > >> > >> >> The process should be well documented at this
>>> point but
>>> > > > > there are a
>>> > > > > > > > >> > >> >> number of steps.
>>> > > > > > > > >> > >> >
>>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
>>> release?
>>> > > > Are
>>> > > > > there
>>> > > > > > > > >> > >> instructions for the adding the code signing Key
>>> to SVN?
>>> > > > > > > > >> > >> >
>>> > > > > > > > >> > >> > I will make a go of it. i will try to mitigate
>>> any
>>> > > > > internet issues by
>>> > > > > > > > >> > >> doing the process for a cloud instance (I assume
>>> that isn't
>>> > > > > a problem?).
>>> > > > > > > > >> > >> >
>>> > > > > > > > >> > >>
>>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
>>> producing
>>> > > > an
>>> > > > > RC may be
>>> > > > > > > > >> > >> time consuming, but you are welcome to try.
>>> Krisztian --
>>> > > > are
>>> > > > > you
>>> > > > > > > > >> > >> available next week to help Micah and potentially
>>> take over
>>> > > > > producing
>>> > > > > > > > >> > >> the RC if there are issues?
>>> > > > > > > > >> > >>
>>> > > > > > > > >> > > Sure, I'll be available next week. We can also
>>> grant access
>>> > > > to
>>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
>>> configuring
>>> > > > all
>>> > > > > > > > >> > > the CI backends can be time consuming.
>>> > > > > > > > >> > >
>>> > > > > > > > >> > >>
>>> > > > > > > > >> > >> > Thanks,
>>> > > > > > > > >> > >> > Micah
>>> > > > > > > > >> > >> >
>>> > > > > > > > >> > >> > [1]
>>> > > > > > > > >> > >>
>>> > > > >
>>> > > >
>>> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>>> > > > > > > > >> > >> >
>>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
>>> > > > > wesmckinn@gmail.com>
>>> > > > > > > > >> > >> wrote:
>>> > > > > > > > >> > >> >>
>>> > > > > > > > >> > >> >> The process should be well documented at this
>>> point but
>>> > > > > there are a
>>> > > > > > > > >> > >> >> number of steps. Note that you need to add your
>>> code
>>> > > > > signing key to
>>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to
>>> do). I
>>> > > > > think it's fine
>>> > > > > > > > >> > >> >> to hand off the process to others after the
>>> VOTE but it
>>> > > > > would be
>>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
>>> producing the
>>> > > > > source and
>>> > > > > > > > >> > >> >> binary artifacts for the vote
>>> > > > > > > > >> > >> >>
>>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah
>>> Kornfield <
>>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > SGTM, as well.
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > I should have a little bit of time next week
>>> if I can
>>> > > > > help as RM but
>>> > > > > > > > >> > >> I have
>>> > > > > > > > >> > >> >> > a couple of concerns:
>>> > > > > > > > >> > >> >> > 1. In the past I've had trouble downloading
>>> and
>>> > > > > validating
>>> > > > > > > > >> > >> releases. I'm a
>>> > > > > > > > >> > >> >> > bit worried, that I might have similar
>>> problems doing
>>> > > > > the necessary
>>> > > > > > > > >> > >> uploads.
>>> > > > > > > > >> > >> >> > 2. My internet connection will likely be not
>>> great, I
>>> > > > > don't know if
>>> > > > > > > > >> > >> this
>>> > > > > > > > >> > >> >> > would make it even less likely to be
>>> successful.
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > Does it become problematic if somehow I would
>>> have to
>>> > > > > abandon the
>>> > > > > > > > >> > >> process
>>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could serve
>>> as a
>>> > > > > backup? Are the
>>> > > > > > > > >> > >> steps
>>> > > > > > > > >> > >> >> > well documented?
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > Thanks,
>>> > > > > > > > >> > >> >> > Micah
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
>>> Richardson <
>>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
>>> > > > > > > > >> > >> >> > wrote:
>>> > > > > > > > >> > >> >> >
>>> > > > > > > > >> > >> >> > > Sounds good to me.
>>> > > > > > > > >> > >> >> > >
>>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
>>> volunteers?
>>> > > > > > > > >> > >> >> > >
>>> > > > > > > > >> > >> >> > > Neal
>>> > > > > > > > >> > >> >> > >
>>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes
>>> McKinney <
>>> > > > > wesmckinn@gmail.com>
>>> > > > > > > > >> > >> wrote:
>>> > > > > > > > >> > >> >> > >
>>> > > > > > > > >> > >> >> > > > hi all,
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to be
>>> able to
>>> > > > > make the 0.15.0
>>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils down"
>>> at the end
>>> > > > > of this week
>>> > > > > > > > >> > >> and
>>> > > > > > > > >> > >> >> > > > see if a release candidate can be
>>> produced next
>>> > > > > Monday September
>>> > > > > > > > >> > >> 23.
>>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >> >> > > > Thanks,
>>> > > > > > > > >> > >> >> > > > Wes
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
>>> McKinney <
>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>> > > > > > > > >> > >> >> > > wrote:
>>> > > > > > > > >> > >> >> > > > >
>>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
>>> planning to
>>> > > > > amend the
>>> > > > > > > > >> > >> Format docs
>>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and also
>>> update
>>> > > > the
>>> > > > > C++ library
>>> > > > > > > > >> > >> >> > > > >
>>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric
>>> Erhardt
>>> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
>>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
>>> > > > > > > > >> > >> >> > > > branch into master before the 0.15
>>> release,
>>> > > > correct?
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
>>> changes are
>>> > > > > ready to be
>>> > > > > > > > >> > >> merged into
>>> > > > > > > > >> > >> >> > > > the alignment branch -
>>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > Eric
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
>>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
>>> emkornfield@gmail.com>
>>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
>>> 10:24 PM
>>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <wesmckinn@gmail.com
>>> >
>>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
>>> niki.lj <
>>> > > > > niki.lj@aliyun.com>
>>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
>>> release
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth
>>> to help
>>> > > > > with some of
>>> > > > > > > > >> > >> the
>>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going
>>> into the
>>> > > > > weekend.
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes
>>> McKinney <
>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>> > > > > > > > >> > >> >> > > > wrote:
>>> > > > > > > > >> > >> >> > > > > >
>>> > > > > > > > >> > >> >> > > > > > > Hi folks,
>>> > > > > > > > >> > >> >> > > > > > >
>>> > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging
>>> and
>>> > > > > integration builds
>>> > > > > > > > >> > >> things
>>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for being
>>> in release
>>> > > > > readiness by
>>> > > > > > > > >> > >> the end
>>> > > > > > > > >> > >> >> > > of
>>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm
>>> planning
>>> > > > > to be working
>>> > > > > > > > >> > >> to close
>>> > > > > > > > >> > >> >> > > as
>>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
>>> help with
>>> > > > > the ongoing
>>> > > > > > > > >> > >> alignment
>>> > > > > > > > >> > >> >> > > > fixes.
>>> > > > > > > > >> > >> >> > > > > > >
>>> > > > > > > > >> > >> >> > > > > > > Wes
>>> > > > > > > > >> > >> >> > > > > > >
>>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
>>> > > > Kornfield
>>> > > > > <
>>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
>>> > > > > > > > >> > >> >> > > > >
>>> > > > > > > > >> > >> >> > > > > > > wrote:
>>> > > > > > > > >> > >> >> > > > > > >
>>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
>>> dashboard of
>>> > > > > the current
>>> > > > > > > > >> > >> issues:
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >>
>>> > > > >
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
>>> > > > > > > > >> > >> >> > > >
>>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>>> > > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
>>> > > > > 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >>
>>> > > > >
>>> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >>
>>> > > > >
>>> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
>>> > > > McKinney <
>>> > > > > > > > >> > >> wesmckinn@gmail.com>
>>> > > > > > > > >> > >> >> > > > wrote:
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > > > > >>> hi all,
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going
>>> to be in
>>> > > > a
>>> > > > > position to
>>> > > > > > > > >> > >> release
>>> > > > > > > > >> > >> >> > > at
>>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I
>>> hope that
>>> > > > one
>>> > > > > more week of
>>> > > > > > > > >> > >> work (or
>>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
>>> there.
>>> > > > Aside
>>> > > > > from merging
>>> > > > > > > > >> > >> the
>>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to
>>> make sure
>>> > > > > that our
>>> > > > > > > > >> > >> packaging jobs
>>> > > > > > > > >> > >> >> > > > > > >>> required for the release
>>> candidate are all
>>> > > > > working.
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from
>>> the
>>> > > > > 0.15.0 backlog
>>> > > > > > > > >> > >> that they
>>> > > > > > > > >> > >> >> > > > don't
>>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end of
>>> next week
>>> > > > > that would
>>> > > > > > > > >> > >> help focus
>>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78
>>> issues in
>>> > > > > 0.15.0 still).
>>> > > > > > > > >> > >> I am
>>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
>>> features
>>> > > > > related to
>>> > > > > > > > >> > >> dictionaries
>>> > > > > > > > >> > >> >> > > > while
>>> > > > > > > > >> > >> >> > > > > > >>> the release window is still open.
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > > > > >>> - Wes
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM
>>> Wes
>>> > > > > McKinney <
>>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > hi,
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to
>>> release the
>>> > > > week
>>> > > > > of September
>>> > > > > > > > >> > >> 9, so
>>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
>>> completed by
>>> > > > > end of next
>>> > > > > > > > >> > >> week.
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
>>> > > > protocol
>>> > > > > alignment
>>> > > > > > > > >> > >> changes for
>>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days
>>> -- I
>>> > > > think
>>> > > > > that getting
>>> > > > > > > > >> > >> the
>>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main
>>> barrier
>>> > > > > to releasing.
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
>>> > > > > > > > >> > >> >> > > > > > >>> > Wes
>>> > > > > > > > >> > >> >> > > > > > >>> >
>>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25
>>> PM Ji Liu
>>> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I
>>> can think
>>> > > > > of several
>>> > > > > > > > >> > >> bugs that
>>> > > > > > > > >> > >> >> > > > need
>>> > > > > > > > >> > >> >> > > > > > >>> > > to
>>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
>>> entries are
>>> > > > > required in
>>> > > > > > > > >> > >> IPC streams
>>> > > > > > > > >> > >> >> > > > > > >>> > > even
>>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
>>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review now,
>>> however
>>> > > > > through this
>>> > > > > > > > >> > >> PR we find
>>> > > > > > > > >> > >> >> > > > > > >>> > > that
>>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading
>>> and
>>> > > > > writing
>>> > > > > > > > >> > >> dictionaries in IPC
>>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with
>>> spec[2] since
>>> > > > it
>>> > > > > assumes all
>>> > > > > > > > >> > >> >> > > > dictionaries
>>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see
>>> details in
>>> > > > > PR comments,
>>> > > > > > > > >> > >> and this
>>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version
>>> 0.15).
>>> > > > > @Micah Kornfield
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit
>>> ints as
>>> > > > > strings in
>>> > > > > > > > >> > >> integration
>>> > > > > > > > >> > >> >> > > > test
>>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
>>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already
>>> checked in,
>>> > > > other
>>> > > > > > > > >> > >> implementations
>>> > > > > > > > >> > >> >> > > seems
>>> > > > > > > > >> > >> >> > > > not.
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory
>>> in
>>> > > > > JdbcAdapter[4]
>>> > > > > > > > >> > >> Caused by
>>> > > > > > > > >> > >> >> > > trying
>>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
>>> contiguous
>>> > > > > batch, fixed
>>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
>>> iteratively
>>> > > > > reading in
>>> > > > > > > > >> > >> >> > > ARROW-6219[5].
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
>>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
>>> > > > > > > > >> > >> >> > > >
>>> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
>>> > > > > > > > >> > >> >> > > >
>>> %7Ccbead81a42104034a4f308d736678a45%7C72f988
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>>> > > > > > > > >> > >> >> > > >
>>> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
>>> 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>>> > > > > > > > >> > >> >> > > >
>>> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
>>> 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >>
>>> > > > >
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
>>> > > > > > > > >> > >> >> > > >
>>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
>>> > > > > > > > >> > >> >> > > >
>>> %7Ccbead81a42104034a4f308d736678a45%7C72f988
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >>
>>> > > > >
>>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >>
>>> > > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > ----------------------------------------------------------------
>>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
>>> > > > > wesmckinn@gmail.com> Send
>>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
>>> To:dev <
>>> > > > > > > > >> > >> dev@arrow.apache.org>
>>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for
>>> 0.15.0
>>> > > > release
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
>>> organizing
>>> > > > > the 0.15.0
>>> > > > > > > > >> > >> backlog some
>>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to
>>> help
>>> > > > with
>>> > > > > grooming
>>> > > > > > > > >> > >> >> > > (particularly
>>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
>>> C++/Python
>>> > > > > where I'm
>>> > > > > > > > >> > >> focusing) that
>>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have
>>> been
>>> > > > > almost 500 JIRA
>>> > > > > > > > >> > >> issues
>>> > > > > > > > >> > >> >> > > opened
>>> > > > > > > > >> > >> >> > > > > > >>> > > since the
>>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should
>>> make sure
>>> > > > > to check
>>> > > > > > > > >> > >> whether
>>> > > > > > > > >> > >> >> > > there's
>>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
>>> serious bugs
>>> > > > > that we should
>>> > > > > > > > >> > >> try to
>>> > > > > > > > >> > >> >> > > fix
>>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23
>>> PM Wes
>>> > > > > McKinney
>>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
>>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in
>>> 0.14.1
>>> > > > > seems to be
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
>>> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
>>> > > > 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause
>>> could be the
>>> > > > > Windows
>>> > > > > > > > >> > >> changes in
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
>>> 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
>>> > > > > volunteer would look
>>> > > > > > > > >> > >> into what
>>> > > > > > > > >> > >> >> > > > > > >>> > > > was
>>> > > > > > > > >> > >> >> > > > > > >>> wrong
>>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on
>>> Windows.
>>> > > > > Otherwise
>>> > > > > > > > >> > >> 0.15.0 Windows
>>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found
>>> at
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>>> > > > > > > > >> > >> 40microsoft.com
>>> > > > > > > > >> > >> >> > > > %7Ccbea
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > >
>>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>>> > > > > > > > >> > >> >> > > > > > >>> > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at
>>> 1:28 PM
>>> > > > > Antoine Pitrou <
>>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
>>> 11:17:07 -0700
>>> > > > > Micah
>>> > > > > > > > >> > >> Kornfield
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com>
>>> wrote:
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could
>>> have
>>> > > > > 32-bit array
>>> > > > > > > > >> > >> lengths and
>>> > > > > > > > >> > >> >> > > > > > >>> variable-length
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
>>> offsets if
>>> > > > we
>>> > > > > wanted (we
>>> > > > > > > > >> > >> just
>>> > > > > > > > >> > >> >> > > > wouldn't
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
>>> > > > > > > > >> > >> >> > > > > > >>> able to
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child
>>> with more
>>> > > > > than INT32_MAX
>>> > > > > > > > >> > >> elements).
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we
>>> could do
>>> > > > > this in C++
>>> > > > > > > > >> > >> but we
>>> > > > > > > > >> > >> >> > > > don't.
>>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced
>>> the
>>> > > > "Large"
>>> > > > > types if we
>>> > > > > > > > >> > >> did.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice
>>> as much
>>> > > > > space as 32-bit
>>> > > > > > > > >> > >> >> > > offsets,
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
>>> > > > > > > > >> > >> >> > > > > > >>> you're
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish
>>> lists or
>>> > > > > strings,
>>> > > > > > > > >> > >> 32-bit
>>> > > > > > > > >> > >> >> > > offsets
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even
>>> with
>>> > > > > 64-bit array
>>> > > > > > > > >> > >> lengths from
>>> > > > > > > > >> > >> >> > > > the
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
>>> > > > > > > > >> > >> >> > > > > > >>> it would
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
>>> have types
>>> > > > > with 32-bit
>>> > > > > > > > >> > >> offsets.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited
>>> address
>>> > > > > space in Java
>>> > > > > > > > >> > >> and
>>> > > > > > > > >> > >> >> > > calling
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
>>> > > > > > > > >> > >> >> > > > > > >>> reference
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
>>> suboptimal.
>>> > > > > If a consumer
>>> > > > > > > > >> > >> uses a
>>> > > > > > > > >> > >> >> > > > "Large"
>>> > > > > > > > >> > >> >> > > > > > >>> type
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is
>>> because they
>>> > > > > need the ability
>>> > > > > > > > >> > >> to store
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
>>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
>>> column,
>>> > > > > otherwise it is
>>> > > > > > > > >> > >> just
>>> > > > > > > > >> > >> >> > > wasting
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
>>> > > > > > > > >> > >> >> > > > > > >>> [1].
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the
>>> individual
>>> > > > > elements
>>> > > > > > > > >> > >> (lists or
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
>>> > > > > > > > >> > >> >> > > > > > >>> are
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is
>>> wasted in
>>> > > > > proportion,
>>> > > > > > > > >> > >> so it may
>>> > > > > > > > >> > >> >> > > be
>>> > > > > > > > >> > >> >> > > > > > >>> simpler in
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
>>> create a
>>> > > > > "Large" type
>>> > > > > > > > >> > >> array.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
>>> theoretically
>>> > > > there
>>> > > > > might be some
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
>>> > > > > > > > >> > >> >> > > > > > >>> benefits on
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to
>>> using
>>> > > > the
>>> > > > > native word
>>> > > > > > > > >> > >> sizes.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
>>> > > > > architectures don't do
>>> > > > > > > > >> > >> that, as
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
>>> > > > > > > > >> > >> >> > > > > > >>> is an
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer
>>> size even
>>> > > > > in
>>> > > > > > > > >> > >> high-performance
>>> > > > > > > > >> > >> >> > > > code.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>>> > > > > > > > >> > >> >> > > > > > >>> > >
>>> > > > > > > > >> > >> >> > > > > > >>>
>>> > > > > > > > >> > >> >> > > > > > >>
>>> > > > > > > > >> > >> >> > > >
>>> > > > > > > > >> > >> >> > >
>>> > > > > > > > >> > >>
>>> > > > > > > > >> > >
>>> > > > >
>>> > > >
>>>
>>
Re: Timeline for 0.15.0 release
Posted by Krisztián Szűcs <sz...@gmail.com>.
The S3 python bindings would be a nice addition to the release.
I don't think we should block on this but the PR is ready. Opinions?
https://github.com/apache/arrow/pull/5423
On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <em...@gmail.com>
wrote:
> OK, I'll start the process today. I'll send up e-mail updates as I make
> progress.
>
> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com> wrote:
>
>> Yes, all systems go as far as I'm concerned.
>>
>> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
>> <ne...@gmail.com> wrote:
>> >
>> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
>> > and it looks like the LICENSE issue is being resolved as I type. So
>> > are we good to go now?
>> >
>> > Neal
>> >
>> >
>> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com>
>> wrote:
>> > >
>> > > I found a last minute issue with DataFusion (Rust) and would
>> appreciate it
>> > > if we could merge ARROW-6086 (PR is
>> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
>> > >
>> > > Thanks,
>> > >
>> > > Andy.
>> > >
>> > >
>> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <
>> emkornfield@gmail.com>
>> > > wrote:
>> > >
>> > > > OK, I'm going to postpone cutting a release until tomorrow (hoping
>> we can
>> > > > issues resolved by then).. I'll also try to review the third-party
>> > > > additions since 14.x.
>> > > >
>> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com>
>> wrote:
>> > > >
>> > > > > I found a licensing issue
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/ARROW-6679
>> > > > >
>> > > > > It might be worth examining third party code added to the project
>> > > > > since 0.14.x to make sure there are no other such issues.
>> > > > >
>> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <wesmckinn@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > I have diagnosed the problem (Thrift "string" data must be
>> UTF-8,
>> > > > > > cannot be arbitrary binary) and am working on a patch right now
>> > > > > >
>> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
>> wesmckinn@gmail.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > I just opened
>> > > > > > >
>> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
>> > > > > > >
>> > > > > > > Please don't cut an RC until I have an opportunity to
>> diagnose this,
>> > > > > > > will report back.
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
>> wesmckinn@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > I'm investigating a possible Parquet-related compatibility
>> bug
>> > > > that I
>> > > > > > > > encountered through some routine testing / benchmarking.
>> I'll
>> > > > report
>> > > > > > > > back once I figure out what is going on (if anything)
>> > > > > > > >
>> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
>> > > > > emkornfield@gmail.com> wrote:
>> > > > > > > > >>
>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
>> you can
>> > > > > get it
>> > > > > > > > >> signed by another PMC member), but is not 100% essential.
>> > > > > > > > >
>> > > > > > > > > That won't be an option for me this week (it seems like I
>> would
>> > > > > need to meet one face-to-face). I'll try to get the GPG checked
>> in and
>> > > > the
>> > > > > rest of the pre-requisites done tomorrow (Monday) to hopefully
>> start the
>> > > > > release on Tuesday (hopefully we can solve the last
>> blocker/integration
>> > > > > tests by then).
>> > > > > > > > >
>> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
>> > > > wesmckinn@gmail.com>
>> > > > > wrote:
>> > > > > > > > >>
>> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
>> you can
>> > > > > get it
>> > > > > > > > >> signed by another PMC member), but is not 100% essential.
>> > > > > > > > >>
>> > > > > > > > >> Speaking of the release, there are at least 2 code
>> changes I
>> > > > still
>> > > > > > > > >> want to get in
>> > > > > > > > >>
>> > > > > > > > >> ARROW-5717
>> > > > > > > > >> ARROW-6353
>> > > > > > > > >>
>> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once the
>> build
>> > > > is
>> > > > > green.
>> > > > > > > > >>
>> > > > > > > > >> There are a couple of Rust patches still marked for
>> 0.15. The
>> > > > rest
>> > > > > > > > >> seems to be documentation and a couple of integration
>> test
>> > > > > failures we
>> > > > > > > > >> should see about fixing in time.
>> > > > > > > > >>
>> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
>> > > > > emkornfield@gmail.com> wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks Krisztián and Wes,
>> > > > > > > > >> > I've gone ahead and started registering myself on all
>> the
>> > > > > packaging sites.
>> > > > > > > > >> >
>> > > > > > > > >> > Is there any review process when adding my GPG key to
>> the SVN
>> > > > > file? [1]
>> > > > > > > > >> > doesn't seem to mention explicitly.
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks,
>> > > > > > > > >> > Micah
>> > > > > > > > >> >
>> > > > > > > > >> > [1]
>> https://www.apache.org/dev/version-control.html#https-svn
>> > > > > > > > >> >
>> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
>> > > > > szucs.krisztian@gmail.com>
>> > > > > > > > >> > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
>> > > > > wesmckinn@gmail.com> wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
>> > > > > emkornfield@gmail.com>
>> > > > > > > > >> > >> wrote:
>> > > > > > > > >> > >> >>
>> > > > > > > > >> > >> >> The process should be well documented at this
>> point but
>> > > > > there are a
>> > > > > > > > >> > >> >> number of steps.
>> > > > > > > > >> > >> >
>> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
>> release?
>> > > > Are
>> > > > > there
>> > > > > > > > >> > >> instructions for the adding the code signing Key to
>> SVN?
>> > > > > > > > >> > >> >
>> > > > > > > > >> > >> > I will make a go of it. i will try to mitigate
>> any
>> > > > > internet issues by
>> > > > > > > > >> > >> doing the process for a cloud instance (I assume
>> that isn't
>> > > > > a problem?).
>> > > > > > > > >> > >> >
>> > > > > > > > >> > >>
>> > > > > > > > >> > >> Setting up a new cloud environment suitable for
>> producing
>> > > > an
>> > > > > RC may be
>> > > > > > > > >> > >> time consuming, but you are welcome to try.
>> Krisztian --
>> > > > are
>> > > > > you
>> > > > > > > > >> > >> available next week to help Micah and potentially
>> take over
>> > > > > producing
>> > > > > > > > >> > >> the RC if there are issues?
>> > > > > > > > >> > >>
>> > > > > > > > >> > > Sure, I'll be available next week. We can also grant
>> access
>> > > > to
>> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
>> configuring
>> > > > all
>> > > > > > > > >> > > the CI backends can be time consuming.
>> > > > > > > > >> > >
>> > > > > > > > >> > >>
>> > > > > > > > >> > >> > Thanks,
>> > > > > > > > >> > >> > Micah
>> > > > > > > > >> > >> >
>> > > > > > > > >> > >> > [1]
>> > > > > > > > >> > >>
>> > > > >
>> > > >
>> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>> > > > > > > > >> > >> >
>> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
>> > > > > wesmckinn@gmail.com>
>> > > > > > > > >> > >> wrote:
>> > > > > > > > >> > >> >>
>> > > > > > > > >> > >> >> The process should be well documented at this
>> point but
>> > > > > there are a
>> > > > > > > > >> > >> >> number of steps. Note that you need to add your
>> code
>> > > > > signing key to
>> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to
>> do). I
>> > > > > think it's fine
>> > > > > > > > >> > >> >> to hand off the process to others after the VOTE
>> but it
>> > > > > would be
>> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
>> producing the
>> > > > > source and
>> > > > > > > > >> > >> >> binary artifacts for the vote
>> > > > > > > > >> > >> >>
>> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield
>> <
>> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > SGTM, as well.
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > I should have a little bit of time next week
>> if I can
>> > > > > help as RM but
>> > > > > > > > >> > >> I have
>> > > > > > > > >> > >> >> > a couple of concerns:
>> > > > > > > > >> > >> >> > 1. In the past I've had trouble downloading
>> and
>> > > > > validating
>> > > > > > > > >> > >> releases. I'm a
>> > > > > > > > >> > >> >> > bit worried, that I might have similar
>> problems doing
>> > > > > the necessary
>> > > > > > > > >> > >> uploads.
>> > > > > > > > >> > >> >> > 2. My internet connection will likely be not
>> great, I
>> > > > > don't know if
>> > > > > > > > >> > >> this
>> > > > > > > > >> > >> >> > would make it even less likely to be
>> successful.
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > Does it become problematic if somehow I would
>> have to
>> > > > > abandon the
>> > > > > > > > >> > >> process
>> > > > > > > > >> > >> >> > mid-release? Is there anyone who could serve
>> as a
>> > > > > backup? Are the
>> > > > > > > > >> > >> steps
>> > > > > > > > >> > >> >> > well documented?
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > Thanks,
>> > > > > > > > >> > >> >> > Micah
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal
>> Richardson <
>> > > > > > > > >> > >> neal.p.richardson@gmail.com>
>> > > > > > > > >> > >> >> > wrote:
>> > > > > > > > >> > >> >> >
>> > > > > > > > >> > >> >> > > Sounds good to me.
>> > > > > > > > >> > >> >> > >
>> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
>> volunteers?
>> > > > > > > > >> > >> >> > >
>> > > > > > > > >> > >> >> > > Neal
>> > > > > > > > >> > >> >> > >
>> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney
>> <
>> > > > > wesmckinn@gmail.com>
>> > > > > > > > >> > >> wrote:
>> > > > > > > > >> > >> >> > >
>> > > > > > > > >> > >> >> > > > hi all,
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >> >> > > > It looks like we're drawing close to be
>> able to
>> > > > > make the 0.15.0
>> > > > > > > > >> > >> >> > > > release. I would suggest "pencils down" at
>> the end
>> > > > > of this week
>> > > > > > > > >> > >> and
>> > > > > > > > >> > >> >> > > > see if a release candidate can be produced
>> next
>> > > > > Monday September
>> > > > > > > > >> > >> 23.
>> > > > > > > > >> > >> >> > > > Any thoughts or objections?
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >> >> > > > Thanks,
>> > > > > > > > >> > >> >> > > > Wes
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
>> McKinney <
>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > > > > > > > >> > >> >> > > wrote:
>> > > > > > > > >> > >> >> > > > >
>> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
>> planning to
>> > > > > amend the
>> > > > > > > > >> > >> Format docs
>> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and also
>> update
>> > > > the
>> > > > > C++ library
>> > > > > > > > >> > >> >> > > > >
>> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric
>> Erhardt
>> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
>> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
>> > > > > > > > >> > >> >> > > > branch into master before the 0.15 release,
>> > > > correct?
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
>> changes are
>> > > > > ready to be
>> > > > > > > > >> > >> merged into
>> > > > > > > > >> > >> >> > > > the alignment branch -
>> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > Eric
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > -----Original Message-----
>> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
>> emkornfield@gmail.com>
>> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019
>> 10:24 PM
>> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
>> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
>> niki.lj <
>> > > > > niki.lj@aliyun.com>
>> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0
>> release
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth
>> to help
>> > > > > with some of
>> > > > > > > > >> > >> the
>> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going into
>> the
>> > > > > weekend.
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes
>> McKinney <
>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > > > > > > > >> > >> >> > > > wrote:
>> > > > > > > > >> > >> >> > > > > >
>> > > > > > > > >> > >> >> > > > > > > Hi folks,
>> > > > > > > > >> > >> >> > > > > > >
>> > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging
>> and
>> > > > > integration builds
>> > > > > > > > >> > >> things
>> > > > > > > > >> > >> >> > > > > > > aren't looking too good for being in
>> release
>> > > > > readiness by
>> > > > > > > > >> > >> the end
>> > > > > > > > >> > >> >> > > of
>> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm
>> planning
>> > > > > to be working
>> > > > > > > > >> > >> to close
>> > > > > > > > >> > >> >> > > as
>> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to
>> help with
>> > > > > the ongoing
>> > > > > > > > >> > >> alignment
>> > > > > > > > >> > >> >> > > > fixes.
>> > > > > > > > >> > >> >> > > > > > >
>> > > > > > > > >> > >> >> > > > > > > Wes
>> > > > > > > > >> > >> >> > > > > > >
>> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
>> > > > Kornfield
>> > > > > <
>> > > > > > > > >> > >> >> > > emkornfield@gmail.com
>> > > > > > > > >> > >> >> > > > >
>> > > > > > > > >> > >> >> > > > > > > wrote:
>> > > > > > > > >> > >> >> > > > > > >
>> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
>> dashboard of
>> > > > > the current
>> > > > > > > > >> > >> issues:
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >>
>> > > > >
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
>> > > > > > > > >> > >> >> > > >
>> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> > > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
>> > > > > 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >>
>> > > > >
>> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >>
>> > > > >
>> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
>> > > > McKinney <
>> > > > > > > > >> > >> wesmckinn@gmail.com>
>> > > > > > > > >> > >> >> > > > wrote:
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > > > > >>> hi all,
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going
>> to be in
>> > > > a
>> > > > > position to
>> > > > > > > > >> > >> release
>> > > > > > > > >> > >> >> > > at
>> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope
>> that
>> > > > one
>> > > > > more week of
>> > > > > > > > >> > >> work (or
>> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
>> there.
>> > > > Aside
>> > > > > from merging
>> > > > > > > > >> > >> the
>> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make
>> sure
>> > > > > that our
>> > > > > > > > >> > >> packaging jobs
>> > > > > > > > >> > >> >> > > > > > >>> required for the release candidate
>> are all
>> > > > > working.
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from
>> the
>> > > > > 0.15.0 backlog
>> > > > > > > > >> > >> that they
>> > > > > > > > >> > >> >> > > > don't
>> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end of
>> next week
>> > > > > that would
>> > > > > > > > >> > >> help focus
>> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78
>> issues in
>> > > > > 0.15.0 still).
>> > > > > > > > >> > >> I am
>> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
>> features
>> > > > > related to
>> > > > > > > > >> > >> dictionaries
>> > > > > > > > >> > >> >> > > > while
>> > > > > > > > >> > >> >> > > > > > >>> the release window is still open.
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > > > > >>> - Wes
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
>> > > > > McKinney <
>> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > hi,
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to release
>> the
>> > > > week
>> > > > > of September
>> > > > > > > > >> > >> 9, so
>> > > > > > > > >> > >> >> > > > > > >>> > development work should be
>> completed by
>> > > > > end of next
>> > > > > > > > >> > >> week.
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
>> > > > protocol
>> > > > > alignment
>> > > > > > > > >> > >> changes for
>> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days
>> -- I
>> > > > think
>> > > > > that getting
>> > > > > > > > >> > >> the
>> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main
>> barrier
>> > > > > to releasing.
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > Thanks
>> > > > > > > > >> > >> >> > > > > > >>> > Wes
>> > > > > > > > >> > >> >> > > > > > >>> >
>> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM
>> Ji Liu
>> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I
>> can think
>> > > > > of several
>> > > > > > > > >> > >> bugs that
>> > > > > > > > >> > >> >> > > > need
>> > > > > > > > >> > >> >> > > > > > >>> > > to
>> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
>> entries are
>> > > > > required in
>> > > > > > > > >> > >> IPC streams
>> > > > > > > > >> > >> >> > > > > > >>> > > even
>> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
>> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review now,
>> however
>> > > > > through this
>> > > > > > > > >> > >> PR we find
>> > > > > > > > >> > >> >> > > > > > >>> > > that
>> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading
>> and
>> > > > > writing
>> > > > > > > > >> > >> dictionaries in IPC
>> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2]
>> since
>> > > > it
>> > > > > assumes all
>> > > > > > > > >> > >> >> > > > dictionaries
>> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see
>> details in
>> > > > > PR comments,
>> > > > > > > > >> > >> and this
>> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version
>> 0.15).
>> > > > > @Micah Kornfield
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit
>> ints as
>> > > > > strings in
>> > > > > > > > >> > >> integration
>> > > > > > > > >> > >> >> > > > test
>> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
>> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already checked
>> in,
>> > > > other
>> > > > > > > > >> > >> implementations
>> > > > > > > > >> > >> >> > > seems
>> > > > > > > > >> > >> >> > > > not.
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
>> > > > > JdbcAdapter[4]
>> > > > > > > > >> > >> Caused by
>> > > > > > > > >> > >> >> > > trying
>> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
>> contiguous
>> > > > > batch, fixed
>> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
>> iteratively
>> > > > > reading in
>> > > > > > > > >> > >> >> > > ARROW-6219[5].
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
>> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > [1]
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
>> > > > > > > > >> > >> >> > > >
>> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
>> > > > > > > > >> > >> >> > > >
>> %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > > > > > > > >> > >> >> > > >
>> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
>> 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > > > > > > > >> > >> >> > > >
>> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
>> 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >>
>> > > > >
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
>> > > > > > > > >> > >> >> > > >
>> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
>> > > > > > > > >> > >> >> > > >
>> %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >>
>> > > > >
>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >>
>> > > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > >
>> > > > > ----------------------------------------------------------------
>> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
>> > > > > wesmckinn@gmail.com> Send
>> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
>> To:dev <
>> > > > > > > > >> > >> dev@arrow.apache.org>
>> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
>> > > > release
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
>> organizing
>> > > > > the 0.15.0
>> > > > > > > > >> > >> backlog some
>> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to
>> help
>> > > > with
>> > > > > grooming
>> > > > > > > > >> > >> >> > > (particularly
>> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
>> C++/Python
>> > > > > where I'm
>> > > > > > > > >> > >> focusing) that
>> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have
>> been
>> > > > > almost 500 JIRA
>> > > > > > > > >> > >> issues
>> > > > > > > > >> > >> >> > > opened
>> > > > > > > > >> > >> >> > > > > > >>> > > since the
>> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should
>> make sure
>> > > > > to check
>> > > > > > > > >> > >> whether
>> > > > > > > > >> > >> >> > > there's
>> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
>> serious bugs
>> > > > > that we should
>> > > > > > > > >> > >> try to
>> > > > > > > > >> > >> >> > > fix
>> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23
>> PM Wes
>> > > > > McKinney
>> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
>> > > > > > > > >> > >> >> > > > > > >>> wrote:
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in
>> 0.14.1
>> > > > > seems to be
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
>> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
>> > > > 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could
>> be the
>> > > > > Windows
>> > > > > > > > >> > >> changes in
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%
>> 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
>> > > > > > > > >> > >> >> > > > > > >>>
>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
>> > > > > volunteer would look
>> > > > > > > > >> > >> into what
>> > > > > > > > >> > >> >> > > > > > >>> > > > was
>> > > > > > > > >> > >> >> > > > > > >>> wrong
>> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on
>> Windows.
>> > > > > Otherwise
>> > > > > > > > >> > >> 0.15.0 Windows
>> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found
>> at
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>> > > > > > > > >> > >> 40microsoft.com
>> > > > > > > > >> > >> >> > > > %7Ccbea
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > >
>> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>> > > > > > > > >> > >> >> > > > > > >>> > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28
>> PM
>> > > > > Antoine Pitrou <
>> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
>> 11:17:07 -0700
>> > > > > Micah
>> > > > > > > > >> > >> Kornfield
>> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com>
>> wrote:
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could
>> have
>> > > > > 32-bit array
>> > > > > > > > >> > >> lengths and
>> > > > > > > > >> > >> >> > > > > > >>> variable-length
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
>> offsets if
>> > > > we
>> > > > > wanted (we
>> > > > > > > > >> > >> just
>> > > > > > > > >> > >> >> > > > wouldn't
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
>> > > > > > > > >> > >> >> > > > > > >>> able to
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with
>> more
>> > > > > than INT32_MAX
>> > > > > > > > >> > >> elements).
>> > > > > > > > >> > >> >> > > > > > >>> > > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we
>> could do
>> > > > > this in C++
>> > > > > > > > >> > >> but we
>> > > > > > > > >> > >> >> > > > don't.
>> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
>> > > > "Large"
>> > > > > types if we
>> > > > > > > > >> > >> did.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice
>> as much
>> > > > > space as 32-bit
>> > > > > > > > >> > >> >> > > offsets,
>> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
>> > > > > > > > >> > >> >> > > > > > >>> you're
>> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish
>> lists or
>> > > > > strings,
>> > > > > > > > >> > >> 32-bit
>> > > > > > > > >> > >> >> > > offsets
>> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even
>> with
>> > > > > 64-bit array
>> > > > > > > > >> > >> lengths from
>> > > > > > > > >> > >> >> > > > the
>> > > > > > > > >> > >> >> > > > > > >>> > > > > start
>> > > > > > > > >> > >> >> > > > > > >>> it would
>> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to
>> have types
>> > > > > with 32-bit
>> > > > > > > > >> > >> offsets.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited
>> address
>> > > > > space in Java
>> > > > > > > > >> > >> and
>> > > > > > > > >> > >> >> > > calling
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
>> > > > > > > > >> > >> >> > > > > > >>> reference
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
>> suboptimal.
>> > > > > If a consumer
>> > > > > > > > >> > >> uses a
>> > > > > > > > >> > >> >> > > > "Large"
>> > > > > > > > >> > >> >> > > > > > >>> type
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because
>> they
>> > > > > need the ability
>> > > > > > > > >> > >> to store
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
>> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
>> column,
>> > > > > otherwise it is
>> > > > > > > > >> > >> just
>> > > > > > > > >> > >> >> > > wasting
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
>> > > > > > > > >> > >> >> > > > > > >>> [1].
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the
>> individual
>> > > > > elements
>> > > > > > > > >> > >> (lists or
>> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
>> > > > > > > > >> > >> >> > > > > > >>> are
>> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is
>> wasted in
>> > > > > proportion,
>> > > > > > > > >> > >> so it may
>> > > > > > > > >> > >> >> > > be
>> > > > > > > > >> > >> >> > > > > > >>> simpler in
>> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
>> create a
>> > > > > "Large" type
>> > > > > > > > >> > >> array.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
>> theoretically
>> > > > there
>> > > > > might be some
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
>> > > > > > > > >> > >> >> > > > > > >>> benefits on
>> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to
>> using
>> > > > the
>> > > > > native word
>> > > > > > > > >> > >> sizes.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
>> > > > > architectures don't do
>> > > > > > > > >> > >> that, as
>> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
>> > > > > > > > >> > >> >> > > > > > >>> is an
>> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer
>> size even
>> > > > > in
>> > > > > > > > >> > >> high-performance
>> > > > > > > > >> > >> >> > > > code.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > > > >
>> > > > > > > > >> > >> >> > > > > > >>> > >
>> > > > > > > > >> > >> >> > > > > > >>>
>> > > > > > > > >> > >> >> > > > > > >>
>> > > > > > > > >> > >> >> > > >
>> > > > > > > > >> > >> >> > >
>> > > > > > > > >> > >>
>> > > > > > > > >> > >
>> > > > >
>> > > >
>>
>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
OK, I'll start the process today. I'll send up e-mail updates as I make
progress.
On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <we...@gmail.com> wrote:
> Yes, all systems go as far as I'm concerned.
>
> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
> <ne...@gmail.com> wrote:
> >
> > Andy's DataFusion issue and Wes's Parquet one have both been merged,
> > and it looks like the LICENSE issue is being resolved as I type. So
> > are we good to go now?
> >
> > Neal
> >
> >
> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > I found a last minute issue with DataFusion (Rust) and would
> appreciate it
> > > if we could merge ARROW-6086 (PR is
> > > https://github.com/apache/arrow/pull/5494) before cutting the RC.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <emkornfield@gmail.com
> >
> > > wrote:
> > >
> > > > OK, I'm going to postpone cutting a release until tomorrow (hoping
> we can
> > > > issues resolved by then).. I'll also try to review the third-party
> > > > additions since 14.x.
> > > >
> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > >
> > > > > I found a licensing issue
> > > > >
> > > > > https://issues.apache.org/jira/browse/ARROW-6679
> > > > >
> > > > > It might be worth examining third party code added to the project
> > > > > since 0.14.x to make sure there are no other such issues.
> > > > >
> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > I have diagnosed the problem (Thrift "string" data must be UTF-8,
> > > > > > cannot be arbitrary binary) and am working on a patch right now
> > > > > >
> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > I just opened
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678
> > > > > > >
> > > > > > > Please don't cut an RC until I have an opportunity to diagnose
> this,
> > > > > > > will report back.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > I'm investigating a possible Parquet-related compatibility
> bug
> > > > that I
> > > > > > > > encountered through some routine testing / benchmarking. I'll
> > > > report
> > > > > > > > back once I figure out what is going on (if anything)
> > > > > > > >
> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> > > > > emkornfield@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
> you can
> > > > > get it
> > > > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > > > >
> > > > > > > > > That won't be an option for me this week (it seems like I
> would
> > > > > need to meet one face-to-face). I'll try to get the GPG checked
> in and
> > > > the
> > > > > rest of the pre-requisites done tomorrow (Monday) to hopefully
> start the
> > > > > release on Tuesday (hopefully we can solve the last
> blocker/integration
> > > > > tests by then).
> > > > > > > > >
> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e.
> you can
> > > > > get it
> > > > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > > > >>
> > > > > > > > >> Speaking of the release, there are at least 2 code
> changes I
> > > > still
> > > > > > > > >> want to get in
> > > > > > > > >>
> > > > > > > > >> ARROW-5717
> > > > > > > > >> ARROW-6353
> > > > > > > > >>
> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once the
> build
> > > > is
> > > > > green.
> > > > > > > > >>
> > > > > > > > >> There are a couple of Rust patches still marked for 0.15.
> The
> > > > rest
> > > > > > > > >> seems to be documentation and a couple of integration test
> > > > > failures we
> > > > > > > > >> should see about fixing in time.
> > > > > > > > >>
> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> > > > > emkornfield@gmail.com> wrote:
> > > > > > > > >> >
> > > > > > > > >> > Thanks Krisztián and Wes,
> > > > > > > > >> > I've gone ahead and started registering myself on all
> the
> > > > > packaging sites.
> > > > > > > > >> >
> > > > > > > > >> > Is there any review process when adding my GPG key to
> the SVN
> > > > > file? [1]
> > > > > > > > >> > doesn't seem to mention explicitly.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Micah
> > > > > > > > >> >
> > > > > > > > >> > [1]
> https://www.apache.org/dev/version-control.html#https-svn
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> > > > > szucs.krisztian@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> > > > > wesmckinn@gmail.com> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> > > > > emkornfield@gmail.com>
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> The process should be well documented at this
> point but
> > > > > there are a
> > > > > > > > >> > >> >> number of steps.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the
> release?
> > > > Are
> > > > > there
> > > > > > > > >> > >> instructions for the adding the code signing Key to
> SVN?
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > I will make a go of it. i will try to mitigate any
> > > > > internet issues by
> > > > > > > > >> > >> doing the process for a cloud instance (I assume
> that isn't
> > > > > a problem?).
> > > > > > > > >> > >> >
> > > > > > > > >> > >>
> > > > > > > > >> > >> Setting up a new cloud environment suitable for
> producing
> > > > an
> > > > > RC may be
> > > > > > > > >> > >> time consuming, but you are welcome to try.
> Krisztian --
> > > > are
> > > > > you
> > > > > > > > >> > >> available next week to help Micah and potentially
> take over
> > > > > producing
> > > > > > > > >> > >> the RC if there are issues?
> > > > > > > > >> > >>
> > > > > > > > >> > > Sure, I'll be available next week. We can also grant
> access
> > > > to
> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because
> configuring
> > > > all
> > > > > > > > >> > > the CI backends can be time consuming.
> > > > > > > > >> > >
> > > > > > > > >> > >>
> > > > > > > > >> > >> > Thanks,
> > > > > > > > >> > >> > Micah
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > [1]
> > > > > > > > >> > >>
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> > > > > wesmckinn@gmail.com>
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> The process should be well documented at this
> point but
> > > > > there are a
> > > > > > > > >> > >> >> number of steps. Note that you need to add your
> code
> > > > > signing key to
> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to
> do). I
> > > > > think it's fine
> > > > > > > > >> > >> >> to hand off the process to others after the VOTE
> but it
> > > > > would be
> > > > > > > > >> > >> >> tricky to have multiple RMs involved with
> producing the
> > > > > source and
> > > > > > > > >> > >> >> binary artifacts for the vote
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > > > > > > >> > >> emkornfield@gmail.com> wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > SGTM, as well.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > I should have a little bit of time next week if
> I can
> > > > > help as RM but
> > > > > > > > >> > >> I have
> > > > > > > > >> > >> >> > a couple of concerns:
> > > > > > > > >> > >> >> > 1. In the past I've had trouble downloading and
> > > > > validating
> > > > > > > > >> > >> releases. I'm a
> > > > > > > > >> > >> >> > bit worried, that I might have similar problems
> doing
> > > > > the necessary
> > > > > > > > >> > >> uploads.
> > > > > > > > >> > >> >> > 2. My internet connection will likely be not
> great, I
> > > > > don't know if
> > > > > > > > >> > >> this
> > > > > > > > >> > >> >> > would make it even less likely to be successful.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > Does it become problematic if somehow I would
> have to
> > > > > abandon the
> > > > > > > > >> > >> process
> > > > > > > > >> > >> >> > mid-release? Is there anyone who could serve
> as a
> > > > > backup? Are the
> > > > > > > > >> > >> steps
> > > > > > > > >> > >> >> > well documented?
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > Thanks,
> > > > > > > > >> > >> >> > Micah
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson
> <
> > > > > > > > >> > >> neal.p.richardson@gmail.com>
> > > > > > > > >> > >> >> > wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > > Sounds good to me.
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any
> volunteers?
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Neal
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> > > > > wesmckinn@gmail.com>
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > > hi all,
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >> >> > > > It looks like we're drawing close to be
> able to
> > > > > make the 0.15.0
> > > > > > > > >> > >> >> > > > release. I would suggest "pencils down" at
> the end
> > > > > of this week
> > > > > > > > >> > >> and
> > > > > > > > >> > >> >> > > > see if a release candidate can be produced
> next
> > > > > Monday September
> > > > > > > > >> > >> 23.
> > > > > > > > >> > >> >> > > > Any thoughts or objections?
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >> >> > > > Thanks,
> > > > > > > > >> > >> >> > > > Wes
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes
> McKinney <
> > > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > > >> > >> >> > > wrote:
> > > > > > > > >> > >> >> > > > >
> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm
> planning to
> > > > > amend the
> > > > > > > > >> > >> Format docs
> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and also
> update
> > > > the
> > > > > C++ library
> > > > > > > > >> > >> >> > > > >
> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric
> Erhardt
> > > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> > > > > > > > >> > >> >> > > > branch into master before the 0.15 release,
> > > > correct?
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment
> changes are
> > > > > ready to be
> > > > > > > > >> > >> merged into
> > > > > > > > >> > >> >> > > > the alignment branch -
> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > Eric
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > -----Original Message-----
> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield <
> emkornfield@gmail.com>
> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24
> PM
> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>;
> niki.lj <
> > > > > niki.lj@aliyun.com>
> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth
> to help
> > > > > with some of
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going into
> the
> > > > > weekend.
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes
> McKinney <
> > > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > > >> > >> >> > > > wrote:
> > > > > > > > >> > >> >> > > > > >
> > > > > > > > >> > >> >> > > > > > > Hi folks,
> > > > > > > > >> > >> >> > > > > > >
> > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging
> and
> > > > > integration builds
> > > > > > > > >> > >> things
> > > > > > > > >> > >> >> > > > > > > aren't looking too good for being in
> release
> > > > > readiness by
> > > > > > > > >> > >> the end
> > > > > > > > >> > >> >> > > of
> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm
> planning
> > > > > to be working
> > > > > > > > >> > >> to close
> > > > > > > > >> > >> >> > > as
> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to help
> with
> > > > > the ongoing
> > > > > > > > >> > >> alignment
> > > > > > > > >> > >> >> > > > fixes.
> > > > > > > > >> > >> >> > > > > > >
> > > > > > > > >> > >> >> > > > > > > Wes
> > > > > > > > >> > >> >> > > > > > >
> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
> > > > Kornfield
> > > > > <
> > > > > > > > >> > >> >> > > emkornfield@gmail.com
> > > > > > > > >> > >> >> > > > >
> > > > > > > > >> > >> >> > > > > > > wrote:
> > > > > > > > >> > >> >> > > > > > >
> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a
> dashboard of
> > > > > the current
> > > > > > > > >> > >> issues:
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >>
> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > > > > >> > >> >> > > > > > >> ki.apache.org
> > > > > > > > >> > >> >> > > >
> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
> > > > > 40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >>
> > > > >
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >>
> > > > >
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
> > > > McKinney <
> > > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > > >> > >> >> > > > wrote:
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > > > > >>> hi all,
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to
> be in
> > > > a
> > > > > position to
> > > > > > > > >> > >> release
> > > > > > > > >> > >> >> > > at
> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope
> that
> > > > one
> > > > > more week of
> > > > > > > > >> > >> work (or
> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us
> there.
> > > > Aside
> > > > > from merging
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make
> sure
> > > > > that our
> > > > > > > > >> > >> packaging jobs
> > > > > > > > >> > >> >> > > > > > >>> required for the release candidate
> are all
> > > > > working.
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from
> the
> > > > > 0.15.0 backlog
> > > > > > > > >> > >> that they
> > > > > > > > >> > >> >> > > > don't
> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end of
> next week
> > > > > that would
> > > > > > > > >> > >> help focus
> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78
> issues in
> > > > > 0.15.0 still).
> > > > > > > > >> > >> I am
> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small
> features
> > > > > related to
> > > > > > > > >> > >> dictionaries
> > > > > > > > >> > >> >> > > > while
> > > > > > > > >> > >> >> > > > > > >>> the release window is still open.
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > > > > >>> - Wes
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
> > > > > McKinney <
> > > > > > > > >> > >> >> > > wesmckinn@gmail.com>
> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > hi,
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to release
> the
> > > > week
> > > > > of September
> > > > > > > > >> > >> 9, so
> > > > > > > > >> > >> >> > > > > > >>> > development work should be
> completed by
> > > > > end of next
> > > > > > > > >> > >> week.
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
> > > > protocol
> > > > > alignment
> > > > > > > > >> > >> changes for
> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days --
> I
> > > > think
> > > > > that getting
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main
> barrier
> > > > > to releasing.
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > Thanks
> > > > > > > > >> > >> >> > > > > > >>> > Wes
> > > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM
> Ji Liu
> > > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I
> can think
> > > > > of several
> > > > > > > > >> > >> bugs that
> > > > > > > > >> > >> >> > > > need
> > > > > > > > >> > >> >> > > > > > >>> > > to
> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary
> entries are
> > > > > required in
> > > > > > > > >> > >> IPC streams
> > > > > > > > >> > >> >> > > > > > >>> > > even
> > > > > > > > >> > >> >> > > > > > >>> when empty[1]
> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review now,
> however
> > > > > through this
> > > > > > > > >> > >> PR we find
> > > > > > > > >> > >> >> > > > > > >>> > > that
> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading
> and
> > > > > writing
> > > > > > > > >> > >> dictionaries in IPC
> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2]
> since
> > > > it
> > > > > assumes all
> > > > > > > > >> > >> >> > > > dictionaries
> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see
> details in
> > > > > PR comments,
> > > > > > > > >> > >> and this
> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version
> 0.15).
> > > > > @Micah Kornfield
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit
> ints as
> > > > > strings in
> > > > > > > > >> > >> integration
> > > > > > > > >> > >> >> > > > test
> > > > > > > > >> > >> >> > > > > > >>> JSON files[3]
> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already checked
> in,
> > > > other
> > > > > > > > >> > >> implementations
> > > > > > > > >> > >> >> > > seems
> > > > > > > > >> > >> >> > > > not.
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
> > > > > JdbcAdapter[4]
> > > > > > > > >> > >> Caused by
> > > > > > > > >> > >> >> > > trying
> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one
> contiguous
> > > > > batch, fixed
> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for
> iteratively
> > > > > reading in
> > > > > > > > >> > >> >> > > ARROW-6219[5].
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > Thanks,
> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > [1]
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > > > > > > >> > >> >> > > >
> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%
> 40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%
> 40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >>
> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org
> > > > > > > > >> > >> >> > > >
> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >>
> > > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >>
> > > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > >
> > > > > ----------------------------------------------------------------
> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> > > > > wesmckinn@gmail.com> Send
> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03
> To:dev <
> > > > > > > > >> > >> dev@arrow.apache.org>
> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
> > > > release
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on
> organizing
> > > > > the 0.15.0
> > > > > > > > >> > >> backlog some
> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to
> help
> > > > with
> > > > > grooming
> > > > > > > > >> > >> >> > > (particularly
> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than
> C++/Python
> > > > > where I'm
> > > > > > > > >> > >> focusing) that
> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have
> been
> > > > > almost 500 JIRA
> > > > > > > > >> > >> issues
> > > > > > > > >> > >> >> > > opened
> > > > > > > > >> > >> >> > > > > > >>> > > since the
> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should
> make sure
> > > > > to check
> > > > > > > > >> > >> whether
> > > > > > > > >> > >> >> > > there's
> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other
> serious bugs
> > > > > that we should
> > > > > > > > >> > >> try to
> > > > > > > > >> > >> >> > > fix
> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM
> Wes
> > > > > McKinney
> > > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in
> 0.14.1
> > > > > seems to be
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> > > > 40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could
> be the
> > > > > Windows
> > > > > > > > >> > >> changes in
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > > > > > > >> > >> >> > > > > > >>>
> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
> > > > > volunteer would look
> > > > > > > > >> > >> into what
> > > > > > > > >> > >> >> > > > > > >>> > > > was
> > > > > > > > >> > >> >> > > > > > >>> wrong
> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on
> Windows.
> > > > > Otherwise
> > > > > > > > >> > >> 0.15.0 Windows
> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > > > > > > >> > >> 40microsoft.com
> > > > > > > > >> > >> >> > > > %7Ccbea
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > >
> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28
> PM
> > > > > Antoine Pitrou <
> > > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019
> 11:17:07 -0700
> > > > > Micah
> > > > > > > > >> > >> Kornfield
> > > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com>
> wrote:
> > > > > > > > >> > >> >> > > > > > >>> > > > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could
> have
> > > > > 32-bit array
> > > > > > > > >> > >> lengths and
> > > > > > > > >> > >> >> > > > > > >>> variable-length
> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit
> offsets if
> > > > we
> > > > > wanted (we
> > > > > > > > >> > >> just
> > > > > > > > >> > >> >> > > > wouldn't
> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be
> > > > > > > > >> > >> >> > > > > > >>> able to
> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with
> more
> > > > > than INT32_MAX
> > > > > > > > >> > >> elements).
> > > > > > > > >> > >> >> > > > > > >>> > > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we
> could do
> > > > > this in C++
> > > > > > > > >> > >> but we
> > > > > > > > >> > >> >> > > > don't.
> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we
> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
> > > > "Large"
> > > > > types if we
> > > > > > > > >> > >> did.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice
> as much
> > > > > space as 32-bit
> > > > > > > > >> > >> >> > > offsets,
> > > > > > > > >> > >> >> > > > > > >>> > > > > so if
> > > > > > > > >> > >> >> > > > > > >>> you're
> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish
> lists or
> > > > > strings,
> > > > > > > > >> > >> 32-bit
> > > > > > > > >> > >> >> > > offsets
> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even
> with
> > > > > 64-bit array
> > > > > > > > >> > >> lengths from
> > > > > > > > >> > >> >> > > > the
> > > > > > > > >> > >> >> > > > > > >>> > > > > start
> > > > > > > > >> > >> >> > > > > > >>> it would
> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have
> types
> > > > > with 32-bit
> > > > > > > > >> > >> offsets.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited
> address
> > > > > space in Java
> > > > > > > > >> > >> and
> > > > > > > > >> > >> >> > > calling
> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a
> > > > > > > > >> > >> >> > > > > > >>> reference
> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems
> suboptimal.
> > > > > If a consumer
> > > > > > > > >> > >> uses a
> > > > > > > > >> > >> >> > > > "Large"
> > > > > > > > >> > >> >> > > > > > >>> type
> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because
> they
> > > > > need the ability
> > > > > > > > >> > >> to store
> > > > > > > > >> > >> >> > > > > > >>> > > > > > more
> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a
> column,
> > > > > otherwise it is
> > > > > > > > >> > >> just
> > > > > > > > >> > >> >> > > wasting
> > > > > > > > >> > >> >> > > > > > >>> > > > > > space
> > > > > > > > >> > >> >> > > > > > >>> [1].
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the
> individual
> > > > > elements
> > > > > > > > >> > >> (lists or
> > > > > > > > >> > >> >> > > > > > >>> > > > > strings)
> > > > > > > > >> > >> >> > > > > > >>> are
> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is
> wasted in
> > > > > proportion,
> > > > > > > > >> > >> so it may
> > > > > > > > >> > >> >> > > be
> > > > > > > > >> > >> >> > > > > > >>> simpler in
> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always
> create a
> > > > > "Large" type
> > > > > > > > >> > >> array.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose
> theoretically
> > > > there
> > > > > might be some
> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance
> > > > > > > > >> > >> >> > > > > > >>> benefits on
> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to
> using
> > > > the
> > > > > native word
> > > > > > > > >> > >> sizes.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
> > > > > architectures don't do
> > > > > > > > >> > >> that, as
> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > > > > > > >> > >> >> > > > > > >>> is an
> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer
> size even
> > > > > in
> > > > > > > > >> > >> high-performance
> > > > > > > > >> > >> >> > > > code.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > > >> > >> >> > > > > > >>>
> > > > > > > > >> > >> >> > > > > > >>
> > > > > > > > >> > >> >> > > >
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >>
> > > > > > > > >> > >
> > > > >
> > > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
Yes, all systems go as far as I'm concerned.
On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson
<ne...@gmail.com> wrote:
>
> Andy's DataFusion issue and Wes's Parquet one have both been merged,
> and it looks like the LICENSE issue is being resolved as I type. So
> are we good to go now?
>
> Neal
>
>
> On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com> wrote:
> >
> > I found a last minute issue with DataFusion (Rust) and would appreciate it
> > if we could merge ARROW-6086 (PR is
> > https://github.com/apache/arrow/pull/5494) before cutting the RC.
> >
> > Thanks,
> >
> > Andy.
> >
> >
> > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > OK, I'm going to postpone cutting a release until tomorrow (hoping we can
> > > issues resolved by then).. I'll also try to review the third-party
> > > additions since 14.x.
> > >
> > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > I found a licensing issue
> > > >
> > > > https://issues.apache.org/jira/browse/ARROW-6679
> > > >
> > > > It might be worth examining third party code added to the project
> > > > since 0.14.x to make sure there are no other such issues.
> > > >
> > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > I have diagnosed the problem (Thrift "string" data must be UTF-8,
> > > > > cannot be arbitrary binary) and am working on a patch right now
> > > > >
> > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > I just opened
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/ARROW-6678
> > > > > >
> > > > > > Please don't cut an RC until I have an opportunity to diagnose this,
> > > > > > will report back.
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > I'm investigating a possible Parquet-related compatibility bug
> > > that I
> > > > > > > encountered through some routine testing / benchmarking. I'll
> > > report
> > > > > > > back once I figure out what is going on (if anything)
> > > > > > >
> > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> > > > emkornfield@gmail.com> wrote:
> > > > > > > >>
> > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > > > get it
> > > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > > >
> > > > > > > > That won't be an option for me this week (it seems like I would
> > > > need to meet one face-to-face). I'll try to get the GPG checked in and
> > > the
> > > > rest of the pre-requisites done tomorrow (Monday) to hopefully start the
> > > > release on Tuesday (hopefully we can solve the last blocker/integration
> > > > tests by then).
> > > > > > > >
> > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > wrote:
> > > > > > > >>
> > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > > > get it
> > > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > > >>
> > > > > > > >> Speaking of the release, there are at least 2 code changes I
> > > still
> > > > > > > >> want to get in
> > > > > > > >>
> > > > > > > >> ARROW-5717
> > > > > > > >> ARROW-6353
> > > > > > > >>
> > > > > > > >> I just pushed updates to ARROW-5717, will merge once the build
> > > is
> > > > green.
> > > > > > > >>
> > > > > > > >> There are a couple of Rust patches still marked for 0.15. The
> > > rest
> > > > > > > >> seems to be documentation and a couple of integration test
> > > > failures we
> > > > > > > >> should see about fixing in time.
> > > > > > > >>
> > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> > > > emkornfield@gmail.com> wrote:
> > > > > > > >> >
> > > > > > > >> > Thanks Krisztián and Wes,
> > > > > > > >> > I've gone ahead and started registering myself on all the
> > > > packaging sites.
> > > > > > > >> >
> > > > > > > >> > Is there any review process when adding my GPG key to the SVN
> > > > file? [1]
> > > > > > > >> > doesn't seem to mention explicitly.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Micah
> > > > > > > >> >
> > > > > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > > > > > > >> >
> > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> > > > szucs.krisztian@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> > > > wesmckinn@gmail.com> wrote:
> > > > > > > >> > >
> > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> > > > emkornfield@gmail.com>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> The process should be well documented at this point but
> > > > there are a
> > > > > > > >> > >> >> number of steps.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Is [1] the up-to-date documentation for the release?
> > > Are
> > > > there
> > > > > > > >> > >> instructions for the adding the code signing Key to SVN?
> > > > > > > >> > >> >
> > > > > > > >> > >> > I will make a go of it. i will try to mitigate any
> > > > internet issues by
> > > > > > > >> > >> doing the process for a cloud instance (I assume that isn't
> > > > a problem?).
> > > > > > > >> > >> >
> > > > > > > >> > >>
> > > > > > > >> > >> Setting up a new cloud environment suitable for producing
> > > an
> > > > RC may be
> > > > > > > >> > >> time consuming, but you are welcome to try. Krisztian --
> > > are
> > > > you
> > > > > > > >> > >> available next week to help Micah and potentially take over
> > > > producing
> > > > > > > >> > >> the RC if there are issues?
> > > > > > > >> > >>
> > > > > > > >> > > Sure, I'll be available next week. We can also grant access
> > > to
> > > > > > > >> > > https://github.com/ursa-labs/crossbow because configuring
> > > all
> > > > > > > >> > > the CI backends can be time consuming.
> > > > > > > >> > >
> > > > > > > >> > >>
> > > > > > > >> > >> > Thanks,
> > > > > > > >> > >> > Micah
> > > > > > > >> > >> >
> > > > > > > >> > >> > [1]
> > > > > > > >> > >>
> > > >
> > > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> The process should be well documented at this point but
> > > > there are a
> > > > > > > >> > >> >> number of steps. Note that you need to add your code
> > > > signing key to
> > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I
> > > > think it's fine
> > > > > > > >> > >> >> to hand off the process to others after the VOTE but it
> > > > would be
> > > > > > > >> > >> >> tricky to have multiple RMs involved with producing the
> > > > source and
> > > > > > > >> > >> >> binary artifacts for the vote
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > > > > > >> > >> emkornfield@gmail.com> wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > SGTM, as well.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I should have a little bit of time next week if I can
> > > > help as RM but
> > > > > > > >> > >> I have
> > > > > > > >> > >> >> > a couple of concerns:
> > > > > > > >> > >> >> > 1. In the past I've had trouble downloading and
> > > > validating
> > > > > > > >> > >> releases. I'm a
> > > > > > > >> > >> >> > bit worried, that I might have similar problems doing
> > > > the necessary
> > > > > > > >> > >> uploads.
> > > > > > > >> > >> >> > 2. My internet connection will likely be not great, I
> > > > don't know if
> > > > > > > >> > >> this
> > > > > > > >> > >> >> > would make it even less likely to be successful.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Does it become problematic if somehow I would have to
> > > > abandon the
> > > > > > > >> > >> process
> > > > > > > >> > >> >> > mid-release? Is there anyone who could serve as a
> > > > backup? Are the
> > > > > > > >> > >> steps
> > > > > > > >> > >> >> > well documented?
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Thanks,
> > > > > > > >> > >> >> > Micah
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > > > > > > >> > >> neal.p.richardson@gmail.com>
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > Sounds good to me.
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Neal
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > > hi all,
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > It looks like we're drawing close to be able to
> > > > make the 0.15.0
> > > > > > > >> > >> >> > > > release. I would suggest "pencils down" at the end
> > > > of this week
> > > > > > > >> > >> and
> > > > > > > >> > >> >> > > > see if a release candidate can be produced next
> > > > Monday September
> > > > > > > >> > >> 23.
> > > > > > > >> > >> >> > > > Any thoughts or objections?
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Thanks,
> > > > > > > >> > >> >> > > > Wes
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > >> > >> >> > > wrote:
> > > > > > > >> > >> >> > > > >
> > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to
> > > > amend the
> > > > > > > >> > >> Format docs
> > > > > > > >> > >> >> > > > > today regarding the EOS issue and also update
> > > the
> > > > C++ library
> > > > > > > >> > >> >> > > > >
> > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> > > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> > > > > > > >> > >> >> > > > branch into master before the 0.15 release,
> > > correct?
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are
> > > > ready to be
> > > > > > > >> > >> merged into
> > > > > > > >> > >> >> > > > the alignment branch -
> > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > Eric
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > -----Original Message-----
> > > > > > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> > > > niki.lj@aliyun.com>
> > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > I should have a little more bandwidth to help
> > > > with some of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > > > packaging starting tomorrow and going into the
> > > > weekend.
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > >> > >> >> > > > wrote:
> > > > > > > >> > >> >> > > > > >
> > > > > > > >> > >> >> > > > > > > Hi folks,
> > > > > > > >> > >> >> > > > > > >
> > > > > > > >> > >> >> > > > > > > With the state of nightly packaging and
> > > > integration builds
> > > > > > > >> > >> things
> > > > > > > >> > >> >> > > > > > > aren't looking too good for being in release
> > > > readiness by
> > > > > > > >> > >> the end
> > > > > > > >> > >> >> > > of
> > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning
> > > > to be working
> > > > > > > >> > >> to close
> > > > > > > >> > >> >> > > as
> > > > > > > >> > >> >> > > > > > > many issues as I can and also to help with
> > > > the ongoing
> > > > > > > >> > >> alignment
> > > > > > > >> > >> >> > > > fixes.
> > > > > > > >> > >> >> > > > > > >
> > > > > > > >> > >> >> > > > > > > Wes
> > > > > > > >> > >> >> > > > > > >
> > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
> > > Kornfield
> > > > <
> > > > > > > >> > >> >> > > emkornfield@gmail.com
> > > > > > > >> > >> >> > > > >
> > > > > > > >> > >> >> > > > > > > wrote:
> > > > > > > >> > >> >> > > > > > >
> > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of
> > > > the current
> > > > > > > >> > >> issues:
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >>
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > > > >> > >> >> > > > > > >> ki.apache.org
> > > > > > > >> > >> >> > > >
> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
> > > > 40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >>
> > > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >>
> > > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
> > > McKinney <
> > > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > > >> > >> >> > > > wrote:
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > > > > >>> hi all,
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in
> > > a
> > > > position to
> > > > > > > >> > >> release
> > > > > > > >> > >> >> > > at
> > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope that
> > > one
> > > > more week of
> > > > > > > >> > >> work (or
> > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us there.
> > > Aside
> > > > from merging
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make sure
> > > > that our
> > > > > > > >> > >> packaging jobs
> > > > > > > >> > >> >> > > > > > >>> required for the release candidate are all
> > > > working.
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from the
> > > > 0.15.0 backlog
> > > > > > > >> > >> that they
> > > > > > > >> > >> >> > > > don't
> > > > > > > >> > >> >> > > > > > >>> think they will finish by end of next week
> > > > that would
> > > > > > > >> > >> help focus
> > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in
> > > > 0.15.0 still).
> > > > > > > >> > >> I am
> > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small features
> > > > related to
> > > > > > > >> > >> dictionaries
> > > > > > > >> > >> >> > > > while
> > > > > > > >> > >> >> > > > > > >>> the release window is still open.
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > > > > >>> - Wes
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
> > > > McKinney <
> > > > > > > >> > >> >> > > wesmckinn@gmail.com>
> > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > hi,
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > I think we should try to release the
> > > week
> > > > of September
> > > > > > > >> > >> 9, so
> > > > > > > >> > >> >> > > > > > >>> > development work should be completed by
> > > > end of next
> > > > > > > >> > >> week.
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
> > > protocol
> > > > alignment
> > > > > > > >> > >> changes for
> > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I
> > > think
> > > > that getting
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main barrier
> > > > to releasing.
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > Thanks
> > > > > > > >> > >> >> > > > > > >>> > Wes
> > > > > > > >> > >> >> > > > > > >>> >
> > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think
> > > > of several
> > > > > > > >> > >> bugs that
> > > > > > > >> > >> >> > > > need
> > > > > > > >> > >> >> > > > > > >>> > > to
> > > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are
> > > > required in
> > > > > > > >> > >> IPC streams
> > > > > > > >> > >> >> > > > > > >>> > > even
> > > > > > > >> > >> >> > > > > > >>> when empty[1]
> > > > > > > >> > >> >> > > > > > >>> > > This one is under review now, however
> > > > through this
> > > > > > > >> > >> PR we find
> > > > > > > >> > >> >> > > > > > >>> > > that
> > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading and
> > > > writing
> > > > > > > >> > >> dictionaries in IPC
> > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since
> > > it
> > > > assumes all
> > > > > > > >> > >> >> > > > dictionaries
> > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see details in
> > > > PR comments,
> > > > > > > >> > >> and this
> > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15).
> > > > @Micah Kornfield
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as
> > > > strings in
> > > > > > > >> > >> integration
> > > > > > > >> > >> >> > > > test
> > > > > > > >> > >> >> > > > > > >>> JSON files[3]
> > > > > > > >> > >> >> > > > > > >>> > > Java side code already checked in,
> > > other
> > > > > > > >> > >> implementations
> > > > > > > >> > >> >> > > seems
> > > > > > > >> > >> >> > > > not.
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
> > > > JdbcAdapter[4]
> > > > > > > >> > >> Caused by
> > > > > > > >> > >> >> > > trying
> > > > > > > >> > >> >> > > > > > >>> > > to load all records in one contiguous
> > > > batch, fixed
> > > > > > > >> > >> >> > > > > > >>> by providing iterator API for iteratively
> > > > reading in
> > > > > > > >> > >> >> > > ARROW-6219[5].
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > Thanks,
> > > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > [1]
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > > > > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >>
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > > > >> > >> >> > > > > > >>> sues.apache.org
> > > > > > > >> > >> >> > > >
> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >>
> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >>
> > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > >
> > > > ----------------------------------------------------------------
> > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> > > > wesmckinn@gmail.com> Send
> > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > > > > > > >> > >> dev@arrow.apache.org>
> > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
> > > release
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing
> > > > the 0.15.0
> > > > > > > >> > >> backlog some
> > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help
> > > with
> > > > grooming
> > > > > > > >> > >> >> > > (particularly
> > > > > > > >> > >> >> > > > > > >>> > > for languages other than C++/Python
> > > > where I'm
> > > > > > > >> > >> focusing) that
> > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have been
> > > > almost 500 JIRA
> > > > > > > >> > >> issues
> > > > > > > >> > >> >> > > opened
> > > > > > > >> > >> >> > > > > > >>> > > since the
> > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure
> > > > to check
> > > > > > > >> > >> whether
> > > > > > > >> > >> >> > > there's
> > > > > > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs
> > > > that we should
> > > > > > > >> > >> try to
> > > > > > > >> > >> >> > > fix
> > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes
> > > > McKinney
> > > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1
> > > > seems to be
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> > > 40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the
> > > > Windows
> > > > > > > >> > >> changes in
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > > > > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
> > > > volunteer would look
> > > > > > > >> > >> into what
> > > > > > > >> > >> >> > > > > > >>> > > > was
> > > > > > > >> > >> >> > > > > > >>> wrong
> > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows.
> > > > Otherwise
> > > > > > > >> > >> 0.15.0 Windows
> > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > > > > > >> > >> 40microsoft.com
> > > > > > > >> > >> >> > > > %7Ccbea
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > >
> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM
> > > > Antoine Pitrou <
> > > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
> > > > Micah
> > > > > > > >> > >> Kornfield
> > > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > > > > >> > >> >> > > > > > >>> > > > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could have
> > > > 32-bit array
> > > > > > > >> > >> lengths and
> > > > > > > >> > >> >> > > > > > >>> variable-length
> > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if
> > > we
> > > > wanted (we
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > > > wouldn't
> > > > > > > >> > >> >> > > > > > >>> > > > > > > be
> > > > > > > >> > >> >> > > > > > >>> able to
> > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with more
> > > > than INT32_MAX
> > > > > > > >> > >> elements).
> > > > > > > >> > >> >> > > > > > >>> > > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do
> > > > this in C++
> > > > > > > >> > >> but we
> > > > > > > >> > >> >> > > > don't.
> > > > > > > >> > >> >> > > > > > >>> I'm not sure we
> > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
> > > "Large"
> > > > types if we
> > > > > > > >> > >> did.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much
> > > > space as 32-bit
> > > > > > > >> > >> >> > > offsets,
> > > > > > > >> > >> >> > > > > > >>> > > > > so if
> > > > > > > >> > >> >> > > > > > >>> you're
> > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or
> > > > strings,
> > > > > > > >> > >> 32-bit
> > > > > > > >> > >> >> > > offsets
> > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with
> > > > 64-bit array
> > > > > > > >> > >> lengths from
> > > > > > > >> > >> >> > > > the
> > > > > > > >> > >> >> > > > > > >>> > > > > start
> > > > > > > >> > >> >> > > > > > >>> it would
> > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types
> > > > with 32-bit
> > > > > > > >> > >> offsets.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address
> > > > space in Java
> > > > > > > >> > >> and
> > > > > > > >> > >> >> > > calling
> > > > > > > >> > >> >> > > > > > >>> > > > > > it a
> > > > > > > >> > >> >> > > > > > >>> reference
> > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal.
> > > > If a consumer
> > > > > > > >> > >> uses a
> > > > > > > >> > >> >> > > > "Large"
> > > > > > > >> > >> >> > > > > > >>> type
> > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because they
> > > > need the ability
> > > > > > > >> > >> to store
> > > > > > > >> > >> >> > > > > > >>> > > > > > more
> > > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a column,
> > > > otherwise it is
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > > wasting
> > > > > > > >> > >> >> > > > > > >>> > > > > > space
> > > > > > > >> > >> >> > > > > > >>> [1].
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual
> > > > elements
> > > > > > > >> > >> (lists or
> > > > > > > >> > >> >> > > > > > >>> > > > > strings)
> > > > > > > >> > >> >> > > > > > >>> are
> > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> > > > proportion,
> > > > > > > >> > >> so it may
> > > > > > > >> > >> >> > > be
> > > > > > > >> > >> >> > > > > > >>> simpler in
> > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always create a
> > > > "Large" type
> > > > > > > >> > >> array.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically
> > > there
> > > > might be some
> > > > > > > >> > >> >> > > > > > >>> > > > > > performance
> > > > > > > >> > >> >> > > > > > >>> benefits on
> > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using
> > > the
> > > > native word
> > > > > > > >> > >> sizes.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
> > > > architectures don't do
> > > > > > > >> > >> that, as
> > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > > > > > >> > >> >> > > > > > >>> is an
> > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer size even
> > > > in
> > > > > > > >> > >> high-performance
> > > > > > > >> > >> >> > > > code.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > Regards
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > > >> > >> >> > > > > > >>> > >
> > > > > > > >> > >> >> > > > > > >>>
> > > > > > > >> > >> >> > > > > > >>
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >>
> > > > > > > >> > >
> > > >
> > >
Re: Timeline for 0.15.0 release
Posted by Neal Richardson <ne...@gmail.com>.
Andy's DataFusion issue and Wes's Parquet one have both been merged,
and it looks like the LICENSE issue is being resolved as I type. So
are we good to go now?
Neal
On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <an...@gmail.com> wrote:
>
> I found a last minute issue with DataFusion (Rust) and would appreciate it
> if we could merge ARROW-6086 (PR is
> https://github.com/apache/arrow/pull/5494) before cutting the RC.
>
> Thanks,
>
> Andy.
>
>
> On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > OK, I'm going to postpone cutting a release until tomorrow (hoping we can
> > issues resolved by then).. I'll also try to review the third-party
> > additions since 14.x.
> >
> > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > I found a licensing issue
> > >
> > > https://issues.apache.org/jira/browse/ARROW-6679
> > >
> > > It might be worth examining third party code added to the project
> > > since 0.14.x to make sure there are no other such issues.
> > >
> > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > I have diagnosed the problem (Thrift "string" data must be UTF-8,
> > > > cannot be arbitrary binary) and am working on a patch right now
> > > >
> > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > I just opened
> > > > >
> > > > > https://issues.apache.org/jira/browse/ARROW-6678
> > > > >
> > > > > Please don't cut an RC until I have an opportunity to diagnose this,
> > > > > will report back.
> > > > >
> > > > >
> > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > I'm investigating a possible Parquet-related compatibility bug
> > that I
> > > > > > encountered through some routine testing / benchmarking. I'll
> > report
> > > > > > back once I figure out what is going on (if anything)
> > > > > >
> > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> > > emkornfield@gmail.com> wrote:
> > > > > > >>
> > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > > get it
> > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > >
> > > > > > > That won't be an option for me this week (it seems like I would
> > > need to meet one face-to-face). I'll try to get the GPG checked in and
> > the
> > > rest of the pre-requisites done tomorrow (Monday) to hopefully start the
> > > release on Tuesday (hopefully we can solve the last blocker/integration
> > > tests by then).
> > > > > > >
> > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > > get it
> > > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > > >>
> > > > > > >> Speaking of the release, there are at least 2 code changes I
> > still
> > > > > > >> want to get in
> > > > > > >>
> > > > > > >> ARROW-5717
> > > > > > >> ARROW-6353
> > > > > > >>
> > > > > > >> I just pushed updates to ARROW-5717, will merge once the build
> > is
> > > green.
> > > > > > >>
> > > > > > >> There are a couple of Rust patches still marked for 0.15. The
> > rest
> > > > > > >> seems to be documentation and a couple of integration test
> > > failures we
> > > > > > >> should see about fixing in time.
> > > > > > >>
> > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> > > emkornfield@gmail.com> wrote:
> > > > > > >> >
> > > > > > >> > Thanks Krisztián and Wes,
> > > > > > >> > I've gone ahead and started registering myself on all the
> > > packaging sites.
> > > > > > >> >
> > > > > > >> > Is there any review process when adding my GPG key to the SVN
> > > file? [1]
> > > > > > >> > doesn't seem to mention explicitly.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Micah
> > > > > > >> >
> > > > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > > > > > >> >
> > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> > > szucs.krisztian@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> > > wesmckinn@gmail.com> wrote:
> > > > > > >> > >
> > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> > > emkornfield@gmail.com>
> > > > > > >> > >> wrote:
> > > > > > >> > >> >>
> > > > > > >> > >> >> The process should be well documented at this point but
> > > there are a
> > > > > > >> > >> >> number of steps.
> > > > > > >> > >> >
> > > > > > >> > >> > Is [1] the up-to-date documentation for the release?
> > Are
> > > there
> > > > > > >> > >> instructions for the adding the code signing Key to SVN?
> > > > > > >> > >> >
> > > > > > >> > >> > I will make a go of it. i will try to mitigate any
> > > internet issues by
> > > > > > >> > >> doing the process for a cloud instance (I assume that isn't
> > > a problem?).
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> > >> Setting up a new cloud environment suitable for producing
> > an
> > > RC may be
> > > > > > >> > >> time consuming, but you are welcome to try. Krisztian --
> > are
> > > you
> > > > > > >> > >> available next week to help Micah and potentially take over
> > > producing
> > > > > > >> > >> the RC if there are issues?
> > > > > > >> > >>
> > > > > > >> > > Sure, I'll be available next week. We can also grant access
> > to
> > > > > > >> > > https://github.com/ursa-labs/crossbow because configuring
> > all
> > > > > > >> > > the CI backends can be time consuming.
> > > > > > >> > >
> > > > > > >> > >>
> > > > > > >> > >> > Thanks,
> > > > > > >> > >> > Micah
> > > > > > >> > >> >
> > > > > > >> > >> > [1]
> > > > > > >> > >>
> > >
> > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > > > > >> > >> >
> > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > > > >> > >> wrote:
> > > > > > >> > >> >>
> > > > > > >> > >> >> The process should be well documented at this point but
> > > there are a
> > > > > > >> > >> >> number of steps. Note that you need to add your code
> > > signing key to
> > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I
> > > think it's fine
> > > > > > >> > >> >> to hand off the process to others after the VOTE but it
> > > would be
> > > > > > >> > >> >> tricky to have multiple RMs involved with producing the
> > > source and
> > > > > > >> > >> >> binary artifacts for the vote
> > > > > > >> > >> >>
> > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > > > > >> > >> emkornfield@gmail.com> wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > SGTM, as well.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > I should have a little bit of time next week if I can
> > > help as RM but
> > > > > > >> > >> I have
> > > > > > >> > >> >> > a couple of concerns:
> > > > > > >> > >> >> > 1. In the past I've had trouble downloading and
> > > validating
> > > > > > >> > >> releases. I'm a
> > > > > > >> > >> >> > bit worried, that I might have similar problems doing
> > > the necessary
> > > > > > >> > >> uploads.
> > > > > > >> > >> >> > 2. My internet connection will likely be not great, I
> > > don't know if
> > > > > > >> > >> this
> > > > > > >> > >> >> > would make it even less likely to be successful.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Does it become problematic if somehow I would have to
> > > abandon the
> > > > > > >> > >> process
> > > > > > >> > >> >> > mid-release? Is there anyone who could serve as a
> > > backup? Are the
> > > > > > >> > >> steps
> > > > > > >> > >> >> > well documented?
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Thanks,
> > > > > > >> > >> >> > Micah
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > > > > > >> > >> neal.p.richardson@gmail.com>
> > > > > > >> > >> >> > wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > > Sounds good to me.
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Neal
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > > > >> > >> wrote:
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > > hi all,
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > It looks like we're drawing close to be able to
> > > make the 0.15.0
> > > > > > >> > >> >> > > > release. I would suggest "pencils down" at the end
> > > of this week
> > > > > > >> > >> and
> > > > > > >> > >> >> > > > see if a release candidate can be produced next
> > > Monday September
> > > > > > >> > >> 23.
> > > > > > >> > >> >> > > > Any thoughts or objections?
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > Thanks,
> > > > > > >> > >> >> > > > Wes
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > >> > >> >> > > wrote:
> > > > > > >> > >> >> > > > >
> > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to
> > > amend the
> > > > > > >> > >> Format docs
> > > > > > >> > >> >> > > > > today regarding the EOS issue and also update
> > the
> > > C++ library
> > > > > > >> > >> >> > > > >
> > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > I assume the plan is to merge the
> > > > > > >> > >> ARROW-6313-flatbuffer-alignment
> > > > > > >> > >> >> > > > branch into master before the 0.15 release,
> > correct?
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are
> > > ready to be
> > > > > > >> > >> merged into
> > > > > > >> > >> >> > > > the alignment branch -
> > > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > Eric
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > -----Original Message-----
> > > > > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> > > niki.lj@aliyun.com>
> > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > I should have a little more bandwidth to help
> > > with some of
> > > > > > >> > >> the
> > > > > > >> > >> >> > > > packaging starting tomorrow and going into the
> > > weekend.
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > >> > >> >> > > > wrote:
> > > > > > >> > >> >> > > > > >
> > > > > > >> > >> >> > > > > > > Hi folks,
> > > > > > >> > >> >> > > > > > >
> > > > > > >> > >> >> > > > > > > With the state of nightly packaging and
> > > integration builds
> > > > > > >> > >> things
> > > > > > >> > >> >> > > > > > > aren't looking too good for being in release
> > > readiness by
> > > > > > >> > >> the end
> > > > > > >> > >> >> > > of
> > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning
> > > to be working
> > > > > > >> > >> to close
> > > > > > >> > >> >> > > as
> > > > > > >> > >> >> > > > > > > many issues as I can and also to help with
> > > the ongoing
> > > > > > >> > >> alignment
> > > > > > >> > >> >> > > > fixes.
> > > > > > >> > >> >> > > > > > >
> > > > > > >> > >> >> > > > > > > Wes
> > > > > > >> > >> >> > > > > > >
> > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
> > Kornfield
> > > <
> > > > > > >> > >> >> > > emkornfield@gmail.com
> > > > > > >> > >> >> > > > >
> > > > > > >> > >> >> > > > > > > wrote:
> > > > > > >> > >> >> > > > > > >
> > > > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of
> > > the current
> > > > > > >> > >> issues:
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > >
> > > > > > >> > >>
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > > >> > >> >> > > > > > >> ki.apache.org
> > > > > > >> > >> >> > > >
> > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
> > > 40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > >
> > > > > > >> > >>
> > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > >
> > > > > > >> > >>
> > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > > >> > >> >> > > > > > >> %3D&reserved=0
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
> > McKinney <
> > > > > > >> > >> wesmckinn@gmail.com>
> > > > > > >> > >> >> > > > wrote:
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > > > > >>> hi all,
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in
> > a
> > > position to
> > > > > > >> > >> release
> > > > > > >> > >> >> > > at
> > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope that
> > one
> > > more week of
> > > > > > >> > >> work (or
> > > > > > >> > >> >> > > > > > >>> less) will be enough to get us there.
> > Aside
> > > from merging
> > > > > > >> > >> the
> > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make sure
> > > that our
> > > > > > >> > >> packaging jobs
> > > > > > >> > >> >> > > > > > >>> required for the release candidate are all
> > > working.
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > > > > >>> If folks could remove issues from the
> > > 0.15.0 backlog
> > > > > > >> > >> that they
> > > > > > >> > >> >> > > > don't
> > > > > > >> > >> >> > > > > > >>> think they will finish by end of next week
> > > that would
> > > > > > >> > >> help focus
> > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in
> > > 0.15.0 still).
> > > > > > >> > >> I am
> > > > > > >> > >> >> > > > > > >>> looking to tackle a few small features
> > > related to
> > > > > > >> > >> dictionaries
> > > > > > >> > >> >> > > > while
> > > > > > >> > >> >> > > > > > >>> the release window is still open.
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > > > > >>> - Wes
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
> > > McKinney <
> > > > > > >> > >> >> > > wesmckinn@gmail.com>
> > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > hi,
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > I think we should try to release the
> > week
> > > of September
> > > > > > >> > >> 9, so
> > > > > > >> > >> >> > > > > > >>> > development work should be completed by
> > > end of next
> > > > > > >> > >> week.
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
> > protocol
> > > alignment
> > > > > > >> > >> changes for
> > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I
> > think
> > > that getting
> > > > > > >> > >> the
> > > > > > >> > >> >> > > > > > >>> > alignment work done is the main barrier
> > > to releasing.
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > Thanks
> > > > > > >> > >> >> > > > > > >>> > Wes
> > > > > > >> > >> >> > > > > > >>> >
> > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think
> > > of several
> > > > > > >> > >> bugs that
> > > > > > >> > >> >> > > > need
> > > > > > >> > >> >> > > > > > >>> > > to
> > > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are
> > > required in
> > > > > > >> > >> IPC streams
> > > > > > >> > >> >> > > > > > >>> > > even
> > > > > > >> > >> >> > > > > > >>> when empty[1]
> > > > > > >> > >> >> > > > > > >>> > > This one is under review now, however
> > > through this
> > > > > > >> > >> PR we find
> > > > > > >> > >> >> > > > > > >>> > > that
> > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading and
> > > writing
> > > > > > >> > >> dictionaries in IPC
> > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since
> > it
> > > assumes all
> > > > > > >> > >> >> > > > dictionaries
> > > > > > >> > >> >> > > > > > >>> are at the start of stream (see details in
> > > PR comments,
> > > > > > >> > >> and this
> > > > > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15).
> > > @Micah Kornfield
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as
> > > strings in
> > > > > > >> > >> integration
> > > > > > >> > >> >> > > > test
> > > > > > >> > >> >> > > > > > >>> JSON files[3]
> > > > > > >> > >> >> > > > > > >>> > > Java side code already checked in,
> > other
> > > > > > >> > >> implementations
> > > > > > >> > >> >> > > seems
> > > > > > >> > >> >> > > > not.
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
> > > JdbcAdapter[4]
> > > > > > >> > >> Caused by
> > > > > > >> > >> >> > > trying
> > > > > > >> > >> >> > > > > > >>> > > to load all records in one contiguous
> > > batch, fixed
> > > > > > >> > >> >> > > > > > >>> by providing iterator API for iteratively
> > > reading in
> > > > > > >> > >> >> > > ARROW-6219[5].
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > Thanks,
> > > > > > >> > >> >> > > > > > >>> > > Ji Liu
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > [1]
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > > > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > >
> > > > > > >> > >>
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > > >> > >> >> > > > > > >>> sues.apache.org
> > > > > > >> > >> >> > > >
> > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > >
> > > > > > >> > >>
> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >>
> > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > >
> > > ----------------------------------------------------------------
> > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> > > wesmckinn@gmail.com> Send
> > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > > > > > >> > >> dev@arrow.apache.org>
> > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
> > release
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing
> > > the 0.15.0
> > > > > > >> > >> backlog some
> > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help
> > with
> > > grooming
> > > > > > >> > >> >> > > (particularly
> > > > > > >> > >> >> > > > > > >>> > > for languages other than C++/Python
> > > where I'm
> > > > > > >> > >> focusing) that
> > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have been
> > > almost 500 JIRA
> > > > > > >> > >> issues
> > > > > > >> > >> >> > > opened
> > > > > > >> > >> >> > > > > > >>> > > since the
> > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure
> > > to check
> > > > > > >> > >> whether
> > > > > > >> > >> >> > > there's
> > > > > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs
> > > that we should
> > > > > > >> > >> try to
> > > > > > >> > >> >> > > fix
> > > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes
> > > McKinney
> > > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > > > > >> > >> >> > > > > > >>> wrote:
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1
> > > seems to be
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> > 40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the
> > > Windows
> > > > > > >> > >> changes in
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > > > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
> > > volunteer would look
> > > > > > >> > >> into what
> > > > > > >> > >> >> > > > > > >>> > > > was
> > > > > > >> > >> >> > > > > > >>> wrong
> > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows.
> > > Otherwise
> > > > > > >> > >> 0.15.0 Windows
> > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > > >> > >> >> > > > > > >>> > > >
> > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > > > > >> > >> 40microsoft.com
> > > > > > >> > >> >> > > > %7Ccbea
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > >
> > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > > >> > >> >> > > > > > >>> > > >
> > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > > >> > >> >> > > > > > >>> > > >
> > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM
> > > Antoine Pitrou <
> > > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
> > > Micah
> > > > > > >> > >> Kornfield
> > > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > > > >> > >> >> > > > > > >>> > > > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could have
> > > 32-bit array
> > > > > > >> > >> lengths and
> > > > > > >> > >> >> > > > > > >>> variable-length
> > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if
> > we
> > > wanted (we
> > > > > > >> > >> just
> > > > > > >> > >> >> > > > wouldn't
> > > > > > >> > >> >> > > > > > >>> > > > > > > be
> > > > > > >> > >> >> > > > > > >>> able to
> > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with more
> > > than INT32_MAX
> > > > > > >> > >> elements).
> > > > > > >> > >> >> > > > > > >>> > > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do
> > > this in C++
> > > > > > >> > >> but we
> > > > > > >> > >> >> > > > don't.
> > > > > > >> > >> >> > > > > > >>> I'm not sure we
> > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
> > "Large"
> > > types if we
> > > > > > >> > >> did.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much
> > > space as 32-bit
> > > > > > >> > >> >> > > offsets,
> > > > > > >> > >> >> > > > > > >>> > > > > so if
> > > > > > >> > >> >> > > > > > >>> you're
> > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or
> > > strings,
> > > > > > >> > >> 32-bit
> > > > > > >> > >> >> > > offsets
> > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with
> > > 64-bit array
> > > > > > >> > >> lengths from
> > > > > > >> > >> >> > > > the
> > > > > > >> > >> >> > > > > > >>> > > > > start
> > > > > > >> > >> >> > > > > > >>> it would
> > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types
> > > with 32-bit
> > > > > > >> > >> offsets.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address
> > > space in Java
> > > > > > >> > >> and
> > > > > > >> > >> >> > > calling
> > > > > > >> > >> >> > > > > > >>> > > > > > it a
> > > > > > >> > >> >> > > > > > >>> reference
> > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal.
> > > If a consumer
> > > > > > >> > >> uses a
> > > > > > >> > >> >> > > > "Large"
> > > > > > >> > >> >> > > > > > >>> type
> > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because they
> > > need the ability
> > > > > > >> > >> to store
> > > > > > >> > >> >> > > > > > >>> > > > > > more
> > > > > > >> > >> >> > > > > > >>> than INT32_MAX
> > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a column,
> > > otherwise it is
> > > > > > >> > >> just
> > > > > > >> > >> >> > > wasting
> > > > > > >> > >> >> > > > > > >>> > > > > > space
> > > > > > >> > >> >> > > > > > >>> [1].
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual
> > > elements
> > > > > > >> > >> (lists or
> > > > > > >> > >> >> > > > > > >>> > > > > strings)
> > > > > > >> > >> >> > > > > > >>> are
> > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> > > proportion,
> > > > > > >> > >> so it may
> > > > > > >> > >> >> > > be
> > > > > > >> > >> >> > > > > > >>> simpler in
> > > > > > >> > >> >> > > > > > >>> > > > > such a case to always create a
> > > "Large" type
> > > > > > >> > >> array.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically
> > there
> > > might be some
> > > > > > >> > >> >> > > > > > >>> > > > > > performance
> > > > > > >> > >> >> > > > > > >>> benefits on
> > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using
> > the
> > > native word
> > > > > > >> > >> sizes.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
> > > architectures don't do
> > > > > > >> > >> that, as
> > > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > > > > >> > >> >> > > > > > >>> is an
> > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer size even
> > > in
> > > > > > >> > >> high-performance
> > > > > > >> > >> >> > > > code.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > Regards
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > > > >
> > > > > > >> > >> >> > > > > > >>> > >
> > > > > > >> > >> >> > > > > > >>>
> > > > > > >> > >> >> > > > > > >>
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > >
> > > > > > >> > >>
> > > > > > >> > >
> > >
> >
Re: Timeline for 0.15.0 release
Posted by Andy Grove <an...@gmail.com>.
I found a last minute issue with DataFusion (Rust) and would appreciate it
if we could merge ARROW-6086 (PR is
https://github.com/apache/arrow/pull/5494) before cutting the RC.
Thanks,
Andy.
On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield <em...@gmail.com>
wrote:
> OK, I'm going to postpone cutting a release until tomorrow (hoping we can
> issues resolved by then).. I'll also try to review the third-party
> additions since 14.x.
>
> On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I found a licensing issue
> >
> > https://issues.apache.org/jira/browse/ARROW-6679
> >
> > It might be worth examining third party code added to the project
> > since 0.14.x to make sure there are no other such issues.
> >
> > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > I have diagnosed the problem (Thrift "string" data must be UTF-8,
> > > cannot be arbitrary binary) and am working on a patch right now
> > >
> > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > I just opened
> > > >
> > > > https://issues.apache.org/jira/browse/ARROW-6678
> > > >
> > > > Please don't cut an RC until I have an opportunity to diagnose this,
> > > > will report back.
> > > >
> > > >
> > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > > >
> > > > > I'm investigating a possible Parquet-related compatibility bug
> that I
> > > > > encountered through some routine testing / benchmarking. I'll
> report
> > > > > back once I figure out what is going on (if anything)
> > > > >
> > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> > emkornfield@gmail.com> wrote:
> > > > > >>
> > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > get it
> > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > >
> > > > > > That won't be an option for me this week (it seems like I would
> > need to meet one face-to-face). I'll try to get the GPG checked in and
> the
> > rest of the pre-requisites done tomorrow (Monday) to hopefully start the
> > release on Tuesday (hopefully we can solve the last blocker/integration
> > tests by then).
> > > > > >
> > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <
> wesmckinn@gmail.com>
> > wrote:
> > > > > >>
> > > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> > get it
> > > > > >> signed by another PMC member), but is not 100% essential.
> > > > > >>
> > > > > >> Speaking of the release, there are at least 2 code changes I
> still
> > > > > >> want to get in
> > > > > >>
> > > > > >> ARROW-5717
> > > > > >> ARROW-6353
> > > > > >>
> > > > > >> I just pushed updates to ARROW-5717, will merge once the build
> is
> > green.
> > > > > >>
> > > > > >> There are a couple of Rust patches still marked for 0.15. The
> rest
> > > > > >> seems to be documentation and a couple of integration test
> > failures we
> > > > > >> should see about fixing in time.
> > > > > >>
> > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> > emkornfield@gmail.com> wrote:
> > > > > >> >
> > > > > >> > Thanks Krisztián and Wes,
> > > > > >> > I've gone ahead and started registering myself on all the
> > packaging sites.
> > > > > >> >
> > > > > >> > Is there any review process when adding my GPG key to the SVN
> > file? [1]
> > > > > >> > doesn't seem to mention explicitly.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Micah
> > > > > >> >
> > > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > > > > >> >
> > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> > szucs.krisztian@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> > wesmckinn@gmail.com> wrote:
> > > > > >> > >
> > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> > emkornfield@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> The process should be well documented at this point but
> > there are a
> > > > > >> > >> >> number of steps.
> > > > > >> > >> >
> > > > > >> > >> > Is [1] the up-to-date documentation for the release?
> Are
> > there
> > > > > >> > >> instructions for the adding the code signing Key to SVN?
> > > > > >> > >> >
> > > > > >> > >> > I will make a go of it. i will try to mitigate any
> > internet issues by
> > > > > >> > >> doing the process for a cloud instance (I assume that isn't
> > a problem?).
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >> Setting up a new cloud environment suitable for producing
> an
> > RC may be
> > > > > >> > >> time consuming, but you are welcome to try. Krisztian --
> are
> > you
> > > > > >> > >> available next week to help Micah and potentially take over
> > producing
> > > > > >> > >> the RC if there are issues?
> > > > > >> > >>
> > > > > >> > > Sure, I'll be available next week. We can also grant access
> to
> > > > > >> > > https://github.com/ursa-labs/crossbow because configuring
> all
> > > > > >> > > the CI backends can be time consuming.
> > > > > >> > >
> > > > > >> > >>
> > > > > >> > >> > Thanks,
> > > > > >> > >> > Micah
> > > > > >> > >> >
> > > > > >> > >> > [1]
> > > > > >> > >>
> >
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > > > >> > >> >
> > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> > wesmckinn@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> The process should be well documented at this point but
> > there are a
> > > > > >> > >> >> number of steps. Note that you need to add your code
> > signing key to
> > > > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I
> > think it's fine
> > > > > >> > >> >> to hand off the process to others after the VOTE but it
> > would be
> > > > > >> > >> >> tricky to have multiple RMs involved with producing the
> > source and
> > > > > >> > >> >> binary artifacts for the vote
> > > > > >> > >> >>
> > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > > > >> > >> emkornfield@gmail.com> wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > SGTM, as well.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I should have a little bit of time next week if I can
> > help as RM but
> > > > > >> > >> I have
> > > > > >> > >> >> > a couple of concerns:
> > > > > >> > >> >> > 1. In the past I've had trouble downloading and
> > validating
> > > > > >> > >> releases. I'm a
> > > > > >> > >> >> > bit worried, that I might have similar problems doing
> > the necessary
> > > > > >> > >> uploads.
> > > > > >> > >> >> > 2. My internet connection will likely be not great, I
> > don't know if
> > > > > >> > >> this
> > > > > >> > >> >> > would make it even less likely to be successful.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Does it become problematic if somehow I would have to
> > abandon the
> > > > > >> > >> process
> > > > > >> > >> >> > mid-release? Is there anyone who could serve as a
> > backup? Are the
> > > > > >> > >> steps
> > > > > >> > >> >> > well documented?
> > > > > >> > >> >> >
> > > > > >> > >> >> > Thanks,
> > > > > >> > >> >> > Micah
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > > > > >> > >> neal.p.richardson@gmail.com>
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > Sounds good to me.
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Neal
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > > hi all,
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > It looks like we're drawing close to be able to
> > make the 0.15.0
> > > > > >> > >> >> > > > release. I would suggest "pencils down" at the end
> > of this week
> > > > > >> > >> and
> > > > > >> > >> >> > > > see if a release candidate can be produced next
> > Monday September
> > > > > >> > >> 23.
> > > > > >> > >> >> > > > Any thoughts or objections?
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Thanks,
> > > > > >> > >> >> > > > Wes
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > > > > >> > >> wesmckinn@gmail.com>
> > > > > >> > >> >> > > wrote:
> > > > > >> > >> >> > > > >
> > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to
> > amend the
> > > > > >> > >> Format docs
> > > > > >> > >> >> > > > > today regarding the EOS issue and also update
> the
> > C++ library
> > > > > >> > >> >> > > > >
> > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > I assume the plan is to merge the
> > > > > >> > >> ARROW-6313-flatbuffer-alignment
> > > > > >> > >> >> > > > branch into master before the 0.15 release,
> correct?
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are
> > ready to be
> > > > > >> > >> merged into
> > > > > >> > >> >> > > > the alignment branch -
> > > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > Eric
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > -----Original Message-----
> > > > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> > niki.lj@aliyun.com>
> > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > I should have a little more bandwidth to help
> > with some of
> > > > > >> > >> the
> > > > > >> > >> >> > > > packaging starting tomorrow and going into the
> > weekend.
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > > > > >> > >> wesmckinn@gmail.com>
> > > > > >> > >> >> > > > wrote:
> > > > > >> > >> >> > > > > >
> > > > > >> > >> >> > > > > > > Hi folks,
> > > > > >> > >> >> > > > > > >
> > > > > >> > >> >> > > > > > > With the state of nightly packaging and
> > integration builds
> > > > > >> > >> things
> > > > > >> > >> >> > > > > > > aren't looking too good for being in release
> > readiness by
> > > > > >> > >> the end
> > > > > >> > >> >> > > of
> > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning
> > to be working
> > > > > >> > >> to close
> > > > > >> > >> >> > > as
> > > > > >> > >> >> > > > > > > many issues as I can and also to help with
> > the ongoing
> > > > > >> > >> alignment
> > > > > >> > >> >> > > > fixes.
> > > > > >> > >> >> > > > > > >
> > > > > >> > >> >> > > > > > > Wes
> > > > > >> > >> >> > > > > > >
> > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah
> Kornfield
> > <
> > > > > >> > >> >> > > emkornfield@gmail.com
> > > > > >> > >> >> > > > >
> > > > > >> > >> >> > > > > > > wrote:
> > > > > >> > >> >> > > > > > >
> > > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of
> > the current
> > > > > >> > >> issues:
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > >
> > > > > >> > >>
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > >> > >> >> > > > > > >> ki.apache.org
> > > > > >> > >> >> > > >
> > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
> > 40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > >
> > > > > >> > >>
> > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > >
> > > > > >> > >>
> > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > >> > >> >> > > > > > >> %3D&reserved=0
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes
> McKinney <
> > > > > >> > >> wesmckinn@gmail.com>
> > > > > >> > >> >> > > > wrote:
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > > > > >>> hi all,
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in
> a
> > position to
> > > > > >> > >> release
> > > > > >> > >> >> > > at
> > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope that
> one
> > more week of
> > > > > >> > >> work (or
> > > > > >> > >> >> > > > > > >>> less) will be enough to get us there.
> Aside
> > from merging
> > > > > >> > >> the
> > > > > >> > >> >> > > > > > >>> alignment changes, we need to make sure
> > that our
> > > > > >> > >> packaging jobs
> > > > > >> > >> >> > > > > > >>> required for the release candidate are all
> > working.
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > > > > >>> If folks could remove issues from the
> > 0.15.0 backlog
> > > > > >> > >> that they
> > > > > >> > >> >> > > > don't
> > > > > >> > >> >> > > > > > >>> think they will finish by end of next week
> > that would
> > > > > >> > >> help focus
> > > > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in
> > 0.15.0 still).
> > > > > >> > >> I am
> > > > > >> > >> >> > > > > > >>> looking to tackle a few small features
> > related to
> > > > > >> > >> dictionaries
> > > > > >> > >> >> > > > while
> > > > > >> > >> >> > > > > > >>> the release window is still open.
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > > > > >>> - Wes
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
> > McKinney <
> > > > > >> > >> >> > > wesmckinn@gmail.com>
> > > > > >> > >> >> > > > > > >>> wrote:
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > hi,
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > I think we should try to release the
> week
> > of September
> > > > > >> > >> 9, so
> > > > > >> > >> >> > > > > > >>> > development work should be completed by
> > end of next
> > > > > >> > >> week.
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the
> protocol
> > alignment
> > > > > >> > >> changes for
> > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I
> think
> > that getting
> > > > > >> > >> the
> > > > > >> > >> >> > > > > > >>> > alignment work done is the main barrier
> > to releasing.
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > Thanks
> > > > > >> > >> >> > > > > > >>> > Wes
> > > > > >> > >> >> > > > > > >>> >
> > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > >> > >> >> > > > > > >>> wrote:
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think
> > of several
> > > > > >> > >> bugs that
> > > > > >> > >> >> > > > need
> > > > > >> > >> >> > > > > > >>> > > to
> > > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are
> > required in
> > > > > >> > >> IPC streams
> > > > > >> > >> >> > > > > > >>> > > even
> > > > > >> > >> >> > > > > > >>> when empty[1]
> > > > > >> > >> >> > > > > > >>> > > This one is under review now, however
> > through this
> > > > > >> > >> PR we find
> > > > > >> > >> >> > > > > > >>> > > that
> > > > > >> > >> >> > > > > > >>> there seems a bug in java reading and
> > writing
> > > > > >> > >> dictionaries in IPC
> > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since
> it
> > assumes all
> > > > > >> > >> >> > > > dictionaries
> > > > > >> > >> >> > > > > > >>> are at the start of stream (see details in
> > PR comments,
> > > > > >> > >> and this
> > > > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15).
> > @Micah Kornfield
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as
> > strings in
> > > > > >> > >> integration
> > > > > >> > >> >> > > > test
> > > > > >> > >> >> > > > > > >>> JSON files[3]
> > > > > >> > >> >> > > > > > >>> > > Java side code already checked in,
> other
> > > > > >> > >> implementations
> > > > > >> > >> >> > > seems
> > > > > >> > >> >> > > > not.
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
> > JdbcAdapter[4]
> > > > > >> > >> Caused by
> > > > > >> > >> >> > > trying
> > > > > >> > >> >> > > > > > >>> > > to load all records in one contiguous
> > batch, fixed
> > > > > >> > >> >> > > > > > >>> by providing iterator API for iteratively
> > reading in
> > > > > >> > >> >> > > ARROW-6219[5].
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > Thanks,
> > > > > >> > >> >> > > > > > >>> > > Ji Liu
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > [1]
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > >
> > > > > >> > >>
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > >> > >> >> > > > > > >>> sues.apache.org
> > > > > >> > >> >> > > >
> > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > >
> > > > > >> > >>
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >>
> > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > >
> > ----------------------------------------------------------------
> > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> > wesmckinn@gmail.com> Send
> > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > > > > >> > >> dev@arrow.apache.org>
> > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0
> release
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing
> > the 0.15.0
> > > > > >> > >> backlog some
> > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help
> with
> > grooming
> > > > > >> > >> >> > > (particularly
> > > > > >> > >> >> > > > > > >>> > > for languages other than C++/Python
> > where I'm
> > > > > >> > >> focusing) that
> > > > > >> > >> >> > > > > > >>> > > would be helpful. There have been
> > almost 500 JIRA
> > > > > >> > >> issues
> > > > > >> > >> >> > > opened
> > > > > >> > >> >> > > > > > >>> > > since the
> > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure
> > to check
> > > > > >> > >> whether
> > > > > >> > >> >> > > there's
> > > > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs
> > that we should
> > > > > >> > >> try to
> > > > > >> > >> >> > > fix
> > > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes
> > McKinney
> > > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > > > >> > >> >> > > > > > >>> wrote:
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1
> > seems to be
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%
> 40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the
> > Windows
> > > > > >> > >> changes in
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
> > volunteer would look
> > > > > >> > >> into what
> > > > > >> > >> >> > > > > > >>> > > > was
> > > > > >> > >> >> > > > > > >>> wrong
> > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows.
> > Otherwise
> > > > > >> > >> 0.15.0 Windows
> > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > >> > >> >> > > > > > >>> > > >
> > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > > > >> > >> 40microsoft.com
> > > > > >> > >> >> > > > %7Ccbea
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > >
> > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > >> > >> >> > > > > > >>> > > >
> > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > >> > >> >> > > > > > >>> > > >
> > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM
> > Antoine Pitrou <
> > > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
> > Micah
> > > > > >> > >> Kornfield
> > > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > > >> > >> >> > > > > > >>> > > > > > >
> > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could have
> > 32-bit array
> > > > > >> > >> lengths and
> > > > > >> > >> >> > > > > > >>> variable-length
> > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if
> we
> > wanted (we
> > > > > >> > >> just
> > > > > >> > >> >> > > > wouldn't
> > > > > >> > >> >> > > > > > >>> > > > > > > be
> > > > > >> > >> >> > > > > > >>> able to
> > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with more
> > than INT32_MAX
> > > > > >> > >> elements).
> > > > > >> > >> >> > > > > > >>> > > > > >
> > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do
> > this in C++
> > > > > >> > >> but we
> > > > > >> > >> >> > > > don't.
> > > > > >> > >> >> > > > > > >>> I'm not sure we
> > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the
> "Large"
> > types if we
> > > > > >> > >> did.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much
> > space as 32-bit
> > > > > >> > >> >> > > offsets,
> > > > > >> > >> >> > > > > > >>> > > > > so if
> > > > > >> > >> >> > > > > > >>> you're
> > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or
> > strings,
> > > > > >> > >> 32-bit
> > > > > >> > >> >> > > offsets
> > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with
> > 64-bit array
> > > > > >> > >> lengths from
> > > > > >> > >> >> > > > the
> > > > > >> > >> >> > > > > > >>> > > > > start
> > > > > >> > >> >> > > > > > >>> it would
> > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types
> > with 32-bit
> > > > > >> > >> offsets.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address
> > space in Java
> > > > > >> > >> and
> > > > > >> > >> >> > > calling
> > > > > >> > >> >> > > > > > >>> > > > > > it a
> > > > > >> > >> >> > > > > > >>> reference
> > > > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal.
> > If a consumer
> > > > > >> > >> uses a
> > > > > >> > >> >> > > > "Large"
> > > > > >> > >> >> > > > > > >>> type
> > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because they
> > need the ability
> > > > > >> > >> to store
> > > > > >> > >> >> > > > > > >>> > > > > > more
> > > > > >> > >> >> > > > > > >>> than INT32_MAX
> > > > > >> > >> >> > > > > > >>> > > > > > child elements in a column,
> > otherwise it is
> > > > > >> > >> just
> > > > > >> > >> >> > > wasting
> > > > > >> > >> >> > > > > > >>> > > > > > space
> > > > > >> > >> >> > > > > > >>> [1].
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual
> > elements
> > > > > >> > >> (lists or
> > > > > >> > >> >> > > > > > >>> > > > > strings)
> > > > > >> > >> >> > > > > > >>> are
> > > > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> > proportion,
> > > > > >> > >> so it may
> > > > > >> > >> >> > > be
> > > > > >> > >> >> > > > > > >>> simpler in
> > > > > >> > >> >> > > > > > >>> > > > > such a case to always create a
> > "Large" type
> > > > > >> > >> array.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically
> there
> > might be some
> > > > > >> > >> >> > > > > > >>> > > > > > performance
> > > > > >> > >> >> > > > > > >>> benefits on
> > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using
> the
> > native word
> > > > > >> > >> sizes.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
> > architectures don't do
> > > > > >> > >> that, as
> > > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > > > >> > >> >> > > > > > >>> is an
> > > > > >> > >> >> > > > > > >>> > > > > extremely common integer size even
> > in
> > > > > >> > >> high-performance
> > > > > >> > >> >> > > > code.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > Regards
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > > > >
> > > > > >> > >> >> > > > > > >>> > >
> > > > > >> > >> >> > > > > > >>>
> > > > > >> > >> >> > > > > > >>
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > >
> > > > > >> > >>
> > > > > >> > >
> >
>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
OK, I'm going to postpone cutting a release until tomorrow (hoping we can
issues resolved by then).. I'll also try to review the third-party
additions since 14.x.
On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
> I found a licensing issue
>
> https://issues.apache.org/jira/browse/ARROW-6679
>
> It might be worth examining third party code added to the project
> since 0.14.x to make sure there are no other such issues.
>
> On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > I have diagnosed the problem (Thrift "string" data must be UTF-8,
> > cannot be arbitrary binary) and am working on a patch right now
> >
> > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > I just opened
> > >
> > > https://issues.apache.org/jira/browse/ARROW-6678
> > >
> > > Please don't cut an RC until I have an opportunity to diagnose this,
> > > will report back.
> > >
> > >
> > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > >
> > > > I'm investigating a possible Parquet-related compatibility bug that I
> > > > encountered through some routine testing / benchmarking. I'll report
> > > > back once I figure out what is going on (if anything)
> > > >
> > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <
> emkornfield@gmail.com> wrote:
> > > > >>
> > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> get it
> > > > >> signed by another PMC member), but is not 100% essential.
> > > > >
> > > > > That won't be an option for me this week (it seems like I would
> need to meet one face-to-face). I'll try to get the GPG checked in and the
> rest of the pre-requisites done tomorrow (Monday) to hopefully start the
> release on Tuesday (hopefully we can solve the last blocker/integration
> tests by then).
> > > > >
> > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > > >>
> > > > >> It's ideal if your GPG key is in the web of trust (i.e. you can
> get it
> > > > >> signed by another PMC member), but is not 100% essential.
> > > > >>
> > > > >> Speaking of the release, there are at least 2 code changes I still
> > > > >> want to get in
> > > > >>
> > > > >> ARROW-5717
> > > > >> ARROW-6353
> > > > >>
> > > > >> I just pushed updates to ARROW-5717, will merge once the build is
> green.
> > > > >>
> > > > >> There are a couple of Rust patches still marked for 0.15. The rest
> > > > >> seems to be documentation and a couple of integration test
> failures we
> > > > >> should see about fixing in time.
> > > > >>
> > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <
> emkornfield@gmail.com> wrote:
> > > > >> >
> > > > >> > Thanks Krisztián and Wes,
> > > > >> > I've gone ahead and started registering myself on all the
> packaging sites.
> > > > >> >
> > > > >> > Is there any review process when adding my GPG key to the SVN
> file? [1]
> > > > >> > doesn't seem to mention explicitly.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Micah
> > > > >> >
> > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > > > >> >
> > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <
> wesmckinn@gmail.com> wrote:
> > > > >> > >
> > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> emkornfield@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >>
> > > > >> > >> >> The process should be well documented at this point but
> there are a
> > > > >> > >> >> number of steps.
> > > > >> > >> >
> > > > >> > >> > Is [1] the up-to-date documentation for the release? Are
> there
> > > > >> > >> instructions for the adding the code signing Key to SVN?
> > > > >> > >> >
> > > > >> > >> > I will make a go of it. i will try to mitigate any
> internet issues by
> > > > >> > >> doing the process for a cloud instance (I assume that isn't
> a problem?).
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >> Setting up a new cloud environment suitable for producing an
> RC may be
> > > > >> > >> time consuming, but you are welcome to try. Krisztian -- are
> you
> > > > >> > >> available next week to help Micah and potentially take over
> producing
> > > > >> > >> the RC if there are issues?
> > > > >> > >>
> > > > >> > > Sure, I'll be available next week. We can also grant access to
> > > > >> > > https://github.com/ursa-labs/crossbow because configuring all
> > > > >> > > the CI backends can be time consuming.
> > > > >> > >
> > > > >> > >>
> > > > >> > >> > Thanks,
> > > > >> > >> > Micah
> > > > >> > >> >
> > > > >> > >> > [1]
> > > > >> > >>
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > > >> > >> >
> > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <
> wesmckinn@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >>
> > > > >> > >> >> The process should be well documented at this point but
> there are a
> > > > >> > >> >> number of steps. Note that you need to add your code
> signing key to
> > > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I
> think it's fine
> > > > >> > >> >> to hand off the process to others after the VOTE but it
> would be
> > > > >> > >> >> tricky to have multiple RMs involved with producing the
> source and
> > > > >> > >> >> binary artifacts for the vote
> > > > >> > >> >>
> > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > > >> > >> emkornfield@gmail.com> wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > SGTM, as well.
> > > > >> > >> >> >
> > > > >> > >> >> > I should have a little bit of time next week if I can
> help as RM but
> > > > >> > >> I have
> > > > >> > >> >> > a couple of concerns:
> > > > >> > >> >> > 1. In the past I've had trouble downloading and
> validating
> > > > >> > >> releases. I'm a
> > > > >> > >> >> > bit worried, that I might have similar problems doing
> the necessary
> > > > >> > >> uploads.
> > > > >> > >> >> > 2. My internet connection will likely be not great, I
> don't know if
> > > > >> > >> this
> > > > >> > >> >> > would make it even less likely to be successful.
> > > > >> > >> >> >
> > > > >> > >> >> > Does it become problematic if somehow I would have to
> abandon the
> > > > >> > >> process
> > > > >> > >> >> > mid-release? Is there anyone who could serve as a
> backup? Are the
> > > > >> > >> steps
> > > > >> > >> >> > well documented?
> > > > >> > >> >> >
> > > > >> > >> >> > Thanks,
> > > > >> > >> >> > Micah
> > > > >> > >> >> >
> > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > > > >> > >> neal.p.richardson@gmail.com>
> > > > >> > >> >> > wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > > Sounds good to me.
> > > > >> > >> >> > >
> > > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > > > >> > >> >> > >
> > > > >> > >> >> > > Neal
> > > > >> > >> >> > >
> > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >> > >
> > > > >> > >> >> > > > hi all,
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > It looks like we're drawing close to be able to
> make the 0.15.0
> > > > >> > >> >> > > > release. I would suggest "pencils down" at the end
> of this week
> > > > >> > >> and
> > > > >> > >> >> > > > see if a release candidate can be produced next
> Monday September
> > > > >> > >> 23.
> > > > >> > >> >> > > > Any thoughts or objections?
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > Thanks,
> > > > >> > >> >> > > > Wes
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > > > >> > >> wesmckinn@gmail.com>
> > > > >> > >> >> > > wrote:
> > > > >> > >> >> > > > >
> > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to
> amend the
> > > > >> > >> Format docs
> > > > >> > >> >> > > > > today regarding the EOS issue and also update the
> C++ library
> > > > >> > >> >> > > > >
> > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > I assume the plan is to merge the
> > > > >> > >> ARROW-6313-flatbuffer-alignment
> > > > >> > >> >> > > > branch into master before the 0.15 release, correct?
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are
> ready to be
> > > > >> > >> merged into
> > > > >> > >> >> > > > the alignment branch -
> > > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > Eric
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > -----Original Message-----
> > > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> niki.lj@aliyun.com>
> > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > I should have a little more bandwidth to help
> with some of
> > > > >> > >> the
> > > > >> > >> >> > > > packaging starting tomorrow and going into the
> weekend.
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > > > >> > >> wesmckinn@gmail.com>
> > > > >> > >> >> > > > wrote:
> > > > >> > >> >> > > > > >
> > > > >> > >> >> > > > > > > Hi folks,
> > > > >> > >> >> > > > > > >
> > > > >> > >> >> > > > > > > With the state of nightly packaging and
> integration builds
> > > > >> > >> things
> > > > >> > >> >> > > > > > > aren't looking too good for being in release
> readiness by
> > > > >> > >> the end
> > > > >> > >> >> > > of
> > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning
> to be working
> > > > >> > >> to close
> > > > >> > >> >> > > as
> > > > >> > >> >> > > > > > > many issues as I can and also to help with
> the ongoing
> > > > >> > >> alignment
> > > > >> > >> >> > > > fixes.
> > > > >> > >> >> > > > > > >
> > > > >> > >> >> > > > > > > Wes
> > > > >> > >> >> > > > > > >
> > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield
> <
> > > > >> > >> >> > > emkornfield@gmail.com
> > > > >> > >> >> > > > >
> > > > >> > >> >> > > > > > > wrote:
> > > > >> > >> >> > > > > > >
> > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of
> the current
> > > > >> > >> issues:
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > >
> > > > >> > >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > >> > >> >> > > > > > >> ki.apache.org
> > > > >> > >> >> > > >
> %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%
> 40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > >
> > > > >> > >>
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > >
> > > > >> > >>
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > >> > >> >> > > > > > >> %3D&reserved=0
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> > > > >> > >> wesmckinn@gmail.com>
> > > > >> > >> >> > > > wrote:
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > > > > >>> hi all,
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in a
> position to
> > > > >> > >> release
> > > > >> > >> >> > > at
> > > > >> > >> >> > > > > > >>> the beginning of next week. I hope that one
> more week of
> > > > >> > >> work (or
> > > > >> > >> >> > > > > > >>> less) will be enough to get us there. Aside
> from merging
> > > > >> > >> the
> > > > >> > >> >> > > > > > >>> alignment changes, we need to make sure
> that our
> > > > >> > >> packaging jobs
> > > > >> > >> >> > > > > > >>> required for the release candidate are all
> working.
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > > > > >>> If folks could remove issues from the
> 0.15.0 backlog
> > > > >> > >> that they
> > > > >> > >> >> > > > don't
> > > > >> > >> >> > > > > > >>> think they will finish by end of next week
> that would
> > > > >> > >> help focus
> > > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in
> 0.15.0 still).
> > > > >> > >> I am
> > > > >> > >> >> > > > > > >>> looking to tackle a few small features
> related to
> > > > >> > >> dictionaries
> > > > >> > >> >> > > > while
> > > > >> > >> >> > > > > > >>> the release window is still open.
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > > > > >>> - Wes
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes
> McKinney <
> > > > >> > >> >> > > wesmckinn@gmail.com>
> > > > >> > >> >> > > > > > >>> wrote:
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > hi,
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > I think we should try to release the week
> of September
> > > > >> > >> 9, so
> > > > >> > >> >> > > > > > >>> > development work should be completed by
> end of next
> > > > >> > >> week.
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the protocol
> alignment
> > > > >> > >> changes for
> > > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I think
> that getting
> > > > >> > >> the
> > > > >> > >> >> > > > > > >>> > alignment work done is the main barrier
> to releasing.
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > Thanks
> > > > >> > >> >> > > > > > >>> > Wes
> > > > >> > >> >> > > > > > >>> >
> > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > >> > >> >> > > > > > >>> wrote:
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think
> of several
> > > > >> > >> bugs that
> > > > >> > >> >> > > > need
> > > > >> > >> >> > > > > > >>> > > to
> > > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are
> required in
> > > > >> > >> IPC streams
> > > > >> > >> >> > > > > > >>> > > even
> > > > >> > >> >> > > > > > >>> when empty[1]
> > > > >> > >> >> > > > > > >>> > > This one is under review now, however
> through this
> > > > >> > >> PR we find
> > > > >> > >> >> > > > > > >>> > > that
> > > > >> > >> >> > > > > > >>> there seems a bug in java reading and
> writing
> > > > >> > >> dictionaries in IPC
> > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it
> assumes all
> > > > >> > >> >> > > > dictionaries
> > > > >> > >> >> > > > > > >>> are at the start of stream (see details in
> PR comments,
> > > > >> > >> and this
> > > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15).
> @Micah Kornfield
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as
> strings in
> > > > >> > >> integration
> > > > >> > >> >> > > > test
> > > > >> > >> >> > > > > > >>> JSON files[3]
> > > > >> > >> >> > > > > > >>> > > Java side code already checked in, other
> > > > >> > >> implementations
> > > > >> > >> >> > > seems
> > > > >> > >> >> > > > not.
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in
> JdbcAdapter[4]
> > > > >> > >> Caused by
> > > > >> > >> >> > > trying
> > > > >> > >> >> > > > > > >>> > > to load all records in one contiguous
> batch, fixed
> > > > >> > >> >> > > > > > >>> by providing iterator API for iteratively
> reading in
> > > > >> > >> >> > > ARROW-6219[5].
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > Thanks,
> > > > >> > >> >> > > > > > >>> > > Ji Liu
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > [1]
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > >
> > > > >> > >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > >> > >> >> > > > > > >>> sues.apache.org
> > > > >> > >> >> > > >
> %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > >
> > > > >> > >>
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > >> > >> >> > > > > > >>>
> > > > >> > >>
> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > >
> ----------------------------------------------------------------
> > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <
> wesmckinn@gmail.com> Send
> > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > > > >> > >> dev@arrow.apache.org>
> > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing
> the 0.15.0
> > > > >> > >> backlog some
> > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help with
> grooming
> > > > >> > >> >> > > (particularly
> > > > >> > >> >> > > > > > >>> > > for languages other than C++/Python
> where I'm
> > > > >> > >> focusing) that
> > > > >> > >> >> > > > > > >>> > > would be helpful. There have been
> almost 500 JIRA
> > > > >> > >> issues
> > > > >> > >> >> > > opened
> > > > >> > >> >> > > > > > >>> > > since the
> > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure
> to check
> > > > >> > >> whether
> > > > >> > >> >> > > there's
> > > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs
> that we should
> > > > >> > >> try to
> > > > >> > >> >> > > fix
> > > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes
> McKinney
> > > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > > >> > >> >> > > > > > >>> wrote:
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1
> seems to be
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the
> Windows
> > > > >> > >> changes in
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a
> volunteer would look
> > > > >> > >> into what
> > > > >> > >> >> > > > > > >>> > > > was
> > > > >> > >> >> > > > > > >>> wrong
> > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows.
> Otherwise
> > > > >> > >> 0.15.0 Windows
> > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > >> > >> >> > > > > > >>> > > >
> F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > > >> > >> 40microsoft.com
> > > > >> > >> >> > > > %7Ccbea
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > >
> 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > >> > >> >> > > > > > >>> > > >
> zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > >> > >> >> > > > > > >>> > > >
> > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM
> Antoine Pitrou <
> > > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
> Micah
> > > > >> > >> Kornfield
> > > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > >> > >> >> > > > > > >>> > > > > > >
> > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > > >> > >> >> > > > > > >>> > > > > > > independent, we could have
> 32-bit array
> > > > >> > >> lengths and
> > > > >> > >> >> > > > > > >>> variable-length
> > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we
> wanted (we
> > > > >> > >> just
> > > > >> > >> >> > > > wouldn't
> > > > >> > >> >> > > > > > >>> > > > > > > be
> > > > >> > >> >> > > > > > >>> able to
> > > > >> > >> >> > > > > > >>> > > > > > > have a List child with more
> than INT32_MAX
> > > > >> > >> elements).
> > > > >> > >> >> > > > > > >>> > > > > >
> > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do
> this in C++
> > > > >> > >> but we
> > > > >> > >> >> > > > don't.
> > > > >> > >> >> > > > > > >>> I'm not sure we
> > > > >> > >> >> > > > > > >>> > > > > > would have introduced the "Large"
> types if we
> > > > >> > >> did.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much
> space as 32-bit
> > > > >> > >> >> > > offsets,
> > > > >> > >> >> > > > > > >>> > > > > so if
> > > > >> > >> >> > > > > > >>> you're
> > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or
> strings,
> > > > >> > >> 32-bit
> > > > >> > >> >> > > offsets
> > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with
> 64-bit array
> > > > >> > >> lengths from
> > > > >> > >> >> > > > the
> > > > >> > >> >> > > > > > >>> > > > > start
> > > > >> > >> >> > > > > > >>> it would
> > > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types
> with 32-bit
> > > > >> > >> offsets.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address
> space in Java
> > > > >> > >> and
> > > > >> > >> >> > > calling
> > > > >> > >> >> > > > > > >>> > > > > > it a
> > > > >> > >> >> > > > > > >>> reference
> > > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal.
> If a consumer
> > > > >> > >> uses a
> > > > >> > >> >> > > > "Large"
> > > > >> > >> >> > > > > > >>> type
> > > > >> > >> >> > > > > > >>> > > > > > presumably it is because they
> need the ability
> > > > >> > >> to store
> > > > >> > >> >> > > > > > >>> > > > > > more
> > > > >> > >> >> > > > > > >>> than INT32_MAX
> > > > >> > >> >> > > > > > >>> > > > > > child elements in a column,
> otherwise it is
> > > > >> > >> just
> > > > >> > >> >> > > wasting
> > > > >> > >> >> > > > > > >>> > > > > > space
> > > > >> > >> >> > > > > > >>> [1].
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual
> elements
> > > > >> > >> (lists or
> > > > >> > >> >> > > > > > >>> > > > > strings)
> > > > >> > >> >> > > > > > >>> are
> > > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> proportion,
> > > > >> > >> so it may
> > > > >> > >> >> > > be
> > > > >> > >> >> > > > > > >>> simpler in
> > > > >> > >> >> > > > > > >>> > > > > such a case to always create a
> "Large" type
> > > > >> > >> array.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there
> might be some
> > > > >> > >> >> > > > > > >>> > > > > > performance
> > > > >> > >> >> > > > > > >>> benefits on
> > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the
> native word
> > > > >> > >> sizes.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit
> architectures don't do
> > > > >> > >> that, as
> > > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > > >> > >> >> > > > > > >>> is an
> > > > >> > >> >> > > > > > >>> > > > > extremely common integer size even
> in
> > > > >> > >> high-performance
> > > > >> > >> >> > > > code.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > Regards
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > > > >
> > > > >> > >> >> > > > > > >>> > >
> > > > >> > >> >> > > > > > >>>
> > > > >> > >> >> > > > > > >>
> > > > >> > >> >> > > >
> > > > >> > >> >> > >
> > > > >> > >>
> > > > >> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
I found a licensing issue
https://issues.apache.org/jira/browse/ARROW-6679
It might be worth examining third party code added to the project
since 0.14.x to make sure there are no other such issues.
On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <we...@gmail.com> wrote:
>
> I have diagnosed the problem (Thrift "string" data must be UTF-8,
> cannot be arbitrary binary) and am working on a patch right now
>
> On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > I just opened
> >
> > https://issues.apache.org/jira/browse/ARROW-6678
> >
> > Please don't cut an RC until I have an opportunity to diagnose this,
> > will report back.
> >
> >
> > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > I'm investigating a possible Parquet-related compatibility bug that I
> > > encountered through some routine testing / benchmarking. I'll report
> > > back once I figure out what is going on (if anything)
> > >
> > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <em...@gmail.com> wrote:
> > > >>
> > > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> > > >> signed by another PMC member), but is not 100% essential.
> > > >
> > > > That won't be an option for me this week (it seems like I would need to meet one face-to-face). I'll try to get the GPG checked in and the rest of the pre-requisites done tomorrow (Monday) to hopefully start the release on Tuesday (hopefully we can solve the last blocker/integration tests by then).
> > > >
> > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
> > > >>
> > > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> > > >> signed by another PMC member), but is not 100% essential.
> > > >>
> > > >> Speaking of the release, there are at least 2 code changes I still
> > > >> want to get in
> > > >>
> > > >> ARROW-5717
> > > >> ARROW-6353
> > > >>
> > > >> I just pushed updates to ARROW-5717, will merge once the build is green.
> > > >>
> > > >> There are a couple of Rust patches still marked for 0.15. The rest
> > > >> seems to be documentation and a couple of integration test failures we
> > > >> should see about fixing in time.
> > > >>
> > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com> wrote:
> > > >> >
> > > >> > Thanks Krisztián and Wes,
> > > >> > I've gone ahead and started registering myself on all the packaging sites.
> > > >> >
> > > >> > Is there any review process when adding my GPG key to the SVN file? [1]
> > > >> > doesn't seem to mention explicitly.
> > > >> >
> > > >> > Thanks,
> > > >> > Micah
> > > >> >
> > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > > >> >
> > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
> > > >> > >
> > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >>
> > > >> > >> >> The process should be well documented at this point but there are a
> > > >> > >> >> number of steps.
> > > >> > >> >
> > > >> > >> > Is [1] the up-to-date documentation for the release? Are there
> > > >> > >> instructions for the adding the code signing Key to SVN?
> > > >> > >> >
> > > >> > >> > I will make a go of it. i will try to mitigate any internet issues by
> > > >> > >> doing the process for a cloud instance (I assume that isn't a problem?).
> > > >> > >> >
> > > >> > >>
> > > >> > >> Setting up a new cloud environment suitable for producing an RC may be
> > > >> > >> time consuming, but you are welcome to try. Krisztian -- are you
> > > >> > >> available next week to help Micah and potentially take over producing
> > > >> > >> the RC if there are issues?
> > > >> > >>
> > > >> > > Sure, I'll be available next week. We can also grant access to
> > > >> > > https://github.com/ursa-labs/crossbow because configuring all
> > > >> > > the CI backends can be time consuming.
> > > >> > >
> > > >> > >>
> > > >> > >> > Thanks,
> > > >> > >> > Micah
> > > >> > >> >
> > > >> > >> > [1]
> > > >> > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > > >> > >> >
> > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >>
> > > >> > >> >> The process should be well documented at this point but there are a
> > > >> > >> >> number of steps. Note that you need to add your code signing key to
> > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
> > > >> > >> >> to hand off the process to others after the VOTE but it would be
> > > >> > >> >> tricky to have multiple RMs involved with producing the source and
> > > >> > >> >> binary artifacts for the vote
> > > >> > >> >>
> > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > > >> > >> emkornfield@gmail.com> wrote:
> > > >> > >> >> >
> > > >> > >> >> > SGTM, as well.
> > > >> > >> >> >
> > > >> > >> >> > I should have a little bit of time next week if I can help as RM but
> > > >> > >> I have
> > > >> > >> >> > a couple of concerns:
> > > >> > >> >> > 1. In the past I've had trouble downloading and validating
> > > >> > >> releases. I'm a
> > > >> > >> >> > bit worried, that I might have similar problems doing the necessary
> > > >> > >> uploads.
> > > >> > >> >> > 2. My internet connection will likely be not great, I don't know if
> > > >> > >> this
> > > >> > >> >> > would make it even less likely to be successful.
> > > >> > >> >> >
> > > >> > >> >> > Does it become problematic if somehow I would have to abandon the
> > > >> > >> process
> > > >> > >> >> > mid-release? Is there anyone who could serve as a backup? Are the
> > > >> > >> steps
> > > >> > >> >> > well documented?
> > > >> > >> >> >
> > > >> > >> >> > Thanks,
> > > >> > >> >> > Micah
> > > >> > >> >> >
> > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > > >> > >> neal.p.richardson@gmail.com>
> > > >> > >> >> > wrote:
> > > >> > >> >> >
> > > >> > >> >> > > Sounds good to me.
> > > >> > >> >> > >
> > > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > > >> > >> >> > >
> > > >> > >> >> > > Neal
> > > >> > >> >> > >
> > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >> > >
> > > >> > >> >> > > > hi all,
> > > >> > >> >> > > >
> > > >> > >> >> > > > It looks like we're drawing close to be able to make the 0.15.0
> > > >> > >> >> > > > release. I would suggest "pencils down" at the end of this week
> > > >> > >> and
> > > >> > >> >> > > > see if a release candidate can be produced next Monday September
> > > >> > >> 23.
> > > >> > >> >> > > > Any thoughts or objections?
> > > >> > >> >> > > >
> > > >> > >> >> > > > Thanks,
> > > >> > >> >> > > > Wes
> > > >> > >> >> > > >
> > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > > >> > >> wesmckinn@gmail.com>
> > > >> > >> >> > > wrote:
> > > >> > >> >> > > > >
> > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> > > >> > >> Format docs
> > > >> > >> >> > > > > today regarding the EOS issue and also update the C++ library
> > > >> > >> >> > > > >
> > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > I assume the plan is to merge the
> > > >> > >> ARROW-6313-flatbuffer-alignment
> > > >> > >> >> > > > branch into master before the 0.15 release, correct?
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> > > >> > >> merged into
> > > >> > >> >> > > > the alignment branch -
> > > >> > >> https://github.com/apache/arrow/pull/5280/
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > Eric
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > -----Original Message-----
> > > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > I should have a little more bandwidth to help with some of
> > > >> > >> the
> > > >> > >> >> > > > packaging starting tomorrow and going into the weekend.
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > > >> > >> wesmckinn@gmail.com>
> > > >> > >> >> > > > wrote:
> > > >> > >> >> > > > > >
> > > >> > >> >> > > > > > > Hi folks,
> > > >> > >> >> > > > > > >
> > > >> > >> >> > > > > > > With the state of nightly packaging and integration builds
> > > >> > >> things
> > > >> > >> >> > > > > > > aren't looking too good for being in release readiness by
> > > >> > >> the end
> > > >> > >> >> > > of
> > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
> > > >> > >> to close
> > > >> > >> >> > > as
> > > >> > >> >> > > > > > > many issues as I can and also to help with the ongoing
> > > >> > >> alignment
> > > >> > >> >> > > > fixes.
> > > >> > >> >> > > > > > >
> > > >> > >> >> > > > > > > Wes
> > > >> > >> >> > > > > > >
> > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > > >> > >> >> > > emkornfield@gmail.com
> > > >> > >> >> > > > >
> > > >> > >> >> > > > > > > wrote:
> > > >> > >> >> > > > > > >
> > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> > > >> > >> issues:
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > >
> > > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > >> > >> >> > > > > > >> ki.apache.org
> > > >> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > >
> > > >> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > >
> > > >> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > >> > >> >> > > > > > >> %3D&reserved=0
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> > > >> > >> wesmckinn@gmail.com>
> > > >> > >> >> > > > wrote:
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > > > > >>> hi all,
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position to
> > > >> > >> release
> > > >> > >> >> > > at
> > > >> > >> >> > > > > > >>> the beginning of next week. I hope that one more week of
> > > >> > >> work (or
> > > >> > >> >> > > > > > >>> less) will be enough to get us there. Aside from merging
> > > >> > >> the
> > > >> > >> >> > > > > > >>> alignment changes, we need to make sure that our
> > > >> > >> packaging jobs
> > > >> > >> >> > > > > > >>> required for the release candidate are all working.
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> > > >> > >> that they
> > > >> > >> >> > > > don't
> > > >> > >> >> > > > > > >>> think they will finish by end of next week that would
> > > >> > >> help focus
> > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
> > > >> > >> I am
> > > >> > >> >> > > > > > >>> looking to tackle a few small features related to
> > > >> > >> dictionaries
> > > >> > >> >> > > > while
> > > >> > >> >> > > > > > >>> the release window is still open.
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > > > > >>> - Wes
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > > >> > >> >> > > wesmckinn@gmail.com>
> > > >> > >> >> > > > > > >>> wrote:
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > hi,
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > I think we should try to release the week of September
> > > >> > >> 9, so
> > > >> > >> >> > > > > > >>> > development work should be completed by end of next
> > > >> > >> week.
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> > > >> > >> changes for
> > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
> > > >> > >> the
> > > >> > >> >> > > > > > >>> > alignment work done is the main barrier to releasing.
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > Thanks
> > > >> > >> >> > > > > > >>> > Wes
> > > >> > >> >> > > > > > >>> >
> > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > >> > >> >> > > > > > >>> wrote:
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> > > >> > >> bugs that
> > > >> > >> >> > > > need
> > > >> > >> >> > > > > > >>> > > to
> > > >> > >> >> > > > > > >>> be fixed or reminded.
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> > > >> > >> IPC streams
> > > >> > >> >> > > > > > >>> > > even
> > > >> > >> >> > > > > > >>> when empty[1]
> > > >> > >> >> > > > > > >>> > > This one is under review now, however through this
> > > >> > >> PR we find
> > > >> > >> >> > > > > > >>> > > that
> > > >> > >> >> > > > > > >>> there seems a bug in java reading and writing
> > > >> > >> dictionaries in IPC
> > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> > > >> > >> >> > > > dictionaries
> > > >> > >> >> > > > > > >>> are at the start of stream (see details in PR comments,
> > > >> > >> and this
> > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> > > >> > >> integration
> > > >> > >> >> > > > test
> > > >> > >> >> > > > > > >>> JSON files[3]
> > > >> > >> >> > > > > > >>> > > Java side code already checked in, other
> > > >> > >> implementations
> > > >> > >> >> > > seems
> > > >> > >> >> > > > not.
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > > >> > >> Caused by
> > > >> > >> >> > > trying
> > > >> > >> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
> > > >> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
> > > >> > >> >> > > ARROW-6219[5].
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > Thanks,
> > > >> > >> >> > > > > > >>> > > Ji Liu
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > [1]
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > >> > >> >> > > > > > >>> > > d=0 [3]
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > >> > >> >> > > > > > >>> > > &reserved=0]
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > >
> > > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > >> > >> >> > > > > > >>> sues.apache.org
> > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > >
> > > >> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > >> > >> >> > > > > > >>>
> > > >> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > ----------------------------------------------------------------
> > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > > >> > >> dev@arrow.apache.org>
> > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> > > >> > >> backlog some
> > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> > > >> > >> >> > > (particularly
> > > >> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> > > >> > >> focusing) that
> > > >> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> > > >> > >> issues
> > > >> > >> >> > > opened
> > > >> > >> >> > > > > > >>> > > since the
> > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> > > >> > >> whether
> > > >> > >> >> > > there's
> > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs that we should
> > > >> > >> try to
> > > >> > >> >> > > fix
> > > >> > >> >> > > > > > >>> > > for 0.15.0.
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > > >> > >> >> > > > > > >>> wrote:
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
> > > >> > >> changes in
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
> > > >> > >> into what
> > > >> > >> >> > > > > > >>> > > > was
> > > >> > >> >> > > > > > >>> wrong
> > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> > > >> > >> 0.15.0 Windows
> > > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > >> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > > >> > >> 40microsoft.com
> > > >> > >> >> > > > %7Ccbea
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > >> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > >> > >> >> > > > > > >>> > > >
> > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> > > >> > >> Kornfield
> > > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > >> > >> >> > > > > > >>> > > > > > >
> > > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > > >> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> > > >> > >> lengths and
> > > >> > >> >> > > > > > >>> variable-length
> > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
> > > >> > >> just
> > > >> > >> >> > > > wouldn't
> > > >> > >> >> > > > > > >>> > > > > > > be
> > > >> > >> >> > > > > > >>> able to
> > > >> > >> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> > > >> > >> elements).
> > > >> > >> >> > > > > > >>> > > > > >
> > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do this in C++
> > > >> > >> but we
> > > >> > >> >> > > > don't.
> > > >> > >> >> > > > > > >>> I'm not sure we
> > > >> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
> > > >> > >> did.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> > > >> > >> >> > > offsets,
> > > >> > >> >> > > > > > >>> > > > > so if
> > > >> > >> >> > > > > > >>> you're
> > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> > > >> > >> 32-bit
> > > >> > >> >> > > offsets
> > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> > > >> > >> lengths from
> > > >> > >> >> > > > the
> > > >> > >> >> > > > > > >>> > > > > start
> > > >> > >> >> > > > > > >>> it would
> > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> > > >> > >> offsets.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > > Going with the limited address space in Java
> > > >> > >> and
> > > >> > >> >> > > calling
> > > >> > >> >> > > > > > >>> > > > > > it a
> > > >> > >> >> > > > > > >>> reference
> > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> > > >> > >> uses a
> > > >> > >> >> > > > "Large"
> > > >> > >> >> > > > > > >>> type
> > > >> > >> >> > > > > > >>> > > > > > presumably it is because they need the ability
> > > >> > >> to store
> > > >> > >> >> > > > > > >>> > > > > > more
> > > >> > >> >> > > > > > >>> than INT32_MAX
> > > >> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> > > >> > >> just
> > > >> > >> >> > > wasting
> > > >> > >> >> > > > > > >>> > > > > > space
> > > >> > >> >> > > > > > >>> [1].
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> > > >> > >> (lists or
> > > >> > >> >> > > > > > >>> > > > > strings)
> > > >> > >> >> > > > > > >>> are
> > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
> > > >> > >> so it may
> > > >> > >> >> > > be
> > > >> > >> >> > > > > > >>> simpler in
> > > >> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> > > >> > >> array.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> > > >> > >> >> > > > > > >>> > > > > > performance
> > > >> > >> >> > > > > > >>> benefits on
> > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
> > > >> > >> sizes.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> > > >> > >> that, as
> > > >> > >> >> > > > > > >>> > > > > 32-bit
> > > >> > >> >> > > > > > >>> is an
> > > >> > >> >> > > > > > >>> > > > > extremely common integer size even in
> > > >> > >> high-performance
> > > >> > >> >> > > > code.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > Regards
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > > Antoine.
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > > > >
> > > >> > >> >> > > > > > >>> > >
> > > >> > >> >> > > > > > >>>
> > > >> > >> >> > > > > > >>
> > > >> > >> >> > > >
> > > >> > >> >> > >
> > > >> > >>
> > > >> > >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
I have diagnosed the problem (Thrift "string" data must be UTF-8,
cannot be arbitrary binary) and am working on a patch right now
On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <we...@gmail.com> wrote:
>
> I just opened
>
> https://issues.apache.org/jira/browse/ARROW-6678
>
> Please don't cut an RC until I have an opportunity to diagnose this,
> will report back.
>
>
> On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > I'm investigating a possible Parquet-related compatibility bug that I
> > encountered through some routine testing / benchmarking. I'll report
> > back once I figure out what is going on (if anything)
> >
> > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <em...@gmail.com> wrote:
> > >>
> > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> > >> signed by another PMC member), but is not 100% essential.
> > >
> > > That won't be an option for me this week (it seems like I would need to meet one face-to-face). I'll try to get the GPG checked in and the rest of the pre-requisites done tomorrow (Monday) to hopefully start the release on Tuesday (hopefully we can solve the last blocker/integration tests by then).
> > >
> > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
> > >>
> > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> > >> signed by another PMC member), but is not 100% essential.
> > >>
> > >> Speaking of the release, there are at least 2 code changes I still
> > >> want to get in
> > >>
> > >> ARROW-5717
> > >> ARROW-6353
> > >>
> > >> I just pushed updates to ARROW-5717, will merge once the build is green.
> > >>
> > >> There are a couple of Rust patches still marked for 0.15. The rest
> > >> seems to be documentation and a couple of integration test failures we
> > >> should see about fixing in time.
> > >>
> > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com> wrote:
> > >> >
> > >> > Thanks Krisztián and Wes,
> > >> > I've gone ahead and started registering myself on all the packaging sites.
> > >> >
> > >> > Is there any review process when adding my GPG key to the SVN file? [1]
> > >> > doesn't seem to mention explicitly.
> > >> >
> > >> > Thanks,
> > >> > Micah
> > >> >
> > >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> > >> >
> > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
> > >> > >
> > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
> > >> > >> wrote:
> > >> > >> >>
> > >> > >> >> The process should be well documented at this point but there are a
> > >> > >> >> number of steps.
> > >> > >> >
> > >> > >> > Is [1] the up-to-date documentation for the release? Are there
> > >> > >> instructions for the adding the code signing Key to SVN?
> > >> > >> >
> > >> > >> > I will make a go of it. i will try to mitigate any internet issues by
> > >> > >> doing the process for a cloud instance (I assume that isn't a problem?).
> > >> > >> >
> > >> > >>
> > >> > >> Setting up a new cloud environment suitable for producing an RC may be
> > >> > >> time consuming, but you are welcome to try. Krisztian -- are you
> > >> > >> available next week to help Micah and potentially take over producing
> > >> > >> the RC if there are issues?
> > >> > >>
> > >> > > Sure, I'll be available next week. We can also grant access to
> > >> > > https://github.com/ursa-labs/crossbow because configuring all
> > >> > > the CI backends can be time consuming.
> > >> > >
> > >> > >>
> > >> > >> > Thanks,
> > >> > >> > Micah
> > >> > >> >
> > >> > >> > [1]
> > >> > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > >> > >> >
> > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> > >> > >> wrote:
> > >> > >> >>
> > >> > >> >> The process should be well documented at this point but there are a
> > >> > >> >> number of steps. Note that you need to add your code signing key to
> > >> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
> > >> > >> >> to hand off the process to others after the VOTE but it would be
> > >> > >> >> tricky to have multiple RMs involved with producing the source and
> > >> > >> >> binary artifacts for the vote
> > >> > >> >>
> > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > >> > >> emkornfield@gmail.com> wrote:
> > >> > >> >> >
> > >> > >> >> > SGTM, as well.
> > >> > >> >> >
> > >> > >> >> > I should have a little bit of time next week if I can help as RM but
> > >> > >> I have
> > >> > >> >> > a couple of concerns:
> > >> > >> >> > 1. In the past I've had trouble downloading and validating
> > >> > >> releases. I'm a
> > >> > >> >> > bit worried, that I might have similar problems doing the necessary
> > >> > >> uploads.
> > >> > >> >> > 2. My internet connection will likely be not great, I don't know if
> > >> > >> this
> > >> > >> >> > would make it even less likely to be successful.
> > >> > >> >> >
> > >> > >> >> > Does it become problematic if somehow I would have to abandon the
> > >> > >> process
> > >> > >> >> > mid-release? Is there anyone who could serve as a backup? Are the
> > >> > >> steps
> > >> > >> >> > well documented?
> > >> > >> >> >
> > >> > >> >> > Thanks,
> > >> > >> >> > Micah
> > >> > >> >> >
> > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > >> > >> neal.p.richardson@gmail.com>
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > Sounds good to me.
> > >> > >> >> > >
> > >> > >> >> > > Do we have a release manager yet? Any volunteers?
> > >> > >> >> > >
> > >> > >> >> > > Neal
> > >> > >> >> > >
> > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> > >> > >> wrote:
> > >> > >> >> > >
> > >> > >> >> > > > hi all,
> > >> > >> >> > > >
> > >> > >> >> > > > It looks like we're drawing close to be able to make the 0.15.0
> > >> > >> >> > > > release. I would suggest "pencils down" at the end of this week
> > >> > >> and
> > >> > >> >> > > > see if a release candidate can be produced next Monday September
> > >> > >> 23.
> > >> > >> >> > > > Any thoughts or objections?
> > >> > >> >> > > >
> > >> > >> >> > > > Thanks,
> > >> > >> >> > > > Wes
> > >> > >> >> > > >
> > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > >> > >> wesmckinn@gmail.com>
> > >> > >> >> > > wrote:
> > >> > >> >> > > > >
> > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> > >> > >> Format docs
> > >> > >> >> > > > > today regarding the EOS issue and also update the C++ library
> > >> > >> >> > > > >
> > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > >> > >> >> > > > > <Er...@microsoft.com> wrote:
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > I assume the plan is to merge the
> > >> > >> ARROW-6313-flatbuffer-alignment
> > >> > >> >> > > > branch into master before the 0.15 release, correct?
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> > >> > >> merged into
> > >> > >> >> > > > the alignment branch -
> > >> > >> https://github.com/apache/arrow/pull/5280/
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > Eric
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > -----Original Message-----
> > >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > I should have a little more bandwidth to help with some of
> > >> > >> the
> > >> > >> >> > > > packaging starting tomorrow and going into the weekend.
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > >> > >> wesmckinn@gmail.com>
> > >> > >> >> > > > wrote:
> > >> > >> >> > > > > >
> > >> > >> >> > > > > > > Hi folks,
> > >> > >> >> > > > > > >
> > >> > >> >> > > > > > > With the state of nightly packaging and integration builds
> > >> > >> things
> > >> > >> >> > > > > > > aren't looking too good for being in release readiness by
> > >> > >> the end
> > >> > >> >> > > of
> > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
> > >> > >> to close
> > >> > >> >> > > as
> > >> > >> >> > > > > > > many issues as I can and also to help with the ongoing
> > >> > >> alignment
> > >> > >> >> > > > fixes.
> > >> > >> >> > > > > > >
> > >> > >> >> > > > > > > Wes
> > >> > >> >> > > > > > >
> > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > >> > >> >> > > emkornfield@gmail.com
> > >> > >> >> > > > >
> > >> > >> >> > > > > > > wrote:
> > >> > >> >> > > > > > >
> > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> > >> > >> issues:
> > >> > >> >> > > > > > >>
> > >> > >> >> > > > > > >>
> > >> > >> >> > > >
> > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > >> > >> >> > > > > > >> ki.apache.org
> > >> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034
> > >> > >> >> > > > > > >>
> > >> > >> >> > > >
> > >> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> > >> >> > > > > > >>
> > >> > >> >> > > >
> > >> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > >> > >> >> > > > > > >> %3D&reserved=0
> > >> > >> >> > > > > > >>
> > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> > >> > >> wesmckinn@gmail.com>
> > >> > >> >> > > > wrote:
> > >> > >> >> > > > > > >>
> > >> > >> >> > > > > > >>> hi all,
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position to
> > >> > >> release
> > >> > >> >> > > at
> > >> > >> >> > > > > > >>> the beginning of next week. I hope that one more week of
> > >> > >> work (or
> > >> > >> >> > > > > > >>> less) will be enough to get us there. Aside from merging
> > >> > >> the
> > >> > >> >> > > > > > >>> alignment changes, we need to make sure that our
> > >> > >> packaging jobs
> > >> > >> >> > > > > > >>> required for the release candidate are all working.
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> > >> > >> that they
> > >> > >> >> > > > don't
> > >> > >> >> > > > > > >>> think they will finish by end of next week that would
> > >> > >> help focus
> > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
> > >> > >> I am
> > >> > >> >> > > > > > >>> looking to tackle a few small features related to
> > >> > >> dictionaries
> > >> > >> >> > > > while
> > >> > >> >> > > > > > >>> the release window is still open.
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > > > > >>> - Wes
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > >> > >> >> > > wesmckinn@gmail.com>
> > >> > >> >> > > > > > >>> wrote:
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > hi,
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > I think we should try to release the week of September
> > >> > >> 9, so
> > >> > >> >> > > > > > >>> > development work should be completed by end of next
> > >> > >> week.
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > Does that seem reasonable?
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> > >> > >> changes for
> > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
> > >> > >> the
> > >> > >> >> > > > > > >>> > alignment work done is the main barrier to releasing.
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > Thanks
> > >> > >> >> > > > > > >>> > Wes
> > >> > >> >> > > > > > >>> >
> > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > >> > >> >> > > > > > >>> wrote:
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> > >> > >> bugs that
> > >> > >> >> > > > need
> > >> > >> >> > > > > > >>> > > to
> > >> > >> >> > > > > > >>> be fixed or reminded.
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> > >> > >> IPC streams
> > >> > >> >> > > > > > >>> > > even
> > >> > >> >> > > > > > >>> when empty[1]
> > >> > >> >> > > > > > >>> > > This one is under review now, however through this
> > >> > >> PR we find
> > >> > >> >> > > > > > >>> > > that
> > >> > >> >> > > > > > >>> there seems a bug in java reading and writing
> > >> > >> dictionaries in IPC
> > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> > >> > >> >> > > > dictionaries
> > >> > >> >> > > > > > >>> are at the start of stream (see details in PR comments,
> > >> > >> and this
> > >> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> > >> > >> integration
> > >> > >> >> > > > test
> > >> > >> >> > > > > > >>> JSON files[3]
> > >> > >> >> > > > > > >>> > > Java side code already checked in, other
> > >> > >> implementations
> > >> > >> >> > > seems
> > >> > >> >> > > > not.
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > >> > >> Caused by
> > >> > >> >> > > trying
> > >> > >> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
> > >> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
> > >> > >> >> > > ARROW-6219[5].
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > Thanks,
> > >> > >> >> > > > > > >>> > > Ji Liu
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > [1]
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > >> > >> >> > > > > > >>> > > reserved=0 [2]
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > >> > >> >> > > > > > >>> > > d=0 [3]
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > >> > >> >> > > > > > >>> > > &reserved=0]
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > >
> > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > >> > >> >> > > > > > >>> sues.apache.org
> > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > >
> > >> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > >> > >> >> > > > > > >>>
> > >> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > ----------------------------------------------------------------
> > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > >> > >> dev@arrow.apache.org>
> > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> > >> > >> backlog some
> > >> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> > >> > >> >> > > (particularly
> > >> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> > >> > >> focusing) that
> > >> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> > >> > >> issues
> > >> > >> >> > > opened
> > >> > >> >> > > > > > >>> > > since the
> > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> > >> > >> whether
> > >> > >> >> > > there's
> > >> > >> >> > > > > > >>> > > any regressions or other serious bugs that we should
> > >> > >> try to
> > >> > >> >> > > fix
> > >> > >> >> > > > > > >>> > > for 0.15.0.
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > >> > >> >> > > > > > >>> > > <we...@gmail.com>
> > >> > >> >> > > > > > >>> wrote:
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
> > >> > >> changes in
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
> > >> > >> into what
> > >> > >> >> > > > > > >>> > > > was
> > >> > >> >> > > > > > >>> wrong
> > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> > >> > >> 0.15.0 Windows
> > >> > >> >> > > > > > >>> > > > wheels will be broken, too
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > >> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > >> > >> 40microsoft.com
> > >> > >> >> > > > %7Ccbea
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > >> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > >> > >> >> > > > > > >>> > > >
> > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> > >> > >> Kornfield
> > >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > >> > >> >> > > > > > >>> > > > > > >
> > >> > >> >> > > > > > >>> > > > > > > In C++ they are
> > >> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> > >> > >> lengths and
> > >> > >> >> > > > > > >>> variable-length
> > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
> > >> > >> just
> > >> > >> >> > > > wouldn't
> > >> > >> >> > > > > > >>> > > > > > > be
> > >> > >> >> > > > > > >>> able to
> > >> > >> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> > >> > >> elements).
> > >> > >> >> > > > > > >>> > > > > >
> > >> > >> >> > > > > > >>> > > > > > I think the point is we could do this in C++
> > >> > >> but we
> > >> > >> >> > > > don't.
> > >> > >> >> > > > > > >>> I'm not sure we
> > >> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
> > >> > >> did.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> > >> > >> >> > > offsets,
> > >> > >> >> > > > > > >>> > > > > so if
> > >> > >> >> > > > > > >>> you're
> > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> > >> > >> 32-bit
> > >> > >> >> > > offsets
> > >> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> > >> > >> lengths from
> > >> > >> >> > > > the
> > >> > >> >> > > > > > >>> > > > > start
> > >> > >> >> > > > > > >>> it would
> > >> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> > >> > >> offsets.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > > Going with the limited address space in Java
> > >> > >> and
> > >> > >> >> > > calling
> > >> > >> >> > > > > > >>> > > > > > it a
> > >> > >> >> > > > > > >>> reference
> > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> > >> > >> uses a
> > >> > >> >> > > > "Large"
> > >> > >> >> > > > > > >>> type
> > >> > >> >> > > > > > >>> > > > > > presumably it is because they need the ability
> > >> > >> to store
> > >> > >> >> > > > > > >>> > > > > > more
> > >> > >> >> > > > > > >>> than INT32_MAX
> > >> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> > >> > >> just
> > >> > >> >> > > wasting
> > >> > >> >> > > > > > >>> > > > > > space
> > >> > >> >> > > > > > >>> [1].
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> > >> > >> (lists or
> > >> > >> >> > > > > > >>> > > > > strings)
> > >> > >> >> > > > > > >>> are
> > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
> > >> > >> so it may
> > >> > >> >> > > be
> > >> > >> >> > > > > > >>> simpler in
> > >> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> > >> > >> array.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> > >> > >> >> > > > > > >>> > > > > > performance
> > >> > >> >> > > > > > >>> benefits on
> > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
> > >> > >> sizes.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> > >> > >> that, as
> > >> > >> >> > > > > > >>> > > > > 32-bit
> > >> > >> >> > > > > > >>> is an
> > >> > >> >> > > > > > >>> > > > > extremely common integer size even in
> > >> > >> high-performance
> > >> > >> >> > > > code.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > Regards
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > > Antoine.
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > > > >
> > >> > >> >> > > > > > >>> > >
> > >> > >> >> > > > > > >>>
> > >> > >> >> > > > > > >>
> > >> > >> >> > > >
> > >> > >> >> > >
> > >> > >>
> > >> > >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
I just opened
https://issues.apache.org/jira/browse/ARROW-6678
Please don't cut an RC until I have an opportunity to diagnose this,
will report back.
On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <we...@gmail.com> wrote:
>
> I'm investigating a possible Parquet-related compatibility bug that I
> encountered through some routine testing / benchmarking. I'll report
> back once I figure out what is going on (if anything)
>
> On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <em...@gmail.com> wrote:
> >>
> >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> >> signed by another PMC member), but is not 100% essential.
> >
> > That won't be an option for me this week (it seems like I would need to meet one face-to-face). I'll try to get the GPG checked in and the rest of the pre-requisites done tomorrow (Monday) to hopefully start the release on Tuesday (hopefully we can solve the last blocker/integration tests by then).
> >
> > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> >> signed by another PMC member), but is not 100% essential.
> >>
> >> Speaking of the release, there are at least 2 code changes I still
> >> want to get in
> >>
> >> ARROW-5717
> >> ARROW-6353
> >>
> >> I just pushed updates to ARROW-5717, will merge once the build is green.
> >>
> >> There are a couple of Rust patches still marked for 0.15. The rest
> >> seems to be documentation and a couple of integration test failures we
> >> should see about fixing in time.
> >>
> >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com> wrote:
> >> >
> >> > Thanks Krisztián and Wes,
> >> > I've gone ahead and started registering myself on all the packaging sites.
> >> >
> >> > Is there any review process when adding my GPG key to the SVN file? [1]
> >> > doesn't seem to mention explicitly.
> >> >
> >> > Thanks,
> >> > Micah
> >> >
> >> > [1] https://www.apache.org/dev/version-control.html#https-svn
> >> >
> >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
> >> > wrote:
> >> >
> >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
> >> > >
> >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
> >> > >> wrote:
> >> > >> >>
> >> > >> >> The process should be well documented at this point but there are a
> >> > >> >> number of steps.
> >> > >> >
> >> > >> > Is [1] the up-to-date documentation for the release? Are there
> >> > >> instructions for the adding the code signing Key to SVN?
> >> > >> >
> >> > >> > I will make a go of it. i will try to mitigate any internet issues by
> >> > >> doing the process for a cloud instance (I assume that isn't a problem?).
> >> > >> >
> >> > >>
> >> > >> Setting up a new cloud environment suitable for producing an RC may be
> >> > >> time consuming, but you are welcome to try. Krisztian -- are you
> >> > >> available next week to help Micah and potentially take over producing
> >> > >> the RC if there are issues?
> >> > >>
> >> > > Sure, I'll be available next week. We can also grant access to
> >> > > https://github.com/ursa-labs/crossbow because configuring all
> >> > > the CI backends can be time consuming.
> >> > >
> >> > >>
> >> > >> > Thanks,
> >> > >> > Micah
> >> > >> >
> >> > >> > [1]
> >> > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> >> > >> >
> >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> >> > >> wrote:
> >> > >> >>
> >> > >> >> The process should be well documented at this point but there are a
> >> > >> >> number of steps. Note that you need to add your code signing key to
> >> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
> >> > >> >> to hand off the process to others after the VOTE but it would be
> >> > >> >> tricky to have multiple RMs involved with producing the source and
> >> > >> >> binary artifacts for the vote
> >> > >> >>
> >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> >> > >> emkornfield@gmail.com> wrote:
> >> > >> >> >
> >> > >> >> > SGTM, as well.
> >> > >> >> >
> >> > >> >> > I should have a little bit of time next week if I can help as RM but
> >> > >> I have
> >> > >> >> > a couple of concerns:
> >> > >> >> > 1. In the past I've had trouble downloading and validating
> >> > >> releases. I'm a
> >> > >> >> > bit worried, that I might have similar problems doing the necessary
> >> > >> uploads.
> >> > >> >> > 2. My internet connection will likely be not great, I don't know if
> >> > >> this
> >> > >> >> > would make it even less likely to be successful.
> >> > >> >> >
> >> > >> >> > Does it become problematic if somehow I would have to abandon the
> >> > >> process
> >> > >> >> > mid-release? Is there anyone who could serve as a backup? Are the
> >> > >> steps
> >> > >> >> > well documented?
> >> > >> >> >
> >> > >> >> > Thanks,
> >> > >> >> > Micah
> >> > >> >> >
> >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> >> > >> neal.p.richardson@gmail.com>
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Sounds good to me.
> >> > >> >> > >
> >> > >> >> > > Do we have a release manager yet? Any volunteers?
> >> > >> >> > >
> >> > >> >> > > Neal
> >> > >> >> > >
> >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> >> > >> wrote:
> >> > >> >> > >
> >> > >> >> > > > hi all,
> >> > >> >> > > >
> >> > >> >> > > > It looks like we're drawing close to be able to make the 0.15.0
> >> > >> >> > > > release. I would suggest "pencils down" at the end of this week
> >> > >> and
> >> > >> >> > > > see if a release candidate can be produced next Monday September
> >> > >> 23.
> >> > >> >> > > > Any thoughts or objections?
> >> > >> >> > > >
> >> > >> >> > > > Thanks,
> >> > >> >> > > > Wes
> >> > >> >> > > >
> >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> >> > >> wesmckinn@gmail.com>
> >> > >> >> > > wrote:
> >> > >> >> > > > >
> >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> >> > >> Format docs
> >> > >> >> > > > > today regarding the EOS issue and also update the C++ library
> >> > >> >> > > > >
> >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> >> > >> >> > > > > <Er...@microsoft.com> wrote:
> >> > >> >> > > > > >
> >> > >> >> > > > > > I assume the plan is to merge the
> >> > >> ARROW-6313-flatbuffer-alignment
> >> > >> >> > > > branch into master before the 0.15 release, correct?
> >> > >> >> > > > > >
> >> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> >> > >> merged into
> >> > >> >> > > > the alignment branch -
> >> > >> https://github.com/apache/arrow/pull/5280/
> >> > >> >> > > > > >
> >> > >> >> > > > > > Eric
> >> > >> >> > > > > >
> >> > >> >> > > > > > -----Original Message-----
> >> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> >> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> >> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> >> > >> >> > > > > >
> >> > >> >> > > > > > I should have a little more bandwidth to help with some of
> >> > >> the
> >> > >> >> > > > packaging starting tomorrow and going into the weekend.
> >> > >> >> > > > > >
> >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> >> > >> wesmckinn@gmail.com>
> >> > >> >> > > > wrote:
> >> > >> >> > > > > >
> >> > >> >> > > > > > > Hi folks,
> >> > >> >> > > > > > >
> >> > >> >> > > > > > > With the state of nightly packaging and integration builds
> >> > >> things
> >> > >> >> > > > > > > aren't looking too good for being in release readiness by
> >> > >> the end
> >> > >> >> > > of
> >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
> >> > >> to close
> >> > >> >> > > as
> >> > >> >> > > > > > > many issues as I can and also to help with the ongoing
> >> > >> alignment
> >> > >> >> > > > fixes.
> >> > >> >> > > > > > >
> >> > >> >> > > > > > > Wes
> >> > >> >> > > > > > >
> >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> >> > >> >> > > emkornfield@gmail.com
> >> > >> >> > > > >
> >> > >> >> > > > > > > wrote:
> >> > >> >> > > > > > >
> >> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> >> > >> issues:
> >> > >> >> > > > > > >>
> >> > >> >> > > > > > >>
> >> > >> >> > > >
> >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >> > >> >> > > > > > >> ki.apache.org
> >> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034
> >> > >> >> > > > > > >>
> >> > >> >> > > >
> >> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> > >> >> > > > > > >>
> >> > >> >> > > >
> >> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >> > >> >> > > > > > >> %3D&reserved=0
> >> > >> >> > > > > > >>
> >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> >> > >> wesmckinn@gmail.com>
> >> > >> >> > > > wrote:
> >> > >> >> > > > > > >>
> >> > >> >> > > > > > >>> hi all,
> >> > >> >> > > > > > >>>
> >> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position to
> >> > >> release
> >> > >> >> > > at
> >> > >> >> > > > > > >>> the beginning of next week. I hope that one more week of
> >> > >> work (or
> >> > >> >> > > > > > >>> less) will be enough to get us there. Aside from merging
> >> > >> the
> >> > >> >> > > > > > >>> alignment changes, we need to make sure that our
> >> > >> packaging jobs
> >> > >> >> > > > > > >>> required for the release candidate are all working.
> >> > >> >> > > > > > >>>
> >> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> >> > >> that they
> >> > >> >> > > > don't
> >> > >> >> > > > > > >>> think they will finish by end of next week that would
> >> > >> help focus
> >> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
> >> > >> I am
> >> > >> >> > > > > > >>> looking to tackle a few small features related to
> >> > >> dictionaries
> >> > >> >> > > > while
> >> > >> >> > > > > > >>> the release window is still open.
> >> > >> >> > > > > > >>>
> >> > >> >> > > > > > >>> - Wes
> >> > >> >> > > > > > >>>
> >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> >> > >> >> > > wesmckinn@gmail.com>
> >> > >> >> > > > > > >>> wrote:
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > hi,
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > I think we should try to release the week of September
> >> > >> 9, so
> >> > >> >> > > > > > >>> > development work should be completed by end of next
> >> > >> week.
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > Does that seem reasonable?
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> >> > >> changes for
> >> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
> >> > >> the
> >> > >> >> > > > > > >>> > alignment work done is the main barrier to releasing.
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > Thanks
> >> > >> >> > > > > > >>> > Wes
> >> > >> >> > > > > > >>> >
> >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> >> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> >> > >> >> > > > > > >>> wrote:
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> >> > >> bugs that
> >> > >> >> > > > need
> >> > >> >> > > > > > >>> > > to
> >> > >> >> > > > > > >>> be fixed or reminded.
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> >> > >> IPC streams
> >> > >> >> > > > > > >>> > > even
> >> > >> >> > > > > > >>> when empty[1]
> >> > >> >> > > > > > >>> > > This one is under review now, however through this
> >> > >> PR we find
> >> > >> >> > > > > > >>> > > that
> >> > >> >> > > > > > >>> there seems a bug in java reading and writing
> >> > >> dictionaries in IPC
> >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> >> > >> >> > > > dictionaries
> >> > >> >> > > > > > >>> are at the start of stream (see details in PR comments,
> >> > >> and this
> >> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> >> > >> integration
> >> > >> >> > > > test
> >> > >> >> > > > > > >>> JSON files[3]
> >> > >> >> > > > > > >>> > > Java side code already checked in, other
> >> > >> implementations
> >> > >> >> > > seems
> >> > >> >> > > > not.
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> >> > >> Caused by
> >> > >> >> > > trying
> >> > >> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
> >> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
> >> > >> >> > > ARROW-6219[5].
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > Thanks,
> >> > >> >> > > > > > >>> > > Ji Liu
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > [1]
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >> > >> >> > > > > > >>> > > reserved=0 [2]
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > >> >> > > > > > >>> > > 2Farrow.apache.org
> >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >> > >> >> > > > > > >>> > > ardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >> > >> >> > > > > > >>> > > d=0 [3]
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >> > >> >> > > > > > >>> > > ;reserved=0 [4]
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > >> >> > > > > > >>> > > 2Fissues.apache.org
> >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d73
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >> > >> >> > > > > > >>> > > &reserved=0]
> >> > >> >> > > > > > >>>
> >> > >> >> > > >
> >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >> > >> >> > > > > > >>> sues.apache.org
> >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> > >> >> > > > > > >>>
> >> > >> >> > > >
> >> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >> > >> >> > > > > > >>>
> >> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > ----------------------------------------------------------------
> >> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> >> > >> dev@arrow.apache.org>
> >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> >> > >> backlog some
> >> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> >> > >> >> > > (particularly
> >> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> >> > >> focusing) that
> >> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> >> > >> issues
> >> > >> >> > > opened
> >> > >> >> > > > > > >>> > > since the
> >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> >> > >> whether
> >> > >> >> > > there's
> >> > >> >> > > > > > >>> > > any regressions or other serious bugs that we should
> >> > >> try to
> >> > >> >> > > fix
> >> > >> >> > > > > > >>> > > for 0.15.0.
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> >> > >> >> > > > > > >>> > > <we...@gmail.com>
> >> > >> >> > > > > > >>> wrote:
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
> >> > >> changes in
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
> >> > >> into what
> >> > >> >> > > > > > >>> > > > was
> >> > >> >> > > > > > >>> wrong
> >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> >> > >> 0.15.0 Windows
> >> > >> >> > > > > > >>> > > > wheels will be broken, too
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > > The bad wheels can be found at
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> >> > >> 40microsoft.com
> >> > >> >> > > > %7Ccbea
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >> > >> >> > > > > > >>> > > >
> >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> >> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> >> > >> Kornfield
> >> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> >> > >> >> > > > > > >>> > > > > > >
> >> > >> >> > > > > > >>> > > > > > > In C++ they are
> >> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> >> > >> lengths and
> >> > >> >> > > > > > >>> variable-length
> >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
> >> > >> just
> >> > >> >> > > > wouldn't
> >> > >> >> > > > > > >>> > > > > > > be
> >> > >> >> > > > > > >>> able to
> >> > >> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> >> > >> elements).
> >> > >> >> > > > > > >>> > > > > >
> >> > >> >> > > > > > >>> > > > > > I think the point is we could do this in C++
> >> > >> but we
> >> > >> >> > > > don't.
> >> > >> >> > > > > > >>> I'm not sure we
> >> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
> >> > >> did.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> >> > >> >> > > offsets,
> >> > >> >> > > > > > >>> > > > > so if
> >> > >> >> > > > > > >>> you're
> >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> >> > >> 32-bit
> >> > >> >> > > offsets
> >> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> >> > >> lengths from
> >> > >> >> > > > the
> >> > >> >> > > > > > >>> > > > > start
> >> > >> >> > > > > > >>> it would
> >> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> >> > >> offsets.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > > Going with the limited address space in Java
> >> > >> and
> >> > >> >> > > calling
> >> > >> >> > > > > > >>> > > > > > it a
> >> > >> >> > > > > > >>> reference
> >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> >> > >> uses a
> >> > >> >> > > > "Large"
> >> > >> >> > > > > > >>> type
> >> > >> >> > > > > > >>> > > > > > presumably it is because they need the ability
> >> > >> to store
> >> > >> >> > > > > > >>> > > > > > more
> >> > >> >> > > > > > >>> than INT32_MAX
> >> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> >> > >> just
> >> > >> >> > > wasting
> >> > >> >> > > > > > >>> > > > > > space
> >> > >> >> > > > > > >>> [1].
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> >> > >> (lists or
> >> > >> >> > > > > > >>> > > > > strings)
> >> > >> >> > > > > > >>> are
> >> > >> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
> >> > >> so it may
> >> > >> >> > > be
> >> > >> >> > > > > > >>> simpler in
> >> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> >> > >> array.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> >> > >> >> > > > > > >>> > > > > > performance
> >> > >> >> > > > > > >>> benefits on
> >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
> >> > >> sizes.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> >> > >> that, as
> >> > >> >> > > > > > >>> > > > > 32-bit
> >> > >> >> > > > > > >>> is an
> >> > >> >> > > > > > >>> > > > > extremely common integer size even in
> >> > >> high-performance
> >> > >> >> > > > code.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > Regards
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > > Antoine.
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > > > >
> >> > >> >> > > > > > >>> > >
> >> > >> >> > > > > > >>>
> >> > >> >> > > > > > >>
> >> > >> >> > > >
> >> > >> >> > >
> >> > >>
> >> > >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
I'm investigating a possible Parquet-related compatibility bug that I
encountered through some routine testing / benchmarking. I'll report
back once I figure out what is going on (if anything)
On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <em...@gmail.com> wrote:
>>
>> It's ideal if your GPG key is in the web of trust (i.e. you can get it
>> signed by another PMC member), but is not 100% essential.
>
> That won't be an option for me this week (it seems like I would need to meet one face-to-face). I'll try to get the GPG checked in and the rest of the pre-requisites done tomorrow (Monday) to hopefully start the release on Tuesday (hopefully we can solve the last blocker/integration tests by then).
>
> On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> It's ideal if your GPG key is in the web of trust (i.e. you can get it
>> signed by another PMC member), but is not 100% essential.
>>
>> Speaking of the release, there are at least 2 code changes I still
>> want to get in
>>
>> ARROW-5717
>> ARROW-6353
>>
>> I just pushed updates to ARROW-5717, will merge once the build is green.
>>
>> There are a couple of Rust patches still marked for 0.15. The rest
>> seems to be documentation and a couple of integration test failures we
>> should see about fixing in time.
>>
>> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com> wrote:
>> >
>> > Thanks Krisztián and Wes,
>> > I've gone ahead and started registering myself on all the packaging sites.
>> >
>> > Is there any review process when adding my GPG key to the SVN file? [1]
>> > doesn't seem to mention explicitly.
>> >
>> > Thanks,
>> > Micah
>> >
>> > [1] https://www.apache.org/dev/version-control.html#https-svn
>> >
>> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
>> > wrote:
>> >
>> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
>> > >
>> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
>> > >> wrote:
>> > >> >>
>> > >> >> The process should be well documented at this point but there are a
>> > >> >> number of steps.
>> > >> >
>> > >> > Is [1] the up-to-date documentation for the release? Are there
>> > >> instructions for the adding the code signing Key to SVN?
>> > >> >
>> > >> > I will make a go of it. i will try to mitigate any internet issues by
>> > >> doing the process for a cloud instance (I assume that isn't a problem?).
>> > >> >
>> > >>
>> > >> Setting up a new cloud environment suitable for producing an RC may be
>> > >> time consuming, but you are welcome to try. Krisztian -- are you
>> > >> available next week to help Micah and potentially take over producing
>> > >> the RC if there are issues?
>> > >>
>> > > Sure, I'll be available next week. We can also grant access to
>> > > https://github.com/ursa-labs/crossbow because configuring all
>> > > the CI backends can be time consuming.
>> > >
>> > >>
>> > >> > Thanks,
>> > >> > Micah
>> > >> >
>> > >> > [1]
>> > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>> > >> >
>> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
>> > >> wrote:
>> > >> >>
>> > >> >> The process should be well documented at this point but there are a
>> > >> >> number of steps. Note that you need to add your code signing key to
>> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
>> > >> >> to hand off the process to others after the VOTE but it would be
>> > >> >> tricky to have multiple RMs involved with producing the source and
>> > >> >> binary artifacts for the vote
>> > >> >>
>> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
>> > >> emkornfield@gmail.com> wrote:
>> > >> >> >
>> > >> >> > SGTM, as well.
>> > >> >> >
>> > >> >> > I should have a little bit of time next week if I can help as RM but
>> > >> I have
>> > >> >> > a couple of concerns:
>> > >> >> > 1. In the past I've had trouble downloading and validating
>> > >> releases. I'm a
>> > >> >> > bit worried, that I might have similar problems doing the necessary
>> > >> uploads.
>> > >> >> > 2. My internet connection will likely be not great, I don't know if
>> > >> this
>> > >> >> > would make it even less likely to be successful.
>> > >> >> >
>> > >> >> > Does it become problematic if somehow I would have to abandon the
>> > >> process
>> > >> >> > mid-release? Is there anyone who could serve as a backup? Are the
>> > >> steps
>> > >> >> > well documented?
>> > >> >> >
>> > >> >> > Thanks,
>> > >> >> > Micah
>> > >> >> >
>> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
>> > >> neal.p.richardson@gmail.com>
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > Sounds good to me.
>> > >> >> > >
>> > >> >> > > Do we have a release manager yet? Any volunteers?
>> > >> >> > >
>> > >> >> > > Neal
>> > >> >> > >
>> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
>> > >> wrote:
>> > >> >> > >
>> > >> >> > > > hi all,
>> > >> >> > > >
>> > >> >> > > > It looks like we're drawing close to be able to make the 0.15.0
>> > >> >> > > > release. I would suggest "pencils down" at the end of this week
>> > >> and
>> > >> >> > > > see if a release candidate can be produced next Monday September
>> > >> 23.
>> > >> >> > > > Any thoughts or objections?
>> > >> >> > > >
>> > >> >> > > > Thanks,
>> > >> >> > > > Wes
>> > >> >> > > >
>> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
>> > >> wesmckinn@gmail.com>
>> > >> >> > > wrote:
>> > >> >> > > > >
>> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
>> > >> Format docs
>> > >> >> > > > > today regarding the EOS issue and also update the C++ library
>> > >> >> > > > >
>> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
>> > >> >> > > > > <Er...@microsoft.com> wrote:
>> > >> >> > > > > >
>> > >> >> > > > > > I assume the plan is to merge the
>> > >> ARROW-6313-flatbuffer-alignment
>> > >> >> > > > branch into master before the 0.15 release, correct?
>> > >> >> > > > > >
>> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
>> > >> merged into
>> > >> >> > > > the alignment branch -
>> > >> https://github.com/apache/arrow/pull/5280/
>> > >> >> > > > > >
>> > >> >> > > > > > Eric
>> > >> >> > > > > >
>> > >> >> > > > > > -----Original Message-----
>> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
>> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
>> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
>> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
>> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
>> > >> >> > > > > >
>> > >> >> > > > > > I should have a little more bandwidth to help with some of
>> > >> the
>> > >> >> > > > packaging starting tomorrow and going into the weekend.
>> > >> >> > > > > >
>> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
>> > >> wesmckinn@gmail.com>
>> > >> >> > > > wrote:
>> > >> >> > > > > >
>> > >> >> > > > > > > Hi folks,
>> > >> >> > > > > > >
>> > >> >> > > > > > > With the state of nightly packaging and integration builds
>> > >> things
>> > >> >> > > > > > > aren't looking too good for being in release readiness by
>> > >> the end
>> > >> >> > > of
>> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
>> > >> to close
>> > >> >> > > as
>> > >> >> > > > > > > many issues as I can and also to help with the ongoing
>> > >> alignment
>> > >> >> > > > fixes.
>> > >> >> > > > > > >
>> > >> >> > > > > > > Wes
>> > >> >> > > > > > >
>> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
>> > >> >> > > emkornfield@gmail.com
>> > >> >> > > > >
>> > >> >> > > > > > > wrote:
>> > >> >> > > > > > >
>> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
>> > >> issues:
>> > >> >> > > > > > >>
>> > >> >> > > > > > >>
>> > >> >> > > >
>> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> > >> >> > > > > > >> ki.apache.org
>> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034
>> > >> >> > > > > > >>
>> > >> >> > > >
>> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > >> >> > > > > > >>
>> > >> >> > > >
>> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> > >> >> > > > > > >> %3D&reserved=0
>> > >> >> > > > > > >>
>> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
>> > >> wesmckinn@gmail.com>
>> > >> >> > > > wrote:
>> > >> >> > > > > > >>
>> > >> >> > > > > > >>> hi all,
>> > >> >> > > > > > >>>
>> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position to
>> > >> release
>> > >> >> > > at
>> > >> >> > > > > > >>> the beginning of next week. I hope that one more week of
>> > >> work (or
>> > >> >> > > > > > >>> less) will be enough to get us there. Aside from merging
>> > >> the
>> > >> >> > > > > > >>> alignment changes, we need to make sure that our
>> > >> packaging jobs
>> > >> >> > > > > > >>> required for the release candidate are all working.
>> > >> >> > > > > > >>>
>> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
>> > >> that they
>> > >> >> > > > don't
>> > >> >> > > > > > >>> think they will finish by end of next week that would
>> > >> help focus
>> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
>> > >> I am
>> > >> >> > > > > > >>> looking to tackle a few small features related to
>> > >> dictionaries
>> > >> >> > > > while
>> > >> >> > > > > > >>> the release window is still open.
>> > >> >> > > > > > >>>
>> > >> >> > > > > > >>> - Wes
>> > >> >> > > > > > >>>
>> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
>> > >> >> > > wesmckinn@gmail.com>
>> > >> >> > > > > > >>> wrote:
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > hi,
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > I think we should try to release the week of September
>> > >> 9, so
>> > >> >> > > > > > >>> > development work should be completed by end of next
>> > >> week.
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > Does that seem reasonable?
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
>> > >> changes for
>> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
>> > >> the
>> > >> >> > > > > > >>> > alignment work done is the main barrier to releasing.
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > Thanks
>> > >> >> > > > > > >>> > Wes
>> > >> >> > > > > > >>> >
>> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
>> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
>> > >> >> > > > > > >>> wrote:
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
>> > >> bugs that
>> > >> >> > > > need
>> > >> >> > > > > > >>> > > to
>> > >> >> > > > > > >>> be fixed or reminded.
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
>> > >> IPC streams
>> > >> >> > > > > > >>> > > even
>> > >> >> > > > > > >>> when empty[1]
>> > >> >> > > > > > >>> > > This one is under review now, however through this
>> > >> PR we find
>> > >> >> > > > > > >>> > > that
>> > >> >> > > > > > >>> there seems a bug in java reading and writing
>> > >> dictionaries in IPC
>> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
>> > >> >> > > > dictionaries
>> > >> >> > > > > > >>> are at the start of stream (see details in PR comments,
>> > >> and this
>> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
>> > >> integration
>> > >> >> > > > test
>> > >> >> > > > > > >>> JSON files[3]
>> > >> >> > > > > > >>> > > Java side code already checked in, other
>> > >> implementations
>> > >> >> > > seems
>> > >> >> > > > not.
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
>> > >> Caused by
>> > >> >> > > trying
>> > >> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
>> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
>> > >> >> > > ARROW-6219[5].
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > Thanks,
>> > >> >> > > > > > >>> > > Ji Liu
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > [1]
>> > >> >> > > > > > >>> > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >> >> > > > > > >>> > >
>> > >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>> > >> >> > > > > > >>> > >
>> > >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>> > >> >> > > > > > >>> > >
>> > >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>> > >> >> > > > > > >>> > > reserved=0 [2]
>> > >> >> > > > > > >>> > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >> >> > > > > > >>> > > 2Farrow.apache.org
>> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>> > >> >> > > > > > >>> > > ardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > >> >> > > > > > >>> > >
>> > >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>> > >> >> > > > > > >>> > >
>> > >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>> > >> >> > > > > > >>> > > d=0 [3]
>> > >> >> > > > > > >>> > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d736678
>> > >> >> > > > > > >>> > >
>> > >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>> > >> >> > > > > > >>> > >
>> > >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>> > >> >> > > > > > >>> > > ;reserved=0 [4]
>> > >> >> > > > > > >>> > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > >> >> > > > > > >>> > > 2Fissues.apache.org
>> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d73
>> > >> >> > > > > > >>> > >
>> > >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>> > >> >> > > > > > >>> > >
>> > >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>> > >> >> > > > > > >>> > > &reserved=0]
>> > >> >> > > > > > >>>
>> > >> >> > > >
>> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>> > >> >> > > > > > >>> sues.apache.org
>> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>> > >> >> > > > > > >>> .Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > >> >> > > > > > >>>
>> > >> >> > > >
>> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>> > >> >> > > > > > >>>
>> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > >
>> > >> >> > > > ----------------------------------------------------------------
>> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
>> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
>> > >> dev@arrow.apache.org>
>> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
>> > >> backlog some
>> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
>> > >> >> > > (particularly
>> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
>> > >> focusing) that
>> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
>> > >> issues
>> > >> >> > > opened
>> > >> >> > > > > > >>> > > since the
>> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
>> > >> whether
>> > >> >> > > there's
>> > >> >> > > > > > >>> > > any regressions or other serious bugs that we should
>> > >> try to
>> > >> >> > > fix
>> > >> >> > > > > > >>> > > for 0.15.0.
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
>> > >> >> > > > > > >>> > > <we...@gmail.com>
>> > >> >> > > > > > >>> wrote:
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
>> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
>> > >> changes in
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
>> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>> > >> >> > > > > > >>> > > > bs%3D&reserved=0
>> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
>> > >> into what
>> > >> >> > > > > > >>> > > > was
>> > >> >> > > > > > >>> wrong
>> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
>> > >> 0.15.0 Windows
>> > >> >> > > > > > >>> > > > wheels will be broken, too
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > > The bad wheels can be found at
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>> > >> 40microsoft.com
>> > >> >> > > > %7Ccbea
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>> > >> >> > > > > > >>> > > >
>> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
>> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
>> > >> Kornfield
>> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
>> > >> >> > > > > > >>> > > > > > >
>> > >> >> > > > > > >>> > > > > > > In C++ they are
>> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
>> > >> lengths and
>> > >> >> > > > > > >>> variable-length
>> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
>> > >> just
>> > >> >> > > > wouldn't
>> > >> >> > > > > > >>> > > > > > > be
>> > >> >> > > > > > >>> able to
>> > >> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
>> > >> elements).
>> > >> >> > > > > > >>> > > > > >
>> > >> >> > > > > > >>> > > > > > I think the point is we could do this in C++
>> > >> but we
>> > >> >> > > > don't.
>> > >> >> > > > > > >>> I'm not sure we
>> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
>> > >> did.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
>> > >> >> > > offsets,
>> > >> >> > > > > > >>> > > > > so if
>> > >> >> > > > > > >>> you're
>> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
>> > >> 32-bit
>> > >> >> > > offsets
>> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
>> > >> lengths from
>> > >> >> > > > the
>> > >> >> > > > > > >>> > > > > start
>> > >> >> > > > > > >>> it would
>> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
>> > >> offsets.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > > Going with the limited address space in Java
>> > >> and
>> > >> >> > > calling
>> > >> >> > > > > > >>> > > > > > it a
>> > >> >> > > > > > >>> reference
>> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
>> > >> uses a
>> > >> >> > > > "Large"
>> > >> >> > > > > > >>> type
>> > >> >> > > > > > >>> > > > > > presumably it is because they need the ability
>> > >> to store
>> > >> >> > > > > > >>> > > > > > more
>> > >> >> > > > > > >>> than INT32_MAX
>> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
>> > >> just
>> > >> >> > > wasting
>> > >> >> > > > > > >>> > > > > > space
>> > >> >> > > > > > >>> [1].
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
>> > >> (lists or
>> > >> >> > > > > > >>> > > > > strings)
>> > >> >> > > > > > >>> are
>> > >> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
>> > >> so it may
>> > >> >> > > be
>> > >> >> > > > > > >>> simpler in
>> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
>> > >> array.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
>> > >> >> > > > > > >>> > > > > > performance
>> > >> >> > > > > > >>> benefits on
>> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
>> > >> sizes.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
>> > >> that, as
>> > >> >> > > > > > >>> > > > > 32-bit
>> > >> >> > > > > > >>> is an
>> > >> >> > > > > > >>> > > > > extremely common integer size even in
>> > >> high-performance
>> > >> >> > > > code.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > Regards
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > > Antoine.
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > > > >
>> > >> >> > > > > > >>> > >
>> > >> >> > > > > > >>>
>> > >> >> > > > > > >>
>> > >> >> > > >
>> > >> >> > >
>> > >>
>> > >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
>
> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> signed by another PMC member), but is not 100% essential.
That won't be an option for me this week (it seems like I would need to
meet one face-to-face). I'll try to get the GPG checked in and the rest of
the pre-requisites done tomorrow (Monday) to hopefully start the release on
Tuesday (hopefully we can solve the last blocker/integration tests by then).
On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> signed by another PMC member), but is not 100% essential.
>
> Speaking of the release, there are at least 2 code changes I still
> want to get in
>
> ARROW-5717
> ARROW-6353
>
> I just pushed updates to ARROW-5717, will merge once the build is green.
>
> There are a couple of Rust patches still marked for 0.15. The rest
> seems to be documentation and a couple of integration test failures we
> should see about fixing in time.
>
> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > Thanks Krisztián and Wes,
> > I've gone ahead and started registering myself on all the packaging
> sites.
> >
> > Is there any review process when adding my GPG key to the SVN file? [1]
> > doesn't seem to mention explicitly.
> >
> > Thanks,
> > Micah
> >
> > [1] https://www.apache.org/dev/version-control.html#https-svn
> >
> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com>
> > wrote:
> >
> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> emkornfield@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> The process should be well documented at this point but there are a
> > >> >> number of steps.
> > >> >
> > >> > Is [1] the up-to-date documentation for the release? Are there
> > >> instructions for the adding the code signing Key to SVN?
> > >> >
> > >> > I will make a go of it. i will try to mitigate any internet issues
> by
> > >> doing the process for a cloud instance (I assume that isn't a
> problem?).
> > >> >
> > >>
> > >> Setting up a new cloud environment suitable for producing an RC may be
> > >> time consuming, but you are welcome to try. Krisztian -- are you
> > >> available next week to help Micah and potentially take over producing
> > >> the RC if there are issues?
> > >>
> > > Sure, I'll be available next week. We can also grant access to
> > > https://github.com/ursa-labs/crossbow because configuring all
> > > the CI backends can be time consuming.
> > >
> > >>
> > >> > Thanks,
> > >> > Micah
> > >> >
> > >> > [1]
> > >>
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > >> >
> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> The process should be well documented at this point but there are a
> > >> >> number of steps. Note that you need to add your code signing key to
> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's
> fine
> > >> >> to hand off the process to others after the VOTE but it would be
> > >> >> tricky to have multiple RMs involved with producing the source and
> > >> >> binary artifacts for the vote
> > >> >>
> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > >> emkornfield@gmail.com> wrote:
> > >> >> >
> > >> >> > SGTM, as well.
> > >> >> >
> > >> >> > I should have a little bit of time next week if I can help as RM
> but
> > >> I have
> > >> >> > a couple of concerns:
> > >> >> > 1. In the past I've had trouble downloading and validating
> > >> releases. I'm a
> > >> >> > bit worried, that I might have similar problems doing the
> necessary
> > >> uploads.
> > >> >> > 2. My internet connection will likely be not great, I don't
> know if
> > >> this
> > >> >> > would make it even less likely to be successful.
> > >> >> >
> > >> >> > Does it become problematic if somehow I would have to abandon the
> > >> process
> > >> >> > mid-release? Is there anyone who could serve as a backup? Are
> the
> > >> steps
> > >> >> > well documented?
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Micah
> > >> >> >
> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > >> neal.p.richardson@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Sounds good to me.
> > >> >> > >
> > >> >> > > Do we have a release manager yet? Any volunteers?
> > >> >> > >
> > >> >> > > Neal
> > >> >> > >
> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> wesmckinn@gmail.com>
> > >> wrote:
> > >> >> > >
> > >> >> > > > hi all,
> > >> >> > > >
> > >> >> > > > It looks like we're drawing close to be able to make the
> 0.15.0
> > >> >> > > > release. I would suggest "pencils down" at the end of this
> week
> > >> and
> > >> >> > > > see if a release candidate can be produced next Monday
> September
> > >> 23.
> > >> >> > > > Any thoughts or objections?
> > >> >> > > >
> > >> >> > > > Thanks,
> > >> >> > > > Wes
> > >> >> > > >
> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > wrote:
> > >> >> > > > >
> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> > >> Format docs
> > >> >> > > > > today regarding the EOS issue and also update the C++
> library
> > >> >> > > > >
> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > >> >> > > > > <Er...@microsoft.com> wrote:
> > >> >> > > > > >
> > >> >> > > > > > I assume the plan is to merge the
> > >> ARROW-6313-flatbuffer-alignment
> > >> >> > > > branch into master before the 0.15 release, correct?
> > >> >> > > > > >
> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> > >> merged into
> > >> >> > > > the alignment branch -
> > >> https://github.com/apache/arrow/pull/5280/
> > >> >> > > > > >
> > >> >> > > > > > Eric
> > >> >> > > > > >
> > >> >> > > > > > -----Original Message-----
> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> niki.lj@aliyun.com>
> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > >> >> > > > > >
> > >> >> > > > > > I should have a little more bandwidth to help with some
> of
> > >> the
> > >> >> > > > packaging starting tomorrow and going into the weekend.
> > >> >> > > > > >
> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > > wrote:
> > >> >> > > > > >
> > >> >> > > > > > > Hi folks,
> > >> >> > > > > > >
> > >> >> > > > > > > With the state of nightly packaging and integration
> builds
> > >> things
> > >> >> > > > > > > aren't looking too good for being in release readiness
> by
> > >> the end
> > >> >> > > of
> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be
> working
> > >> to close
> > >> >> > > as
> > >> >> > > > > > > many issues as I can and also to help with the ongoing
> > >> alignment
> > >> >> > > > fixes.
> > >> >> > > > > > >
> > >> >> > > > > > > Wes
> > >> >> > > > > > >
> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > >> >> > > emkornfield@gmail.com
> > >> >> > > > >
> > >> >> > > > > > > wrote:
> > >> >> > > > > > >
> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> > >> issues:
> > >> >> > > > > > >>
> > >> >> > > > > > >>
> > >> >> > > >
> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > >> >> > > > > > >> ki.apache.org
> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034
> > >> >> > > > > > >>
> > >> >> > > >
> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> >> > > > > > >>
> > >> >> > > >
> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > >> >> > > > > > >> %3D&reserved=0
> > >> >> > > > > > >>
> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > > wrote:
> > >> >> > > > > > >>
> > >> >> > > > > > >>> hi all,
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position
> to
> > >> release
> > >> >> > > at
> > >> >> > > > > > >>> the beginning of next week. I hope that one more
> week of
> > >> work (or
> > >> >> > > > > > >>> less) will be enough to get us there. Aside from
> merging
> > >> the
> > >> >> > > > > > >>> alignment changes, we need to make sure that our
> > >> packaging jobs
> > >> >> > > > > > >>> required for the release candidate are all working.
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> > >> that they
> > >> >> > > > don't
> > >> >> > > > > > >>> think they will finish by end of next week that would
> > >> help focus
> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0
> still).
> > >> I am
> > >> >> > > > > > >>> looking to tackle a few small features related to
> > >> dictionaries
> > >> >> > > > while
> > >> >> > > > > > >>> the release window is still open.
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> - Wes
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > >> >> > > wesmckinn@gmail.com>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > hi,
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > I think we should try to release the week of
> September
> > >> 9, so
> > >> >> > > > > > >>> > development work should be completed by end of next
> > >> week.
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > Does that seem reasonable?
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> > >> changes for
> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that
> getting
> > >> the
> > >> >> > > > > > >>> > alignment work done is the main barrier to
> releasing.
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > Thanks
> > >> >> > > > > > >>> > Wes
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> > >> bugs that
> > >> >> > > > need
> > >> >> > > > > > >>> > > to
> > >> >> > > > > > >>> be fixed or reminded.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> > >> IPC streams
> > >> >> > > > > > >>> > > even
> > >> >> > > > > > >>> when empty[1]
> > >> >> > > > > > >>> > > This one is under review now, however through
> this
> > >> PR we find
> > >> >> > > > > > >>> > > that
> > >> >> > > > > > >>> there seems a bug in java reading and writing
> > >> dictionaries in IPC
> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes
> all
> > >> >> > > > dictionaries
> > >> >> > > > > > >>> are at the start of stream (see details in PR
> comments,
> > >> and this
> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah
> Kornfield
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> > >> integration
> > >> >> > > > test
> > >> >> > > > > > >>> JSON files[3]
> > >> >> > > > > > >>> > > Java side code already checked in, other
> > >> implementations
> > >> >> > > seems
> > >> >> > > > not.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > >> Caused by
> > >> >> > > trying
> > >> >> > > > > > >>> > > to load all records in one contiguous batch,
> fixed
> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
> > >> >> > > ARROW-6219[5].
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > Thanks,
> > >> >> > > > > > >>> > > Ji Liu
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > [1]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > >> >> > > > > > >>> > >
> > >> >> > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > >> >> > > > > > >>> > >
> > >> >> > > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > >> >> > > > > > >>> > > reserved=0 [2]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> >> > > > > > >>> > >
> > >> >> > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > >> >> > > > > > >>> > >
> > >> >> > > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > >> >> > > > > > >>> > > d=0 [3]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > >> >> > > > > > >>> > >
> > >> >> > > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > >> >> > > > > > >>> > > &reserved=0]
> > >> >> > > > > > >>>
> > >> >> > > >
> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > >> >> > > > > > >>> sues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> >> > > > > > >>>
> > >> >> > > >
> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > >> >> > > > > > >>>
> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > >
> ----------------------------------------------------------------
> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > >> dev@arrow.apache.org>
> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> > >> backlog some
> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> > >> >> > > (particularly
> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> > >> focusing) that
> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> > >> issues
> > >> >> > > opened
> > >> >> > > > > > >>> > > since the
> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> > >> whether
> > >> >> > > there's
> > >> >> > > > > > >>> > > any regressions or other serious bugs that we
> should
> > >> try to
> > >> >> > > fix
> > >> >> > > > > > >>> > > for 0.15.0.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > >> >> > > > > > >>> > > <we...@gmail.com>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
> > >> changes in
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would
> look
> > >> into what
> > >> >> > > > > > >>> > > > was
> > >> >> > > > > > >>> wrong
> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> > >> 0.15.0 Windows
> > >> >> > > > > > >>> > > > wheels will be broken, too
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > >> 40microsoft.com
> > >> >> > > > %7Ccbea
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou
> <
> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> > >> Kornfield
> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > >> >> > > > > > >>> > > > > > >
> > >> >> > > > > > >>> > > > > > > In C++ they are
> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> > >> lengths and
> > >> >> > > > > > >>> variable-length
> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted
> (we
> > >> just
> > >> >> > > > wouldn't
> > >> >> > > > > > >>> > > > > > > be
> > >> >> > > > > > >>> able to
> > >> >> > > > > > >>> > > > > > > have a List child with more than
> INT32_MAX
> > >> elements).
> > >> >> > > > > > >>> > > > > >
> > >> >> > > > > > >>> > > > > > I think the point is we could do this in
> C++
> > >> but we
> > >> >> > > > don't.
> > >> >> > > > > > >>> I'm not sure we
> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if
> we
> > >> did.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as
> 32-bit
> > >> >> > > offsets,
> > >> >> > > > > > >>> > > > > so if
> > >> >> > > > > > >>> you're
> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> > >> 32-bit
> > >> >> > > offsets
> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> > >> lengths from
> > >> >> > > > the
> > >> >> > > > > > >>> > > > > start
> > >> >> > > > > > >>> it would
> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> > >> offsets.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > > Going with the limited address space in
> Java
> > >> and
> > >> >> > > calling
> > >> >> > > > > > >>> > > > > > it a
> > >> >> > > > > > >>> reference
> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a
> consumer
> > >> uses a
> > >> >> > > > "Large"
> > >> >> > > > > > >>> type
> > >> >> > > > > > >>> > > > > > presumably it is because they need the
> ability
> > >> to store
> > >> >> > > > > > >>> > > > > > more
> > >> >> > > > > > >>> than INT32_MAX
> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> > >> just
> > >> >> > > wasting
> > >> >> > > > > > >>> > > > > > space
> > >> >> > > > > > >>> [1].
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> > >> (lists or
> > >> >> > > > > > >>> > > > > strings)
> > >> >> > > > > > >>> are
> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> proportion,
> > >> so it may
> > >> >> > > be
> > >> >> > > > > > >>> simpler in
> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> > >> array.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be
> some
> > >> >> > > > > > >>> > > > > > performance
> > >> >> > > > > > >>> benefits on
> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native
> word
> > >> sizes.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures
> don't do
> > >> that, as
> > >> >> > > > > > >>> > > > > 32-bit
> > >> >> > > > > > >>> is an
> > >> >> > > > > > >>> > > > > extremely common integer size even in
> > >> high-performance
> > >> >> > > > code.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Regards
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Antoine.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>>
> > >> >> > > > > > >>
> > >> >> > > >
> > >> >> > >
> > >>
> > >
>
Re: Timeline for 0.15.0 release
Posted by Andy Grove <an...@gmail.com>.
There's been quite a bit of activity in DataFusion over the past few weeks,
and there are currently two issues that I would like to see merged in time
for the release:
ARROW-6660: Minor docs update
ARROW-6089: Physical query plan for selection operator
ARROW-6089 is part of the new query execution implementation that I talked
about on the mailing list just recently and is new functionality that
doesn't impact any existing users, so maybe this one could be rubber stamp
approved if there are no objections. The equivalent PRs for the other
operators (projection and aggregate) were already merged.
On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <we...@gmail.com> wrote:
> It's ideal if your GPG key is in the web of trust (i.e. you can get it
> signed by another PMC member), but is not 100% essential.
>
> Speaking of the release, there are at least 2 code changes I still
> want to get in
>
> ARROW-5717
> ARROW-6353
>
> I just pushed updates to ARROW-5717, will merge once the build is green.
>
> There are a couple of Rust patches still marked for 0.15. The rest
> seems to be documentation and a couple of integration test failures we
> should see about fixing in time.
>
> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > Thanks Krisztián and Wes,
> > I've gone ahead and started registering myself on all the packaging
> sites.
> >
> > Is there any review process when adding my GPG key to the SVN file? [1]
> > doesn't seem to mention explicitly.
> >
> > Thanks,
> > Micah
> >
> > [1] https://www.apache.org/dev/version-control.html#https-svn
> >
> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com>
> > wrote:
> >
> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <
> emkornfield@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> The process should be well documented at this point but there are a
> > >> >> number of steps.
> > >> >
> > >> > Is [1] the up-to-date documentation for the release? Are there
> > >> instructions for the adding the code signing Key to SVN?
> > >> >
> > >> > I will make a go of it. i will try to mitigate any internet issues
> by
> > >> doing the process for a cloud instance (I assume that isn't a
> problem?).
> > >> >
> > >>
> > >> Setting up a new cloud environment suitable for producing an RC may be
> > >> time consuming, but you are welcome to try. Krisztian -- are you
> > >> available next week to help Micah and potentially take over producing
> > >> the RC if there are issues?
> > >>
> > > Sure, I'll be available next week. We can also grant access to
> > > https://github.com/ursa-labs/crossbow because configuring all
> > > the CI backends can be time consuming.
> > >
> > >>
> > >> > Thanks,
> > >> > Micah
> > >> >
> > >> > [1]
> > >>
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > >> >
> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> The process should be well documented at this point but there are a
> > >> >> number of steps. Note that you need to add your code signing key to
> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's
> fine
> > >> >> to hand off the process to others after the VOTE but it would be
> > >> >> tricky to have multiple RMs involved with producing the source and
> > >> >> binary artifacts for the vote
> > >> >>
> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> > >> emkornfield@gmail.com> wrote:
> > >> >> >
> > >> >> > SGTM, as well.
> > >> >> >
> > >> >> > I should have a little bit of time next week if I can help as RM
> but
> > >> I have
> > >> >> > a couple of concerns:
> > >> >> > 1. In the past I've had trouble downloading and validating
> > >> releases. I'm a
> > >> >> > bit worried, that I might have similar problems doing the
> necessary
> > >> uploads.
> > >> >> > 2. My internet connection will likely be not great, I don't
> know if
> > >> this
> > >> >> > would make it even less likely to be successful.
> > >> >> >
> > >> >> > Does it become problematic if somehow I would have to abandon the
> > >> process
> > >> >> > mid-release? Is there anyone who could serve as a backup? Are
> the
> > >> steps
> > >> >> > well documented?
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Micah
> > >> >> >
> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> > >> neal.p.richardson@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Sounds good to me.
> > >> >> > >
> > >> >> > > Do we have a release manager yet? Any volunteers?
> > >> >> > >
> > >> >> > > Neal
> > >> >> > >
> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <
> wesmckinn@gmail.com>
> > >> wrote:
> > >> >> > >
> > >> >> > > > hi all,
> > >> >> > > >
> > >> >> > > > It looks like we're drawing close to be able to make the
> 0.15.0
> > >> >> > > > release. I would suggest "pencils down" at the end of this
> week
> > >> and
> > >> >> > > > see if a release candidate can be produced next Monday
> September
> > >> 23.
> > >> >> > > > Any thoughts or objections?
> > >> >> > > >
> > >> >> > > > Thanks,
> > >> >> > > > Wes
> > >> >> > > >
> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > wrote:
> > >> >> > > > >
> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> > >> Format docs
> > >> >> > > > > today regarding the EOS issue and also update the C++
> library
> > >> >> > > > >
> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > >> >> > > > > <Er...@microsoft.com> wrote:
> > >> >> > > > > >
> > >> >> > > > > > I assume the plan is to merge the
> > >> ARROW-6313-flatbuffer-alignment
> > >> >> > > > branch into master before the 0.15 release, correct?
> > >> >> > > > > >
> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> > >> merged into
> > >> >> > > > the alignment branch -
> > >> https://github.com/apache/arrow/pull/5280/
> > >> >> > > > > >
> > >> >> > > > > > Eric
> > >> >> > > > > >
> > >> >> > > > > > -----Original Message-----
> > >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> > >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <
> niki.lj@aliyun.com>
> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> > >> >> > > > > >
> > >> >> > > > > > I should have a little more bandwidth to help with some
> of
> > >> the
> > >> >> > > > packaging starting tomorrow and going into the weekend.
> > >> >> > > > > >
> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > > wrote:
> > >> >> > > > > >
> > >> >> > > > > > > Hi folks,
> > >> >> > > > > > >
> > >> >> > > > > > > With the state of nightly packaging and integration
> builds
> > >> things
> > >> >> > > > > > > aren't looking too good for being in release readiness
> by
> > >> the end
> > >> >> > > of
> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be
> working
> > >> to close
> > >> >> > > as
> > >> >> > > > > > > many issues as I can and also to help with the ongoing
> > >> alignment
> > >> >> > > > fixes.
> > >> >> > > > > > >
> > >> >> > > > > > > Wes
> > >> >> > > > > > >
> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > >> >> > > emkornfield@gmail.com
> > >> >> > > > >
> > >> >> > > > > > > wrote:
> > >> >> > > > > > >
> > >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> > >> issues:
> > >> >> > > > > > >>
> > >> >> > > > > > >>
> > >> >> > > >
> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > >> >> > > > > > >> ki.apache.org
> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034
> > >> >> > > > > > >>
> > >> >> > > >
> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> >> > > > > > >>
> > >> >> > > >
> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > >> >> > > > > > >> %3D&reserved=0
> > >> >> > > > > > >>
> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> > >> wesmckinn@gmail.com>
> > >> >> > > > wrote:
> > >> >> > > > > > >>
> > >> >> > > > > > >>> hi all,
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> It doesn't seem like we're going to be in a position
> to
> > >> release
> > >> >> > > at
> > >> >> > > > > > >>> the beginning of next week. I hope that one more
> week of
> > >> work (or
> > >> >> > > > > > >>> less) will be enough to get us there. Aside from
> merging
> > >> the
> > >> >> > > > > > >>> alignment changes, we need to make sure that our
> > >> packaging jobs
> > >> >> > > > > > >>> required for the release candidate are all working.
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> > >> that they
> > >> >> > > > don't
> > >> >> > > > > > >>> think they will finish by end of next week that would
> > >> help focus
> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0
> still).
> > >> I am
> > >> >> > > > > > >>> looking to tackle a few small features related to
> > >> dictionaries
> > >> >> > > > while
> > >> >> > > > > > >>> the release window is still open.
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> - Wes
> > >> >> > > > > > >>>
> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > >> >> > > wesmckinn@gmail.com>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > hi,
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > I think we should try to release the week of
> September
> > >> 9, so
> > >> >> > > > > > >>> > development work should be completed by end of next
> > >> week.
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > Does that seem reasonable?
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> > >> changes for
> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that
> getting
> > >> the
> > >> >> > > > > > >>> > alignment work done is the main barrier to
> releasing.
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > Thanks
> > >> >> > > > > > >>> > Wes
> > >> >> > > > > > >>> >
> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> > >> bugs that
> > >> >> > > > need
> > >> >> > > > > > >>> > > to
> > >> >> > > > > > >>> be fixed or reminded.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> > >> IPC streams
> > >> >> > > > > > >>> > > even
> > >> >> > > > > > >>> when empty[1]
> > >> >> > > > > > >>> > > This one is under review now, however through
> this
> > >> PR we find
> > >> >> > > > > > >>> > > that
> > >> >> > > > > > >>> there seems a bug in java reading and writing
> > >> dictionaries in IPC
> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes
> all
> > >> >> > > > dictionaries
> > >> >> > > > > > >>> are at the start of stream (see details in PR
> comments,
> > >> and this
> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah
> Kornfield
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> > >> integration
> > >> >> > > > test
> > >> >> > > > > > >>> JSON files[3]
> > >> >> > > > > > >>> > > Java side code already checked in, other
> > >> implementations
> > >> >> > > seems
> > >> >> > > > not.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > >> Caused by
> > >> >> > > trying
> > >> >> > > > > > >>> > > to load all records in one contiguous batch,
> fixed
> > >> >> > > > > > >>> by providing iterator API for iteratively reading in
> > >> >> > > ARROW-6219[5].
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > Thanks,
> > >> >> > > > > > >>> > > Ji Liu
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > [1]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > >> >> > > > > > >>> > >
> > >> >> > > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > >> >> > > > > > >>> > >
> > >> >> > > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > >> >> > > > > > >>> > > reserved=0 [2]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Farrow.apache.org
> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > >> >> > > > > > >>> > > ardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> >> > > > > > >>> > >
> > >> >> > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > >> >> > > > > > >>> > >
> > >> >> > > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > >> >> > > > > > >>> > > d=0 [3]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678
> > >> >> > > > > > >>> > >
> > >> >> > > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > >> >> > > > > > >>> > > ;reserved=0 [4]
> > >> >> > > > > > >>> > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >> >> > > > > > >>> > > 2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d73
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > >> >> > > > > > >>> > >
> > >> >> > > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > >> >> > > > > > >>> > > &reserved=0]
> > >> >> > > > > > >>>
> > >> >> > > >
> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > >> >> > > > > > >>> sues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > >> >> > > > > > >>> .Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >> >> > > > > > >>>
> > >> >> > > >
> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > >> >> > > > > > >>>
> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > >
> > >> >> > > >
> ----------------------------------------------------------------
> > >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> > >> dev@arrow.apache.org>
> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> > >> backlog some
> > >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> > >> >> > > (particularly
> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> > >> focusing) that
> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> > >> issues
> > >> >> > > opened
> > >> >> > > > > > >>> > > since the
> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> > >> whether
> > >> >> > > there's
> > >> >> > > > > > >>> > > any regressions or other serious bugs that we
> should
> > >> try to
> > >> >> > > fix
> > >> >> > > > > > >>> > > for 0.15.0.
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > >> >> > > > > > >>> > > <we...@gmail.com>
> > >> >> > > > > > >>> wrote:
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > > F%2Fissues.apache.org
> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > I think the root cause could be the Windows
> > >> changes in
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > >> >> > > > %7Ccbead81a42104034a4f308d736678a
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > >> >> > > > > > >>> > > > bs%3D&reserved=0
> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would
> look
> > >> into what
> > >> >> > > > > > >>> > > > was
> > >> >> > > > > > >>> wrong
> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> > >> 0.15.0 Windows
> > >> >> > > > > > >>> > > > wheels will be broken, too
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > The bad wheels can be found at
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> > >> 40microsoft.com
> > >> >> > > > %7Ccbea
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > >> >> > > > > > >>> > > >
> > >> >> > > >
> 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > >> >> > > > > > >>> > > >
> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou
> <
> > >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> > >> Kornfield
> > >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > >> >> > > > > > >>> > > > > > >
> > >> >> > > > > > >>> > > > > > > In C++ they are
> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> > >> lengths and
> > >> >> > > > > > >>> variable-length
> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted
> (we
> > >> just
> > >> >> > > > wouldn't
> > >> >> > > > > > >>> > > > > > > be
> > >> >> > > > > > >>> able to
> > >> >> > > > > > >>> > > > > > > have a List child with more than
> INT32_MAX
> > >> elements).
> > >> >> > > > > > >>> > > > > >
> > >> >> > > > > > >>> > > > > > I think the point is we could do this in
> C++
> > >> but we
> > >> >> > > > don't.
> > >> >> > > > > > >>> I'm not sure we
> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types if
> we
> > >> did.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as
> 32-bit
> > >> >> > > offsets,
> > >> >> > > > > > >>> > > > > so if
> > >> >> > > > > > >>> you're
> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> > >> 32-bit
> > >> >> > > offsets
> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> > >> lengths from
> > >> >> > > > the
> > >> >> > > > > > >>> > > > > start
> > >> >> > > > > > >>> it would
> > >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> > >> offsets.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > > Going with the limited address space in
> Java
> > >> and
> > >> >> > > calling
> > >> >> > > > > > >>> > > > > > it a
> > >> >> > > > > > >>> reference
> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a
> consumer
> > >> uses a
> > >> >> > > > "Large"
> > >> >> > > > > > >>> type
> > >> >> > > > > > >>> > > > > > presumably it is because they need the
> ability
> > >> to store
> > >> >> > > > > > >>> > > > > > more
> > >> >> > > > > > >>> than INT32_MAX
> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> > >> just
> > >> >> > > wasting
> > >> >> > > > > > >>> > > > > > space
> > >> >> > > > > > >>> [1].
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> > >> (lists or
> > >> >> > > > > > >>> > > > > strings)
> > >> >> > > > > > >>> are
> > >> >> > > > > > >>> > > > > large, not much space is wasted in
> proportion,
> > >> so it may
> > >> >> > > be
> > >> >> > > > > > >>> simpler in
> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> > >> array.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be
> some
> > >> >> > > > > > >>> > > > > > performance
> > >> >> > > > > > >>> benefits on
> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native
> word
> > >> sizes.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures
> don't do
> > >> that, as
> > >> >> > > > > > >>> > > > > 32-bit
> > >> >> > > > > > >>> is an
> > >> >> > > > > > >>> > > > > extremely common integer size even in
> > >> high-performance
> > >> >> > > > code.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Regards
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > > Antoine.
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > > > >
> > >> >> > > > > > >>> > >
> > >> >> > > > > > >>>
> > >> >> > > > > > >>
> > >> >> > > >
> > >> >> > >
> > >>
> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
It's ideal if your GPG key is in the web of trust (i.e. you can get it
signed by another PMC member), but is not 100% essential.
Speaking of the release, there are at least 2 code changes I still
want to get in
ARROW-5717
ARROW-6353
I just pushed updates to ARROW-5717, will merge once the build is green.
There are a couple of Rust patches still marked for 0.15. The rest
seems to be documentation and a couple of integration test failures we
should see about fixing in time.
On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <em...@gmail.com> wrote:
>
> Thanks Krisztián and Wes,
> I've gone ahead and started registering myself on all the packaging sites.
>
> Is there any review process when adding my GPG key to the SVN file? [1]
> doesn't seem to mention explicitly.
>
> Thanks,
> Micah
>
> [1] https://www.apache.org/dev/version-control.html#https-svn
>
> On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
> wrote:
>
> > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
> >
> >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
> >> wrote:
> >> >>
> >> >> The process should be well documented at this point but there are a
> >> >> number of steps.
> >> >
> >> > Is [1] the up-to-date documentation for the release? Are there
> >> instructions for the adding the code signing Key to SVN?
> >> >
> >> > I will make a go of it. i will try to mitigate any internet issues by
> >> doing the process for a cloud instance (I assume that isn't a problem?).
> >> >
> >>
> >> Setting up a new cloud environment suitable for producing an RC may be
> >> time consuming, but you are welcome to try. Krisztian -- are you
> >> available next week to help Micah and potentially take over producing
> >> the RC if there are issues?
> >>
> > Sure, I'll be available next week. We can also grant access to
> > https://github.com/ursa-labs/crossbow because configuring all
> > the CI backends can be time consuming.
> >
> >>
> >> > Thanks,
> >> > Micah
> >> >
> >> > [1]
> >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> >> >
> >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> >> wrote:
> >> >>
> >> >> The process should be well documented at this point but there are a
> >> >> number of steps. Note that you need to add your code signing key to
> >> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
> >> >> to hand off the process to others after the VOTE but it would be
> >> >> tricky to have multiple RMs involved with producing the source and
> >> >> binary artifacts for the vote
> >> >>
> >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
> >> emkornfield@gmail.com> wrote:
> >> >> >
> >> >> > SGTM, as well.
> >> >> >
> >> >> > I should have a little bit of time next week if I can help as RM but
> >> I have
> >> >> > a couple of concerns:
> >> >> > 1. In the past I've had trouble downloading and validating
> >> releases. I'm a
> >> >> > bit worried, that I might have similar problems doing the necessary
> >> uploads.
> >> >> > 2. My internet connection will likely be not great, I don't know if
> >> this
> >> >> > would make it even less likely to be successful.
> >> >> >
> >> >> > Does it become problematic if somehow I would have to abandon the
> >> process
> >> >> > mid-release? Is there anyone who could serve as a backup? Are the
> >> steps
> >> >> > well documented?
> >> >> >
> >> >> > Thanks,
> >> >> > Micah
> >> >> >
> >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> >> neal.p.richardson@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Sounds good to me.
> >> >> > >
> >> >> > > Do we have a release manager yet? Any volunteers?
> >> >> > >
> >> >> > > Neal
> >> >> > >
> >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> >> wrote:
> >> >> > >
> >> >> > > > hi all,
> >> >> > > >
> >> >> > > > It looks like we're drawing close to be able to make the 0.15.0
> >> >> > > > release. I would suggest "pencils down" at the end of this week
> >> and
> >> >> > > > see if a release candidate can be produced next Monday September
> >> 23.
> >> >> > > > Any thoughts or objections?
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > > Wes
> >> >> > > >
> >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> >> wesmckinn@gmail.com>
> >> >> > > wrote:
> >> >> > > > >
> >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> >> Format docs
> >> >> > > > > today regarding the EOS issue and also update the C++ library
> >> >> > > > >
> >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> >> >> > > > > <Er...@microsoft.com> wrote:
> >> >> > > > > >
> >> >> > > > > > I assume the plan is to merge the
> >> ARROW-6313-flatbuffer-alignment
> >> >> > > > branch into master before the 0.15 release, correct?
> >> >> > > > > >
> >> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> >> merged into
> >> >> > > > the alignment branch -
> >> https://github.com/apache/arrow/pull/5280/
> >> >> > > > > >
> >> >> > > > > > Eric
> >> >> > > > > >
> >> >> > > > > > -----Original Message-----
> >> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> >> >> > > > > > To: Wes McKinney <we...@gmail.com>
> >> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> >> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> >> >> > > > > >
> >> >> > > > > > I should have a little more bandwidth to help with some of
> >> the
> >> >> > > > packaging starting tomorrow and going into the weekend.
> >> >> > > > > >
> >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> >> wesmckinn@gmail.com>
> >> >> > > > wrote:
> >> >> > > > > >
> >> >> > > > > > > Hi folks,
> >> >> > > > > > >
> >> >> > > > > > > With the state of nightly packaging and integration builds
> >> things
> >> >> > > > > > > aren't looking too good for being in release readiness by
> >> the end
> >> >> > > of
> >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
> >> to close
> >> >> > > as
> >> >> > > > > > > many issues as I can and also to help with the ongoing
> >> alignment
> >> >> > > > fixes.
> >> >> > > > > > >
> >> >> > > > > > > Wes
> >> >> > > > > > >
> >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> >> >> > > emkornfield@gmail.com
> >> >> > > > >
> >> >> > > > > > > wrote:
> >> >> > > > > > >
> >> >> > > > > > >> Just for reference [1] has a dashboard of the current
> >> issues:
> >> >> > > > > > >>
> >> >> > > > > > >>
> >> >> > > >
> >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >> >> > > > > > >> ki.apache.org
> >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034
> >> >> > > > > > >>
> >> >> > > >
> >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> >> > > > > > >>
> >> >> > > >
> >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >> >> > > > > > >> %3D&reserved=0
> >> >> > > > > > >>
> >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> >> wesmckinn@gmail.com>
> >> >> > > > wrote:
> >> >> > > > > > >>
> >> >> > > > > > >>> hi all,
> >> >> > > > > > >>>
> >> >> > > > > > >>> It doesn't seem like we're going to be in a position to
> >> release
> >> >> > > at
> >> >> > > > > > >>> the beginning of next week. I hope that one more week of
> >> work (or
> >> >> > > > > > >>> less) will be enough to get us there. Aside from merging
> >> the
> >> >> > > > > > >>> alignment changes, we need to make sure that our
> >> packaging jobs
> >> >> > > > > > >>> required for the release candidate are all working.
> >> >> > > > > > >>>
> >> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
> >> that they
> >> >> > > > don't
> >> >> > > > > > >>> think they will finish by end of next week that would
> >> help focus
> >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
> >> I am
> >> >> > > > > > >>> looking to tackle a few small features related to
> >> dictionaries
> >> >> > > > while
> >> >> > > > > > >>> the release window is still open.
> >> >> > > > > > >>>
> >> >> > > > > > >>> - Wes
> >> >> > > > > > >>>
> >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> >> >> > > wesmckinn@gmail.com>
> >> >> > > > > > >>> wrote:
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > hi,
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > I think we should try to release the week of September
> >> 9, so
> >> >> > > > > > >>> > development work should be completed by end of next
> >> week.
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > Does that seem reasonable?
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> >> changes for
> >> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
> >> the
> >> >> > > > > > >>> > alignment work done is the main barrier to releasing.
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > Thanks
> >> >> > > > > > >>> > Wes
> >> >> > > > > > >>> >
> >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> >> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> >> >> > > > > > >>> wrote:
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> >> bugs that
> >> >> > > > need
> >> >> > > > > > >>> > > to
> >> >> > > > > > >>> be fixed or reminded.
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
> >> IPC streams
> >> >> > > > > > >>> > > even
> >> >> > > > > > >>> when empty[1]
> >> >> > > > > > >>> > > This one is under review now, however through this
> >> PR we find
> >> >> > > > > > >>> > > that
> >> >> > > > > > >>> there seems a bug in java reading and writing
> >> dictionaries in IPC
> >> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> >> >> > > > dictionaries
> >> >> > > > > > >>> are at the start of stream (see details in PR comments,
> >> and this
> >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> >> integration
> >> >> > > > test
> >> >> > > > > > >>> JSON files[3]
> >> >> > > > > > >>> > > Java side code already checked in, other
> >> implementations
> >> >> > > seems
> >> >> > > > not.
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> >> Caused by
> >> >> > > trying
> >> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
> >> >> > > > > > >>> by providing iterator API for iteratively reading in
> >> >> > > ARROW-6219[5].
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > Thanks,
> >> >> > > > > > >>> > > Ji Liu
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > [1]
> >> >> > > > > > >>> > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> >> > > > > > >>> > >
> >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> >> >> > > > > > >>> > >
> >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >> >> > > > > > >>> > >
> >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >> >> > > > > > >>> > > reserved=0 [2]
> >> >> > > > > > >>> > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> >> > > > > > >>> > > 2Farrow.apache.org
> >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >> >> > > > > > >>> > > ardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> >> > > > > > >>> > >
> >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >> >> > > > > > >>> > >
> >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >> >> > > > > > >>> > > d=0 [3]
> >> >> > > > > > >>> > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> >> > > > > > >>> > > 2Fissues.apache.org
> >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d736678
> >> >> > > > > > >>> > >
> >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >> >> > > > > > >>> > >
> >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >> >> > > > > > >>> > > ;reserved=0 [4]
> >> >> > > > > > >>> > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> >> > > > > > >>> > > 2Fissues.apache.org
> >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d73
> >> >> > > > > > >>> > >
> >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >> >> > > > > > >>> > >
> >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >> >> > > > > > >>> > > &reserved=0]
> >> >> > > > > > >>>
> >> >> > > >
> >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >> >> > > > > > >>> sues.apache.org
> >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >> >> > > > > > >>> .Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> >> > > > > > >>>
> >> >> > > >
> >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >> >> > > > > > >>>
> >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > >
> >> >> > > > ----------------------------------------------------------------
> >> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> >> dev@arrow.apache.org>
> >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> >> backlog some
> >> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> >> >> > > (particularly
> >> >> > > > > > >>> > > for languages other than C++/Python where I'm
> >> focusing) that
> >> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> >> issues
> >> >> > > opened
> >> >> > > > > > >>> > > since the
> >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> >> whether
> >> >> > > there's
> >> >> > > > > > >>> > > any regressions or other serious bugs that we should
> >> try to
> >> >> > > fix
> >> >> > > > > > >>> > > for 0.15.0.
> >> >> > > > > > >>> > >
> >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> >> >> > > > > > >>> > > <we...@gmail.com>
> >> >> > > > > > >>> wrote:
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> >> > > > > > >>> > > > F%2Fissues.apache.org
> >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d
> >> >> > > > > > >>> > > >
> >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> >> > > > > > >>> > > >
> >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > > I think the root cause could be the Windows
> >> changes in
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> >> > > > > > >>> > > >
> >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> >> >> > > > %7Ccbead81a42104034a4f308d736678a
> >> >> > > > > > >>> > > >
> >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >> >> > > > > > >>> > > >
> >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >> >> > > > > > >>> > > > bs%3D&reserved=0
> >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
> >> into what
> >> >> > > > > > >>> > > > was
> >> >> > > > > > >>> wrong
> >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
> >> 0.15.0 Windows
> >> >> > > > > > >>> > > > wheels will be broken, too
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > > The bad wheels can be found at
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > >
> >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> >> > > > > > >>> > > >
> >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> >> 40microsoft.com
> >> >> > > > %7Ccbea
> >> >> > > > > > >>> > > >
> >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >> >> > > > > > >>> > > >
> >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >> >> > > > > > >>> > > >
> >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> >> >> > > > > > >>> solipsis@pitrou.net> wrote:
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
> >> Kornfield
> >> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> >> >> > > > > > >>> > > > > > >
> >> >> > > > > > >>> > > > > > > In C++ they are
> >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> >> lengths and
> >> >> > > > > > >>> variable-length
> >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
> >> just
> >> >> > > > wouldn't
> >> >> > > > > > >>> > > > > > > be
> >> >> > > > > > >>> able to
> >> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> >> elements).
> >> >> > > > > > >>> > > > > >
> >> >> > > > > > >>> > > > > > I think the point is we could do this in C++
> >> but we
> >> >> > > > don't.
> >> >> > > > > > >>> I'm not sure we
> >> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
> >> did.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> >> >> > > offsets,
> >> >> > > > > > >>> > > > > so if
> >> >> > > > > > >>> you're
> >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
> >> 32-bit
> >> >> > > offsets
> >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> >> lengths from
> >> >> > > > the
> >> >> > > > > > >>> > > > > start
> >> >> > > > > > >>> it would
> >> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> >> offsets.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > > Going with the limited address space in Java
> >> and
> >> >> > > calling
> >> >> > > > > > >>> > > > > > it a
> >> >> > > > > > >>> reference
> >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> >> uses a
> >> >> > > > "Large"
> >> >> > > > > > >>> type
> >> >> > > > > > >>> > > > > > presumably it is because they need the ability
> >> to store
> >> >> > > > > > >>> > > > > > more
> >> >> > > > > > >>> than INT32_MAX
> >> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
> >> just
> >> >> > > wasting
> >> >> > > > > > >>> > > > > > space
> >> >> > > > > > >>> [1].
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > Probably. Though if the individual elements
> >> (lists or
> >> >> > > > > > >>> > > > > strings)
> >> >> > > > > > >>> are
> >> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
> >> so it may
> >> >> > > be
> >> >> > > > > > >>> simpler in
> >> >> > > > > > >>> > > > > such a case to always create a "Large" type
> >> array.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> >> >> > > > > > >>> > > > > > performance
> >> >> > > > > > >>> benefits on
> >> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
> >> sizes.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> >> that, as
> >> >> > > > > > >>> > > > > 32-bit
> >> >> > > > > > >>> is an
> >> >> > > > > > >>> > > > > extremely common integer size even in
> >> high-performance
> >> >> > > > code.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > Regards
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > > Antoine.
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > > > >
> >> >> > > > > > >>> > >
> >> >> > > > > > >>>
> >> >> > > > > > >>
> >> >> > > >
> >> >> > >
> >>
> >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
Thanks Krisztián and Wes,
I've gone ahead and started registering myself on all the packaging sites.
Is there any review process when adding my GPG key to the SVN file? [1]
doesn't seem to mention explicitly.
Thanks,
Micah
[1] https://www.apache.org/dev/version-control.html#https-svn
On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs <sz...@gmail.com>
wrote:
> On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
>
>> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
>> wrote:
>> >>
>> >> The process should be well documented at this point but there are a
>> >> number of steps.
>> >
>> > Is [1] the up-to-date documentation for the release? Are there
>> instructions for the adding the code signing Key to SVN?
>> >
>> > I will make a go of it. i will try to mitigate any internet issues by
>> doing the process for a cloud instance (I assume that isn't a problem?).
>> >
>>
>> Setting up a new cloud environment suitable for producing an RC may be
>> time consuming, but you are welcome to try. Krisztian -- are you
>> available next week to help Micah and potentially take over producing
>> the RC if there are issues?
>>
> Sure, I'll be available next week. We can also grant access to
> https://github.com/ursa-labs/crossbow because configuring all
> the CI backends can be time consuming.
>
>>
>> > Thanks,
>> > Micah
>> >
>> > [1]
>> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>> >
>> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
>> wrote:
>> >>
>> >> The process should be well documented at this point but there are a
>> >> number of steps. Note that you need to add your code signing key to
>> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
>> >> to hand off the process to others after the VOTE but it would be
>> >> tricky to have multiple RMs involved with producing the source and
>> >> binary artifacts for the vote
>> >>
>> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <
>> emkornfield@gmail.com> wrote:
>> >> >
>> >> > SGTM, as well.
>> >> >
>> >> > I should have a little bit of time next week if I can help as RM but
>> I have
>> >> > a couple of concerns:
>> >> > 1. In the past I've had trouble downloading and validating
>> releases. I'm a
>> >> > bit worried, that I might have similar problems doing the necessary
>> uploads.
>> >> > 2. My internet connection will likely be not great, I don't know if
>> this
>> >> > would make it even less likely to be successful.
>> >> >
>> >> > Does it become problematic if somehow I would have to abandon the
>> process
>> >> > mid-release? Is there anyone who could serve as a backup? Are the
>> steps
>> >> > well documented?
>> >> >
>> >> > Thanks,
>> >> > Micah
>> >> >
>> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
>> neal.p.richardson@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > Sounds good to me.
>> >> > >
>> >> > > Do we have a release manager yet? Any volunteers?
>> >> > >
>> >> > > Neal
>> >> > >
>> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
>> wrote:
>> >> > >
>> >> > > > hi all,
>> >> > > >
>> >> > > > It looks like we're drawing close to be able to make the 0.15.0
>> >> > > > release. I would suggest "pencils down" at the end of this week
>> and
>> >> > > > see if a release candidate can be produced next Monday September
>> 23.
>> >> > > > Any thoughts or objections?
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Wes
>> >> > > >
>> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
>> wesmckinn@gmail.com>
>> >> > > wrote:
>> >> > > > >
>> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
>> Format docs
>> >> > > > > today regarding the EOS issue and also update the C++ library
>> >> > > > >
>> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
>> >> > > > > <Er...@microsoft.com> wrote:
>> >> > > > > >
>> >> > > > > > I assume the plan is to merge the
>> ARROW-6313-flatbuffer-alignment
>> >> > > > branch into master before the 0.15 release, correct?
>> >> > > > > >
>> >> > > > > > BTW - I believe the C# alignment changes are ready to be
>> merged into
>> >> > > > the alignment branch -
>> https://github.com/apache/arrow/pull/5280/
>> >> > > > > >
>> >> > > > > > Eric
>> >> > > > > >
>> >> > > > > > -----Original Message-----
>> >> > > > > > From: Micah Kornfield <em...@gmail.com>
>> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
>> >> > > > > > To: Wes McKinney <we...@gmail.com>
>> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
>> >> > > > > > Subject: Re: Timeline for 0.15.0 release
>> >> > > > > >
>> >> > > > > > I should have a little more bandwidth to help with some of
>> the
>> >> > > > packaging starting tomorrow and going into the weekend.
>> >> > > > > >
>> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
>> wesmckinn@gmail.com>
>> >> > > > wrote:
>> >> > > > > >
>> >> > > > > > > Hi folks,
>> >> > > > > > >
>> >> > > > > > > With the state of nightly packaging and integration builds
>> things
>> >> > > > > > > aren't looking too good for being in release readiness by
>> the end
>> >> > > of
>> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
>> to close
>> >> > > as
>> >> > > > > > > many issues as I can and also to help with the ongoing
>> alignment
>> >> > > > fixes.
>> >> > > > > > >
>> >> > > > > > > Wes
>> >> > > > > > >
>> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
>> >> > > emkornfield@gmail.com
>> >> > > > >
>> >> > > > > > > wrote:
>> >> > > > > > >
>> >> > > > > > >> Just for reference [1] has a dashboard of the current
>> issues:
>> >> > > > > > >>
>> >> > > > > > >>
>> >> > > >
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> >> > > > > > >> ki.apache.org
>> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034
>> >> > > > > > >>
>> >> > > >
>> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> >> > > > > > >>
>> >> > > >
>> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> >> > > > > > >> %3D&reserved=0
>> >> > > > > > >>
>> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
>> wesmckinn@gmail.com>
>> >> > > > wrote:
>> >> > > > > > >>
>> >> > > > > > >>> hi all,
>> >> > > > > > >>>
>> >> > > > > > >>> It doesn't seem like we're going to be in a position to
>> release
>> >> > > at
>> >> > > > > > >>> the beginning of next week. I hope that one more week of
>> work (or
>> >> > > > > > >>> less) will be enough to get us there. Aside from merging
>> the
>> >> > > > > > >>> alignment changes, we need to make sure that our
>> packaging jobs
>> >> > > > > > >>> required for the release candidate are all working.
>> >> > > > > > >>>
>> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog
>> that they
>> >> > > > don't
>> >> > > > > > >>> think they will finish by end of next week that would
>> help focus
>> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
>> I am
>> >> > > > > > >>> looking to tackle a few small features related to
>> dictionaries
>> >> > > > while
>> >> > > > > > >>> the release window is still open.
>> >> > > > > > >>>
>> >> > > > > > >>> - Wes
>> >> > > > > > >>>
>> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
>> >> > > wesmckinn@gmail.com>
>> >> > > > > > >>> wrote:
>> >> > > > > > >>> >
>> >> > > > > > >>> > hi,
>> >> > > > > > >>> >
>> >> > > > > > >>> > I think we should try to release the week of September
>> 9, so
>> >> > > > > > >>> > development work should be completed by end of next
>> week.
>> >> > > > > > >>> >
>> >> > > > > > >>> > Does that seem reasonable?
>> >> > > > > > >>> >
>> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
>> changes for
>> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
>> the
>> >> > > > > > >>> > alignment work done is the main barrier to releasing.
>> >> > > > > > >>> >
>> >> > > > > > >>> > Thanks
>> >> > > > > > >>> > Wes
>> >> > > > > > >>> >
>> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
>> >> > > > > > >>> > <ni...@aliyun.com.invalid>
>> >> > > > > > >>> wrote:
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
>> bugs that
>> >> > > > need
>> >> > > > > > >>> > > to
>> >> > > > > > >>> be fixed or reminded.
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in
>> IPC streams
>> >> > > > > > >>> > > even
>> >> > > > > > >>> when empty[1]
>> >> > > > > > >>> > > This one is under review now, however through this
>> PR we find
>> >> > > > > > >>> > > that
>> >> > > > > > >>> there seems a bug in java reading and writing
>> dictionaries in IPC
>> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
>> >> > > > dictionaries
>> >> > > > > > >>> are at the start of stream (see details in PR comments,
>> and this
>> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
>> integration
>> >> > > > test
>> >> > > > > > >>> JSON files[3]
>> >> > > > > > >>> > > Java side code already checked in, other
>> implementations
>> >> > > seems
>> >> > > > not.
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
>> Caused by
>> >> > > trying
>> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
>> >> > > > > > >>> by providing iterator API for iteratively reading in
>> >> > > ARROW-6219[5].
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > Thanks,
>> >> > > > > > >>> > > Ji Liu
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > [1]
>> >> > > > > > >>> > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> >> > > > > > >>> > >
>> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
>> >> > > > > > >>> > >
>> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>> >> > > > > > >>> > >
>> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>> >> > > > > > >>> > > reserved=0 [2]
>> >> > > > > > >>> > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> >> > > > > > >>> > > 2Farrow.apache.org
>> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>> >> > > > > > >>> > > ardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> >> > > > > > >>> > >
>> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>> >> > > > > > >>> > >
>> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>> >> > > > > > >>> > > d=0 [3]
>> >> > > > > > >>> > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> >> > > > > > >>> > > 2Fissues.apache.org
>> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d736678
>> >> > > > > > >>> > >
>> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>> >> > > > > > >>> > >
>> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>> >> > > > > > >>> > > ;reserved=0 [4]
>> >> > > > > > >>> > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> >> > > > > > >>> > > 2Fissues.apache.org
>> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d73
>> >> > > > > > >>> > >
>> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>> >> > > > > > >>> > >
>> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>> >> > > > > > >>> > > &reserved=0]
>> >> > > > > > >>>
>> >> > > >
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>> >> > > > > > >>> sues.apache.org
>> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>> >> > > > > > >>> .Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> >> > > > > > >>>
>> >> > > >
>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>> >> > > > > > >>>
>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>> >> > > > > > >>> > >
>> >> > > > > > >>> > >
>> >> > > > > > >>> > >
>> >> > > > > > >>> > >
>> >> > > > ----------------------------------------------------------------
>> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
>> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
>> dev@arrow.apache.org>
>> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
>> backlog some
>> >> > > > > > >>> > > this week, if anyone wants to help with grooming
>> >> > > (particularly
>> >> > > > > > >>> > > for languages other than C++/Python where I'm
>> focusing) that
>> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
>> issues
>> >> > > opened
>> >> > > > > > >>> > > since the
>> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
>> whether
>> >> > > there's
>> >> > > > > > >>> > > any regressions or other serious bugs that we should
>> try to
>> >> > > fix
>> >> > > > > > >>> > > for 0.15.0.
>> >> > > > > > >>> > >
>> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
>> >> > > > > > >>> > > <we...@gmail.com>
>> >> > > > > > >>> wrote:
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> >> > > > > > >>> > > > F%2Fissues.apache.org
>> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d
>> >> > > > > > >>> > > >
>> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> >> > > > > > >>> > > >
>> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>> >> > > > > > >>> > > > WEWg0%3D&reserved=0
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > > I think the root cause could be the Windows
>> changes in
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> >> > > > > > >>> > > >
>> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
>> >> > > > %7Ccbead81a42104034a4f308d736678a
>> >> > > > > > >>> > > >
>> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>> >> > > > > > >>> > > >
>> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>> >> > > > > > >>> > > > bs%3D&reserved=0
>> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
>> into what
>> >> > > > > > >>> > > > was
>> >> > > > > > >>> wrong
>> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise
>> 0.15.0 Windows
>> >> > > > > > >>> > > > wheels will be broken, too
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > > The bad wheels can be found at
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > >
>> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> >> > > > > > >>> > > >
>> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
>> 40microsoft.com
>> >> > > > %7Ccbea
>> >> > > > > > >>> > > >
>> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>> >> > > > > > >>> > > >
>> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>> >> > > > > > >>> > > >
>> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
>> >> > > > > > >>> solipsis@pitrou.net> wrote:
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah
>> Kornfield
>> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
>> >> > > > > > >>> > > > > > >
>> >> > > > > > >>> > > > > > > In C++ they are
>> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
>> lengths and
>> >> > > > > > >>> variable-length
>> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
>> just
>> >> > > > wouldn't
>> >> > > > > > >>> > > > > > > be
>> >> > > > > > >>> able to
>> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
>> elements).
>> >> > > > > > >>> > > > > >
>> >> > > > > > >>> > > > > > I think the point is we could do this in C++
>> but we
>> >> > > > don't.
>> >> > > > > > >>> I'm not sure we
>> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
>> did.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
>> >> > > offsets,
>> >> > > > > > >>> > > > > so if
>> >> > > > > > >>> you're
>> >> > > > > > >>> > > > > storing lots of small-ish lists or strings,
>> 32-bit
>> >> > > offsets
>> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
>> lengths from
>> >> > > > the
>> >> > > > > > >>> > > > > start
>> >> > > > > > >>> it would
>> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
>> offsets.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > > Going with the limited address space in Java
>> and
>> >> > > calling
>> >> > > > > > >>> > > > > > it a
>> >> > > > > > >>> reference
>> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
>> uses a
>> >> > > > "Large"
>> >> > > > > > >>> type
>> >> > > > > > >>> > > > > > presumably it is because they need the ability
>> to store
>> >> > > > > > >>> > > > > > more
>> >> > > > > > >>> than INT32_MAX
>> >> > > > > > >>> > > > > > child elements in a column, otherwise it is
>> just
>> >> > > wasting
>> >> > > > > > >>> > > > > > space
>> >> > > > > > >>> [1].
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > Probably. Though if the individual elements
>> (lists or
>> >> > > > > > >>> > > > > strings)
>> >> > > > > > >>> are
>> >> > > > > > >>> > > > > large, not much space is wasted in proportion,
>> so it may
>> >> > > be
>> >> > > > > > >>> simpler in
>> >> > > > > > >>> > > > > such a case to always create a "Large" type
>> array.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
>> >> > > > > > >>> > > > > > performance
>> >> > > > > > >>> benefits on
>> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
>> sizes.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
>> that, as
>> >> > > > > > >>> > > > > 32-bit
>> >> > > > > > >>> is an
>> >> > > > > > >>> > > > > extremely common integer size even in
>> high-performance
>> >> > > > code.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > Regards
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > > Antoine.
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > > > >
>> >> > > > > > >>> > >
>> >> > > > > > >>>
>> >> > > > > > >>
>> >> > > >
>> >> > >
>>
>
Re: Timeline for 0.15.0 release
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <we...@gmail.com> wrote:
> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com>
> wrote:
> >>
> >> The process should be well documented at this point but there are a
> >> number of steps.
> >
> > Is [1] the up-to-date documentation for the release? Are there
> instructions for the adding the code signing Key to SVN?
> >
> > I will make a go of it. i will try to mitigate any internet issues by
> doing the process for a cloud instance (I assume that isn't a problem?).
> >
>
> Setting up a new cloud environment suitable for producing an RC may be
> time consuming, but you are welcome to try. Krisztian -- are you
> available next week to help Micah and potentially take over producing
> the RC if there are issues?
>
Sure, I'll be available next week. We can also grant access to
https://github.com/ursa-labs/crossbow because configuring all
the CI backends can be time consuming.
>
> > Thanks,
> > Micah
> >
> > [1]
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> >
> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com>
> wrote:
> >>
> >> The process should be well documented at this point but there are a
> >> number of steps. Note that you need to add your code signing key to
> >> the KEYS file in SVN (that's not very hard to do). I think it's fine
> >> to hand off the process to others after the VOTE but it would be
> >> tricky to have multiple RMs involved with producing the source and
> >> binary artifacts for the vote
> >>
> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >> >
> >> > SGTM, as well.
> >> >
> >> > I should have a little bit of time next week if I can help as RM but
> I have
> >> > a couple of concerns:
> >> > 1. In the past I've had trouble downloading and validating releases.
> I'm a
> >> > bit worried, that I might have similar problems doing the necessary
> uploads.
> >> > 2. My internet connection will likely be not great, I don't know if
> this
> >> > would make it even less likely to be successful.
> >> >
> >> > Does it become problematic if somehow I would have to abandon the
> process
> >> > mid-release? Is there anyone who could serve as a backup? Are the
> steps
> >> > well documented?
> >> >
> >> > Thanks,
> >> > Micah
> >> >
> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> neal.p.richardson@gmail.com>
> >> > wrote:
> >> >
> >> > > Sounds good to me.
> >> > >
> >> > > Do we have a release manager yet? Any volunteers?
> >> > >
> >> > > Neal
> >> > >
> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> wrote:
> >> > >
> >> > > > hi all,
> >> > > >
> >> > > > It looks like we're drawing close to be able to make the 0.15.0
> >> > > > release. I would suggest "pencils down" at the end of this week
> and
> >> > > > see if a release candidate can be produced next Monday September
> 23.
> >> > > > Any thoughts or objections?
> >> > > >
> >> > > > Thanks,
> >> > > > Wes
> >> > > >
> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <
> wesmckinn@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the
> Format docs
> >> > > > > today regarding the EOS issue and also update the C++ library
> >> > > > >
> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> >> > > > > <Er...@microsoft.com> wrote:
> >> > > > > >
> >> > > > > > I assume the plan is to merge the
> ARROW-6313-flatbuffer-alignment
> >> > > > branch into master before the 0.15 release, correct?
> >> > > > > >
> >> > > > > > BTW - I believe the C# alignment changes are ready to be
> merged into
> >> > > > the alignment branch -
> https://github.com/apache/arrow/pull/5280/
> >> > > > > >
> >> > > > > > Eric
> >> > > > > >
> >> > > > > > -----Original Message-----
> >> > > > > > From: Micah Kornfield <em...@gmail.com>
> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> >> > > > > > To: Wes McKinney <we...@gmail.com>
> >> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> >> > > > > > Subject: Re: Timeline for 0.15.0 release
> >> > > > > >
> >> > > > > > I should have a little more bandwidth to help with some of the
> >> > > > packaging starting tomorrow and going into the weekend.
> >> > > > > >
> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> wesmckinn@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hi folks,
> >> > > > > > >
> >> > > > > > > With the state of nightly packaging and integration builds
> things
> >> > > > > > > aren't looking too good for being in release readiness by
> the end
> >> > > of
> >> > > > > > > this week but maybe I'm wrong. I'm planning to be working
> to close
> >> > > as
> >> > > > > > > many issues as I can and also to help with the ongoing
> alignment
> >> > > > fixes.
> >> > > > > > >
> >> > > > > > > Wes
> >> > > > > > >
> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> >> > > emkornfield@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > >> Just for reference [1] has a dashboard of the current
> issues:
> >> > > > > > >>
> >> > > > > > >>
> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >> > > > > > >> ki.apache.org
> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034
> >> > > > > > >>
> >> > > >
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> > > > > > >>
> >> > > >
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >> > > > > > >> %3D&reserved=0
> >> > > > > > >>
> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> wesmckinn@gmail.com>
> >> > > > wrote:
> >> > > > > > >>
> >> > > > > > >>> hi all,
> >> > > > > > >>>
> >> > > > > > >>> It doesn't seem like we're going to be in a position to
> release
> >> > > at
> >> > > > > > >>> the beginning of next week. I hope that one more week of
> work (or
> >> > > > > > >>> less) will be enough to get us there. Aside from merging
> the
> >> > > > > > >>> alignment changes, we need to make sure that our
> packaging jobs
> >> > > > > > >>> required for the release candidate are all working.
> >> > > > > > >>>
> >> > > > > > >>> If folks could remove issues from the 0.15.0 backlog that
> they
> >> > > > don't
> >> > > > > > >>> think they will finish by end of next week that would
> help focus
> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still).
> I am
> >> > > > > > >>> looking to tackle a few small features related to
> dictionaries
> >> > > > while
> >> > > > > > >>> the release window is still open.
> >> > > > > > >>>
> >> > > > > > >>> - Wes
> >> > > > > > >>>
> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> >> > > wesmckinn@gmail.com>
> >> > > > > > >>> wrote:
> >> > > > > > >>> >
> >> > > > > > >>> > hi,
> >> > > > > > >>> >
> >> > > > > > >>> > I think we should try to release the week of September
> 9, so
> >> > > > > > >>> > development work should be completed by end of next
> week.
> >> > > > > > >>> >
> >> > > > > > >>> > Does that seem reasonable?
> >> > > > > > >>> >
> >> > > > > > >>> > I plan to get up a patch for the protocol alignment
> changes for
> >> > > > > > >>> > C++ in the next couple of days -- I think that getting
> the
> >> > > > > > >>> > alignment work done is the main barrier to releasing.
> >> > > > > > >>> >
> >> > > > > > >>> > Thanks
> >> > > > > > >>> > Wes
> >> > > > > > >>> >
> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> >> > > > > > >>> > <ni...@aliyun.com.invalid>
> >> > > > > > >>> wrote:
> >> > > > > > >>> > >
> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of several
> bugs that
> >> > > > need
> >> > > > > > >>> > > to
> >> > > > > > >>> be fixed or reminded.
> >> > > > > > >>> > >
> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC
> streams
> >> > > > > > >>> > > even
> >> > > > > > >>> when empty[1]
> >> > > > > > >>> > > This one is under review now, however through this PR
> we find
> >> > > > > > >>> > > that
> >> > > > > > >>> there seems a bug in java reading and writing
> dictionaries in IPC
> >> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> >> > > > dictionaries
> >> > > > > > >>> are at the start of stream (see details in PR comments,
> and this
> >> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> >> > > > > > >>> > >
> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> integration
> >> > > > test
> >> > > > > > >>> JSON files[3]
> >> > > > > > >>> > > Java side code already checked in, other
> implementations
> >> > > seems
> >> > > > not.
> >> > > > > > >>> > >
> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused
> by
> >> > > trying
> >> > > > > > >>> > > to load all records in one contiguous batch, fixed
> >> > > > > > >>> by providing iterator API for iteratively reading in
> >> > > ARROW-6219[5].
> >> > > > > > >>> > >
> >> > > > > > >>> > > Thanks,
> >> > > > > > >>> > > Ji Liu
> >> > > > > > >>> > >
> >> > > > > > >>> > > [1]
> >> > > > > > >>> > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > > > > > >>> > >
> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >> > > > > > >>> > > ric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d736678a45%7
> >> > > > > > >>> > >
> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >> > > > > > >>> > >
> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >> > > > > > >>> > > reserved=0 [2]
> >> > > > > > >>> > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > > > > > >>> > > 2Farrow.apache.org
> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >> > > > > > >>> > > ardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> > > > > > >>> > >
> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >> > > > > > >>> > >
> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >> > > > > > >>> > > d=0 [3]
> >> > > > > > >>> > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > > > > > >>> > > 2Fissues.apache.org
> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d736678
> >> > > > > > >>> > >
> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >> > > > > > >>> > >
> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >> > > > > > >>> > > ;reserved=0 [4]
> >> > > > > > >>> > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >> > > > > > >>> > > 2Fissues.apache.org
> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d73
> >> > > > > > >>> > >
> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >> > > > > > >>> > >
> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >> > > > > > >>> > > &reserved=0]
> >> > > > > > >>>
> >> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >> > > > > > >>> sues.apache.org
> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >> > > > > > >>> .Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> >> > > > > > >>>
> >> > > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >> > > > > > >>>
> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >> > > > > > >>> > >
> >> > > > > > >>> > >
> >> > > > > > >>> > >
> >> > > > > > >>> > >
> >> > > > ----------------------------------------------------------------
> >> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <
> dev@arrow.apache.org>
> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> >> > > > > > >>> > >
> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0
> backlog some
> >> > > > > > >>> > > this week, if anyone wants to help with grooming
> >> > > (particularly
> >> > > > > > >>> > > for languages other than C++/Python where I'm
> focusing) that
> >> > > > > > >>> > > would be helpful. There have been almost 500 JIRA
> issues
> >> > > opened
> >> > > > > > >>> > > since the
> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check
> whether
> >> > > there's
> >> > > > > > >>> > > any regressions or other serious bugs that we should
> try to
> >> > > fix
> >> > > > > > >>> > > for 0.15.0.
> >> > > > > > >>> > >
> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> >> > > > > > >>> > > <we...@gmail.com>
> >> > > > > > >>> wrote:
> >> > > > > > >>> > > >
> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> >> > > > > > >>> > > >
> >> > > > > > >>> > > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > > > > > >>> > > > F%2Fissues.apache.org
> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d
> >> > > > > > >>> > > >
> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> > > > > > >>> > > >
> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >> > > > > > >>> > > > WEWg0%3D&reserved=0
> >> > > > > > >>> > > >
> >> > > > > > >>> > > > I think the root cause could be the Windows changes
> in
> >> > > > > > >>> > > >
> >> > > > > > >>> > > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > > > > > >>> > > >
> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> >> > > > %7Ccbead81a42104034a4f308d736678a
> >> > > > > > >>> > > >
> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >> > > > > > >>> > > >
> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >> > > > > > >>> > > > bs%3D&reserved=0
> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >> > > > > > >>> > > >
> >> > > > > > >>> > > > I would be appreciative if a volunteer would look
> into what
> >> > > > > > >>> > > > was
> >> > > > > > >>> wrong
> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0
> Windows
> >> > > > > > >>> > > > wheels will be broken, too
> >> > > > > > >>> > > >
> >> > > > > > >>> > > > The bad wheels can be found at
> >> > > > > > >>> > > >
> >> > > > > > >>> > > >
> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >> > > > > > >>> > > >
> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> 40microsoft.com
> >> > > > %7Ccbea
> >> > > > > > >>> > > >
> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >> > > > > > >>> > > >
> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >> > > > > > >>> > > >
> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> >> > > > > > >>> solipsis@pitrou.net> wrote:
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> >> > > > > > >>> > > > > <em...@gmail.com> wrote:
> >> > > > > > >>> > > > > > >
> >> > > > > > >>> > > > > > > In C++ they are
> >> > > > > > >>> > > > > > > independent, we could have 32-bit array
> lengths and
> >> > > > > > >>> variable-length
> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we
> just
> >> > > > wouldn't
> >> > > > > > >>> > > > > > > be
> >> > > > > > >>> able to
> >> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> elements).
> >> > > > > > >>> > > > > >
> >> > > > > > >>> > > > > > I think the point is we could do this in C++
> but we
> >> > > > don't.
> >> > > > > > >>> I'm not sure we
> >> > > > > > >>> > > > > > would have introduced the "Large" types if we
> did.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> >> > > offsets,
> >> > > > > > >>> > > > > so if
> >> > > > > > >>> you're
> >> > > > > > >>> > > > > storing lots of small-ish lists or strings, 32-bit
> >> > > offsets
> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array
> lengths from
> >> > > > the
> >> > > > > > >>> > > > > start
> >> > > > > > >>> it would
> >> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> offsets.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > > Going with the limited address space in Java and
> >> > > calling
> >> > > > > > >>> > > > > > it a
> >> > > > > > >>> reference
> >> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> uses a
> >> > > > "Large"
> >> > > > > > >>> type
> >> > > > > > >>> > > > > > presumably it is because they need the ability
> to store
> >> > > > > > >>> > > > > > more
> >> > > > > > >>> than INT32_MAX
> >> > > > > > >>> > > > > > child elements in a column, otherwise it is just
> >> > > wasting
> >> > > > > > >>> > > > > > space
> >> > > > > > >>> [1].
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > Probably. Though if the individual elements
> (lists or
> >> > > > > > >>> > > > > strings)
> >> > > > > > >>> are
> >> > > > > > >>> > > > > large, not much space is wasted in proportion, so
> it may
> >> > > be
> >> > > > > > >>> simpler in
> >> > > > > > >>> > > > > such a case to always create a "Large" type array.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> >> > > > > > >>> > > > > > performance
> >> > > > > > >>> benefits on
> >> > > > > > >>> > > > > > 64-bit architectures to using the native word
> sizes.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> that, as
> >> > > > > > >>> > > > > 32-bit
> >> > > > > > >>> is an
> >> > > > > > >>> > > > > extremely common integer size even in
> high-performance
> >> > > > code.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > Regards
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > > Antoine.
> >> > > > > > >>> > > > >
> >> > > > > > >>> > > > >
> >> > > > > > >>> > >
> >> > > > > > >>>
> >> > > > > > >>
> >> > > >
> >> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield <em...@gmail.com> wrote:
>>
>> The process should be well documented at this point but there are a
>> number of steps.
>
> Is [1] the up-to-date documentation for the release? Are there instructions for the adding the code signing Key to SVN?
>
> I will make a go of it. i will try to mitigate any internet issues by doing the process for a cloud instance (I assume that isn't a problem?).
>
Setting up a new cloud environment suitable for producing an RC may be
time consuming, but you are welcome to try. Krisztian -- are you
available next week to help Micah and potentially take over producing
the RC if there are issues?
> Thanks,
> Micah
>
> [1] https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>
> On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com> wrote:
>>
>> The process should be well documented at this point but there are a
>> number of steps. Note that you need to add your code signing key to
>> the KEYS file in SVN (that's not very hard to do). I think it's fine
>> to hand off the process to others after the VOTE but it would be
>> tricky to have multiple RMs involved with producing the source and
>> binary artifacts for the vote
>>
>> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <em...@gmail.com> wrote:
>> >
>> > SGTM, as well.
>> >
>> > I should have a little bit of time next week if I can help as RM but I have
>> > a couple of concerns:
>> > 1. In the past I've had trouble downloading and validating releases. I'm a
>> > bit worried, that I might have similar problems doing the necessary uploads.
>> > 2. My internet connection will likely be not great, I don't know if this
>> > would make it even less likely to be successful.
>> >
>> > Does it become problematic if somehow I would have to abandon the process
>> > mid-release? Is there anyone who could serve as a backup? Are the steps
>> > well documented?
>> >
>> > Thanks,
>> > Micah
>> >
>> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <ne...@gmail.com>
>> > wrote:
>> >
>> > > Sounds good to me.
>> > >
>> > > Do we have a release manager yet? Any volunteers?
>> > >
>> > > Neal
>> > >
>> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com> wrote:
>> > >
>> > > > hi all,
>> > > >
>> > > > It looks like we're drawing close to be able to make the 0.15.0
>> > > > release. I would suggest "pencils down" at the end of this week and
>> > > > see if a release candidate can be produced next Monday September 23.
>> > > > Any thoughts or objections?
>> > > >
>> > > > Thanks,
>> > > > Wes
>> > > >
>> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > hi Eric -- yes, that's correct. I'm planning to amend the Format docs
>> > > > > today regarding the EOS issue and also update the C++ library
>> > > > >
>> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
>> > > > > <Er...@microsoft.com> wrote:
>> > > > > >
>> > > > > > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment
>> > > > branch into master before the 0.15 release, correct?
>> > > > > >
>> > > > > > BTW - I believe the C# alignment changes are ready to be merged into
>> > > > the alignment branch - https://github.com/apache/arrow/pull/5280/
>> > > > > >
>> > > > > > Eric
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Micah Kornfield <em...@gmail.com>
>> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
>> > > > > > To: Wes McKinney <we...@gmail.com>
>> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
>> > > > > > Subject: Re: Timeline for 0.15.0 release
>> > > > > >
>> > > > > > I should have a little more bandwidth to help with some of the
>> > > > packaging starting tomorrow and going into the weekend.
>> > > > > >
>> > > > > > On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi folks,
>> > > > > > >
>> > > > > > > With the state of nightly packaging and integration builds things
>> > > > > > > aren't looking too good for being in release readiness by the end
>> > > of
>> > > > > > > this week but maybe I'm wrong. I'm planning to be working to close
>> > > as
>> > > > > > > many issues as I can and also to help with the ongoing alignment
>> > > > fixes.
>> > > > > > >
>> > > > > > > Wes
>> > > > > > >
>> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
>> > > emkornfield@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Just for reference [1] has a dashboard of the current issues:
>> > > > > > >>
>> > > > > > >>
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> > > > > > >> ki.apache.org
>> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034
>> > > > > > >>
>> > > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > > > > > >>
>> > > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> > > > > > >> %3D&reserved=0
>> > > > > > >>
>> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com>
>> > > > wrote:
>> > > > > > >>
>> > > > > > >>> hi all,
>> > > > > > >>>
>> > > > > > >>> It doesn't seem like we're going to be in a position to release
>> > > at
>> > > > > > >>> the beginning of next week. I hope that one more week of work (or
>> > > > > > >>> less) will be enough to get us there. Aside from merging the
>> > > > > > >>> alignment changes, we need to make sure that our packaging jobs
>> > > > > > >>> required for the release candidate are all working.
>> > > > > > >>>
>> > > > > > >>> If folks could remove issues from the 0.15.0 backlog that they
>> > > > don't
>> > > > > > >>> think they will finish by end of next week that would help focus
>> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
>> > > > > > >>> looking to tackle a few small features related to dictionaries
>> > > > while
>> > > > > > >>> the release window is still open.
>> > > > > > >>>
>> > > > > > >>> - Wes
>> > > > > > >>>
>> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
>> > > wesmckinn@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>> >
>> > > > > > >>> > hi,
>> > > > > > >>> >
>> > > > > > >>> > I think we should try to release the week of September 9, so
>> > > > > > >>> > development work should be completed by end of next week.
>> > > > > > >>> >
>> > > > > > >>> > Does that seem reasonable?
>> > > > > > >>> >
>> > > > > > >>> > I plan to get up a patch for the protocol alignment changes for
>> > > > > > >>> > C++ in the next couple of days -- I think that getting the
>> > > > > > >>> > alignment work done is the main barrier to releasing.
>> > > > > > >>> >
>> > > > > > >>> > Thanks
>> > > > > > >>> > Wes
>> > > > > > >>> >
>> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
>> > > > > > >>> > <ni...@aliyun.com.invalid>
>> > > > > > >>> wrote:
>> > > > > > >>> > >
>> > > > > > >>> > > Hi, Wes, on the java side, I can think of several bugs that
>> > > > need
>> > > > > > >>> > > to
>> > > > > > >>> be fixed or reminded.
>> > > > > > >>> > >
>> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
>> > > > > > >>> > > even
>> > > > > > >>> when empty[1]
>> > > > > > >>> > > This one is under review now, however through this PR we find
>> > > > > > >>> > > that
>> > > > > > >>> there seems a bug in java reading and writing dictionaries in IPC
>> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
>> > > > dictionaries
>> > > > > > >>> are at the start of stream (see details in PR comments, and this
>> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
>> > > > > > >>> > >
>> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration
>> > > > test
>> > > > > > >>> JSON files[3]
>> > > > > > >>> > > Java side code already checked in, other implementations
>> > > seems
>> > > > not.
>> > > > > > >>> > >
>> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by
>> > > trying
>> > > > > > >>> > > to load all records in one contiguous batch, fixed
>> > > > > > >>> by providing iterator API for iteratively reading in
>> > > ARROW-6219[5].
>> > > > > > >>> > >
>> > > > > > >>> > > Thanks,
>> > > > > > >>> > > Ji Liu
>> > > > > > >>> > >
>> > > > > > >>> > > [1]
>> > > > > > >>> > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > >>> > >
>> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>> > > > > > >>> > > ric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d736678a45%7
>> > > > > > >>> > >
>> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>> > > > > > >>> > >
>> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>> > > > > > >>> > > reserved=0 [2]
>> > > > > > >>> > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > >>> > > 2Farrow.apache.org
>> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>> > > > > > >>> > > ardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > > > > > >>> > >
>> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>> > > > > > >>> > >
>> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>> > > > > > >>> > > d=0 [3]
>> > > > > > >>> > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > >>> > > 2Fissues.apache.org
>> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d736678
>> > > > > > >>> > >
>> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>> > > > > > >>> > >
>> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>> > > > > > >>> > > ;reserved=0 [4]
>> > > > > > >>> > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>> > > > > > >>> > > 2Fissues.apache.org
>> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d73
>> > > > > > >>> > >
>> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>> > > > > > >>> > >
>> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>> > > > > > >>> > > &reserved=0]
>> > > > > > >>>
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>> > > > > > >>> sues.apache.org
>> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>> > > > > > >>> .Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
>> > > > > > >>>
>> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>> > > > > > >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>> > > > > > >>> > >
>> > > > > > >>> > >
>> > > > > > >>> > >
>> > > > > > >>> > >
>> > > > ----------------------------------------------------------------
>> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
>> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
>> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
>> > > > > > >>> > >
>> > > > > > >>> > > I'm going to work some on organizing the 0.15.0 backlog some
>> > > > > > >>> > > this week, if anyone wants to help with grooming
>> > > (particularly
>> > > > > > >>> > > for languages other than C++/Python where I'm focusing) that
>> > > > > > >>> > > would be helpful. There have been almost 500 JIRA issues
>> > > opened
>> > > > > > >>> > > since the
>> > > > > > >>> > > 0.14.0 release, so we should make sure to check whether
>> > > there's
>> > > > > > >>> > > any regressions or other serious bugs that we should try to
>> > > fix
>> > > > > > >>> > > for 0.15.0.
>> > > > > > >>> > >
>> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
>> > > > > > >>> > > <we...@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>> > > >
>> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
>> > > > > > >>> > > >
>> > > > > > >>> > > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > >>> > > > F%2Fissues.apache.org
>> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
>> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d
>> > > > > > >>> > > >
>> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> > > > > > >>> > > >
>> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>> > > > > > >>> > > > WEWg0%3D&reserved=0
>> > > > > > >>> > > >
>> > > > > > >>> > > > I think the root cause could be the Windows changes in
>> > > > > > >>> > > >
>> > > > > > >>> > > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > >>> > > >
>> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
>> > > > %7Ccbead81a42104034a4f308d736678a
>> > > > > > >>> > > >
>> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>> > > > > > >>> > > >
>> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>> > > > > > >>> > > > bs%3D&reserved=0
>> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>> > > > > > >>> > > >
>> > > > > > >>> > > > I would be appreciative if a volunteer would look into what
>> > > > > > >>> > > > was
>> > > > > > >>> wrong
>> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
>> > > > > > >>> > > > wheels will be broken, too
>> > > > > > >>> > > >
>> > > > > > >>> > > > The bad wheels can be found at
>> > > > > > >>> > > >
>> > > > > > >>> > > >
>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> > > > > > >>> > > >
>> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com
>> > > > %7Ccbea
>> > > > > > >>> > > >
>> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>> > > > > > >>> > > >
>> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>> > > > > > >>> > > >
>> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
>> > > > > > >>> solipsis@pitrou.net> wrote:
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
>> > > > > > >>> > > > > <em...@gmail.com> wrote:
>> > > > > > >>> > > > > > >
>> > > > > > >>> > > > > > > In C++ they are
>> > > > > > >>> > > > > > > independent, we could have 32-bit array lengths and
>> > > > > > >>> variable-length
>> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we just
>> > > > wouldn't
>> > > > > > >>> > > > > > > be
>> > > > > > >>> able to
>> > > > > > >>> > > > > > > have a List child with more than INT32_MAX elements).
>> > > > > > >>> > > > > >
>> > > > > > >>> > > > > > I think the point is we could do this in C++ but we
>> > > > don't.
>> > > > > > >>> I'm not sure we
>> > > > > > >>> > > > > > would have introduced the "Large" types if we did.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
>> > > offsets,
>> > > > > > >>> > > > > so if
>> > > > > > >>> you're
>> > > > > > >>> > > > > storing lots of small-ish lists or strings, 32-bit
>> > > offsets
>> > > > > > >>> > > > > are preferrable. So even with 64-bit array lengths from
>> > > > the
>> > > > > > >>> > > > > start
>> > > > > > >>> it would
>> > > > > > >>> > > > > still be beneficial to have types with 32-bit offsets.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > > Going with the limited address space in Java and
>> > > calling
>> > > > > > >>> > > > > > it a
>> > > > > > >>> reference
>> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer uses a
>> > > > "Large"
>> > > > > > >>> type
>> > > > > > >>> > > > > > presumably it is because they need the ability to store
>> > > > > > >>> > > > > > more
>> > > > > > >>> than INT32_MAX
>> > > > > > >>> > > > > > child elements in a column, otherwise it is just
>> > > wasting
>> > > > > > >>> > > > > > space
>> > > > > > >>> [1].
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > Probably. Though if the individual elements (lists or
>> > > > > > >>> > > > > strings)
>> > > > > > >>> are
>> > > > > > >>> > > > > large, not much space is wasted in proportion, so it may
>> > > be
>> > > > > > >>> simpler in
>> > > > > > >>> > > > > such a case to always create a "Large" type array.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
>> > > > > > >>> > > > > > performance
>> > > > > > >>> benefits on
>> > > > > > >>> > > > > > 64-bit architectures to using the native word sizes.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do that, as
>> > > > > > >>> > > > > 32-bit
>> > > > > > >>> is an
>> > > > > > >>> > > > > extremely common integer size even in high-performance
>> > > > code.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > Regards
>> > > > > > >>> > > > >
>> > > > > > >>> > > > > Antoine.
>> > > > > > >>> > > > >
>> > > > > > >>> > > > >
>> > > > > > >>> > >
>> > > > > > >>>
>> > > > > > >>
>> > > >
>> > >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
>
> The process should be well documented at this point but there are a
> number of steps.
Is [1] the up-to-date documentation for the release? Are there
instructions for the adding the code signing Key to SVN?
I will make a go of it. i will try to mitigate any internet issues by
doing the process for a cloud instance (I assume that isn't a problem?).
Thanks,
Micah
[1]
https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <we...@gmail.com> wrote:
> The process should be well documented at this point but there are a
> number of steps. Note that you need to add your code signing key to
> the KEYS file in SVN (that's not very hard to do). I think it's fine
> to hand off the process to others after the VOTE but it would be
> tricky to have multiple RMs involved with producing the source and
> binary artifacts for the vote
>
> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > SGTM, as well.
> >
> > I should have a little bit of time next week if I can help as RM but I
> have
> > a couple of concerns:
> > 1. In the past I've had trouble downloading and validating releases.
> I'm a
> > bit worried, that I might have similar problems doing the necessary
> uploads.
> > 2. My internet connection will likely be not great, I don't know if this
> > would make it even less likely to be successful.
> >
> > Does it become problematic if somehow I would have to abandon the process
> > mid-release? Is there anyone who could serve as a backup? Are the steps
> > well documented?
> >
> > Thanks,
> > Micah
> >
> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <
> neal.p.richardson@gmail.com>
> > wrote:
> >
> > > Sounds good to me.
> > >
> > > Do we have a release manager yet? Any volunteers?
> > >
> > > Neal
> > >
> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > hi all,
> > > >
> > > > It looks like we're drawing close to be able to make the 0.15.0
> > > > release. I would suggest "pencils down" at the end of this week and
> > > > see if a release candidate can be produced next Monday September 23.
> > > > Any thoughts or objections?
> > > >
> > > > Thanks,
> > > > Wes
> > > >
> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > hi Eric -- yes, that's correct. I'm planning to amend the Format
> docs
> > > > > today regarding the EOS issue and also update the C++ library
> > > > >
> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > > <Er...@microsoft.com> wrote:
> > > > > >
> > > > > > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment
> > > > branch into master before the 0.15 release, correct?
> > > > > >
> > > > > > BTW - I believe the C# alignment changes are ready to be merged
> into
> > > > the alignment branch - https://github.com/apache/arrow/pull/5280/
> > > > > >
> > > > > > Eric
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Micah Kornfield <em...@gmail.com>
> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > > > To: Wes McKinney <we...@gmail.com>
> > > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > > > > > Subject: Re: Timeline for 0.15.0 release
> > > > > >
> > > > > > I should have a little more bandwidth to help with some of the
> > > > packaging starting tomorrow and going into the weekend.
> > > > > >
> > > > > > On Tuesday, September 10, 2019, Wes McKinney <
> wesmckinn@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > With the state of nightly packaging and integration builds
> things
> > > > > > > aren't looking too good for being in release readiness by the
> end
> > > of
> > > > > > > this week but maybe I'm wrong. I'm planning to be working to
> close
> > > as
> > > > > > > many issues as I can and also to help with the ongoing
> alignment
> > > > fixes.
> > > > > > >
> > > > > > > Wes
> > > > > > >
> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > > emkornfield@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Just for reference [1] has a dashboard of the current issues:
> > > > > > >>
> > > > > > >>
> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > > >> ki.apache.org
> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034
> > > > > > >>
> > > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > >>
> > > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > > >> %3D&reserved=0
> > > > > > >>
> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >>> hi all,
> > > > > > >>>
> > > > > > >>> It doesn't seem like we're going to be in a position to
> release
> > > at
> > > > > > >>> the beginning of next week. I hope that one more week of
> work (or
> > > > > > >>> less) will be enough to get us there. Aside from merging the
> > > > > > >>> alignment changes, we need to make sure that our packaging
> jobs
> > > > > > >>> required for the release candidate are all working.
> > > > > > >>>
> > > > > > >>> If folks could remove issues from the 0.15.0 backlog that
> they
> > > > don't
> > > > > > >>> think they will finish by end of next week that would help
> focus
> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> > > > > > >>> looking to tackle a few small features related to
> dictionaries
> > > > while
> > > > > > >>> the release window is still open.
> > > > > > >>>
> > > > > > >>> - Wes
> > > > > > >>>
> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > hi,
> > > > > > >>> >
> > > > > > >>> > I think we should try to release the week of September 9,
> so
> > > > > > >>> > development work should be completed by end of next week.
> > > > > > >>> >
> > > > > > >>> > Does that seem reasonable?
> > > > > > >>> >
> > > > > > >>> > I plan to get up a patch for the protocol alignment
> changes for
> > > > > > >>> > C++ in the next couple of days -- I think that getting the
> > > > > > >>> > alignment work done is the main barrier to releasing.
> > > > > > >>> >
> > > > > > >>> > Thanks
> > > > > > >>> > Wes
> > > > > > >>> >
> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > > >>> wrote:
> > > > > > >>> > >
> > > > > > >>> > > Hi, Wes, on the java side, I can think of several bugs
> that
> > > > need
> > > > > > >>> > > to
> > > > > > >>> be fixed or reminded.
> > > > > > >>> > >
> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC
> streams
> > > > > > >>> > > even
> > > > > > >>> when empty[1]
> > > > > > >>> > > This one is under review now, however through this PR we
> find
> > > > > > >>> > > that
> > > > > > >>> there seems a bug in java reading and writing dictionaries
> in IPC
> > > > > > >>> which is Inconsistent with spec[2] since it assumes all
> > > > dictionaries
> > > > > > >>> are at the start of stream (see details in PR comments, and
> this
> > > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > > > > > >>> > >
> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in
> integration
> > > > test
> > > > > > >>> JSON files[3]
> > > > > > >>> > > Java side code already checked in, other implementations
> > > seems
> > > > not.
> > > > > > >>> > >
> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by
> > > trying
> > > > > > >>> > > to load all records in one contiguous batch, fixed
> > > > > > >>> by providing iterator API for iteratively reading in
> > > ARROW-6219[5].
> > > > > > >>> > >
> > > > > > >>> > > Thanks,
> > > > > > >>> > > Ji Liu
> > > > > > >>> > >
> > > > > > >>> > > [1]
> > > > > > >>> > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >>> > >
> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > > >>> > > ric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > > >>> > >
> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > > >>> > >
> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > > >>> > > reserved=0 [2]
> > > > > > >>> > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >>> > > 2Farrow.apache.org
> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > > >>> > > ardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > >>> > >
> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > > >>> > >
> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > > >>> > > d=0 [3]
> > > > > > >>> > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >>> > > 2Fissues.apache.org
> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d736678
> > > > > > >>> > >
> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > > >>> > >
> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > > >>> > > ;reserved=0 [4]
> > > > > > >>> > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > > >>> > > 2Fissues.apache.org
> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d73
> > > > > > >>> > >
> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > > >>> > >
> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > > >>> > > &reserved=0]
> > > > > > >>>
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > > >>> sues.apache.org
> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > > >>> .Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > > >>>
> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > > >>>
> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > ----------------------------------------------------------------
> > > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > > > > >>> > >
> > > > > > >>> > > I'm going to work some on organizing the 0.15.0 backlog
> some
> > > > > > >>> > > this week, if anyone wants to help with grooming
> > > (particularly
> > > > > > >>> > > for languages other than C++/Python where I'm focusing)
> that
> > > > > > >>> > > would be helpful. There have been almost 500 JIRA issues
> > > opened
> > > > > > >>> > > since the
> > > > > > >>> > > 0.14.0 release, so we should make sure to check whether
> > > there's
> > > > > > >>> > > any regressions or other serious bugs that we should try
> to
> > > fix
> > > > > > >>> > > for 0.15.0.
> > > > > > >>> > >
> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > > > > > >>> > > <we...@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>> > > >
> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > > > > > >>> > > >
> > > > > > >>> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >>> > > > F%2Fissues.apache.org
> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d
> > > > > > >>> > > >
> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > > >>> > > >
> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > > >>> > > >
> > > > > > >>> > > > I think the root cause could be the Windows changes in
> > > > > > >>> > > >
> > > > > > >>> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >>> > > >
> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > > %7Ccbead81a42104034a4f308d736678a
> > > > > > >>> > > >
> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > > >>> > > >
> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > > >>> > > > bs%3D&reserved=0
> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > > >>> > > >
> > > > > > >>> > > > I would be appreciative if a volunteer would look into
> what
> > > > > > >>> > > > was
> > > > > > >>> wrong
> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0
> Windows
> > > > > > >>> > > > wheels will be broken, too
> > > > > > >>> > > >
> > > > > > >>> > > > The bad wheels can be found at
> > > > > > >>> > > >
> > > > > > >>> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > >>> > > >
> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%
> 40microsoft.com
> > > > %7Ccbea
> > > > > > >>> > > >
> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > > >>> > > >
> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > > >>> > > >
> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > > > > > >>> solipsis@pitrou.net> wrote:
> > > > > > >>> > > > >
> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> > > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > In C++ they are
> > > > > > >>> > > > > > > independent, we could have 32-bit array lengths
> and
> > > > > > >>> variable-length
> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we just
> > > > wouldn't
> > > > > > >>> > > > > > > be
> > > > > > >>> able to
> > > > > > >>> > > > > > > have a List child with more than INT32_MAX
> elements).
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > I think the point is we could do this in C++ but we
> > > > don't.
> > > > > > >>> I'm not sure we
> > > > > > >>> > > > > > would have introduced the "Large" types if we did.
> > > > > > >>> > > > >
> > > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> > > offsets,
> > > > > > >>> > > > > so if
> > > > > > >>> you're
> > > > > > >>> > > > > storing lots of small-ish lists or strings, 32-bit
> > > offsets
> > > > > > >>> > > > > are preferrable. So even with 64-bit array lengths
> from
> > > > the
> > > > > > >>> > > > > start
> > > > > > >>> it would
> > > > > > >>> > > > > still be beneficial to have types with 32-bit
> offsets.
> > > > > > >>> > > > >
> > > > > > >>> > > > > > Going with the limited address space in Java and
> > > calling
> > > > > > >>> > > > > > it a
> > > > > > >>> reference
> > > > > > >>> > > > > > implementation seems suboptimal. If a consumer
> uses a
> > > > "Large"
> > > > > > >>> type
> > > > > > >>> > > > > > presumably it is because they need the ability to
> store
> > > > > > >>> > > > > > more
> > > > > > >>> than INT32_MAX
> > > > > > >>> > > > > > child elements in a column, otherwise it is just
> > > wasting
> > > > > > >>> > > > > > space
> > > > > > >>> [1].
> > > > > > >>> > > > >
> > > > > > >>> > > > > Probably. Though if the individual elements (lists or
> > > > > > >>> > > > > strings)
> > > > > > >>> are
> > > > > > >>> > > > > large, not much space is wasted in proportion, so it
> may
> > > be
> > > > > > >>> simpler in
> > > > > > >>> > > > > such a case to always create a "Large" type array.
> > > > > > >>> > > > >
> > > > > > >>> > > > > > [1] I suppose theoretically there might be some
> > > > > > >>> > > > > > performance
> > > > > > >>> benefits on
> > > > > > >>> > > > > > 64-bit architectures to using the native word
> sizes.
> > > > > > >>> > > > >
> > > > > > >>> > > > > Concretely, common 64-bit architectures don't do
> that, as
> > > > > > >>> > > > > 32-bit
> > > > > > >>> is an
> > > > > > >>> > > > > extremely common integer size even in
> high-performance
> > > > code.
> > > > > > >>> > > > >
> > > > > > >>> > > > > Regards
> > > > > > >>> > > > >
> > > > > > >>> > > > > Antoine.
> > > > > > >>> > > > >
> > > > > > >>> > > > >
> > > > > > >>> > >
> > > > > > >>>
> > > > > > >>
> > > >
> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
The process should be well documented at this point but there are a
number of steps. Note that you need to add your code signing key to
the KEYS file in SVN (that's not very hard to do). I think it's fine
to hand off the process to others after the VOTE but it would be
tricky to have multiple RMs involved with producing the source and
binary artifacts for the vote
On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield <em...@gmail.com> wrote:
>
> SGTM, as well.
>
> I should have a little bit of time next week if I can help as RM but I have
> a couple of concerns:
> 1. In the past I've had trouble downloading and validating releases. I'm a
> bit worried, that I might have similar problems doing the necessary uploads.
> 2. My internet connection will likely be not great, I don't know if this
> would make it even less likely to be successful.
>
> Does it become problematic if somehow I would have to abandon the process
> mid-release? Is there anyone who could serve as a backup? Are the steps
> well documented?
>
> Thanks,
> Micah
>
> On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <ne...@gmail.com>
> wrote:
>
> > Sounds good to me.
> >
> > Do we have a release manager yet? Any volunteers?
> >
> > Neal
> >
> > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi all,
> > >
> > > It looks like we're drawing close to be able to make the 0.15.0
> > > release. I would suggest "pencils down" at the end of this week and
> > > see if a release candidate can be produced next Monday September 23.
> > > Any thoughts or objections?
> > >
> > > Thanks,
> > > Wes
> > >
> > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > hi Eric -- yes, that's correct. I'm planning to amend the Format docs
> > > > today regarding the EOS issue and also update the C++ library
> > > >
> > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > > <Er...@microsoft.com> wrote:
> > > > >
> > > > > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment
> > > branch into master before the 0.15 release, correct?
> > > > >
> > > > > BTW - I believe the C# alignment changes are ready to be merged into
> > > the alignment branch - https://github.com/apache/arrow/pull/5280/
> > > > >
> > > > > Eric
> > > > >
> > > > > -----Original Message-----
> > > > > From: Micah Kornfield <em...@gmail.com>
> > > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > > To: Wes McKinney <we...@gmail.com>
> > > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > > > > Subject: Re: Timeline for 0.15.0 release
> > > > >
> > > > > I should have a little more bandwidth to help with some of the
> > > packaging starting tomorrow and going into the weekend.
> > > > >
> > > > > On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > With the state of nightly packaging and integration builds things
> > > > > > aren't looking too good for being in release readiness by the end
> > of
> > > > > > this week but maybe I'm wrong. I'm planning to be working to close
> > as
> > > > > > many issues as I can and also to help with the ongoing alignment
> > > fixes.
> > > > > >
> > > > > > Wes
> > > > > >
> > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> > emkornfield@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Just for reference [1] has a dashboard of the current issues:
> > > > > >>
> > > > > >>
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > > >> ki.apache.org
> > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034
> > > > > >>
> > > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > >>
> > > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > > >> %3D&reserved=0
> > > > > >>
> > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >>> hi all,
> > > > > >>>
> > > > > >>> It doesn't seem like we're going to be in a position to release
> > at
> > > > > >>> the beginning of next week. I hope that one more week of work (or
> > > > > >>> less) will be enough to get us there. Aside from merging the
> > > > > >>> alignment changes, we need to make sure that our packaging jobs
> > > > > >>> required for the release candidate are all working.
> > > > > >>>
> > > > > >>> If folks could remove issues from the 0.15.0 backlog that they
> > > don't
> > > > > >>> think they will finish by end of next week that would help focus
> > > > > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> > > > > >>> looking to tackle a few small features related to dictionaries
> > > while
> > > > > >>> the release window is still open.
> > > > > >>>
> > > > > >>> - Wes
> > > > > >>>
> > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> > hi,
> > > > > >>> >
> > > > > >>> > I think we should try to release the week of September 9, so
> > > > > >>> > development work should be completed by end of next week.
> > > > > >>> >
> > > > > >>> > Does that seem reasonable?
> > > > > >>> >
> > > > > >>> > I plan to get up a patch for the protocol alignment changes for
> > > > > >>> > C++ in the next couple of days -- I think that getting the
> > > > > >>> > alignment work done is the main barrier to releasing.
> > > > > >>> >
> > > > > >>> > Thanks
> > > > > >>> > Wes
> > > > > >>> >
> > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > > >>> > <ni...@aliyun.com.invalid>
> > > > > >>> wrote:
> > > > > >>> > >
> > > > > >>> > > Hi, Wes, on the java side, I can think of several bugs that
> > > need
> > > > > >>> > > to
> > > > > >>> be fixed or reminded.
> > > > > >>> > >
> > > > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
> > > > > >>> > > even
> > > > > >>> when empty[1]
> > > > > >>> > > This one is under review now, however through this PR we find
> > > > > >>> > > that
> > > > > >>> there seems a bug in java reading and writing dictionaries in IPC
> > > > > >>> which is Inconsistent with spec[2] since it assumes all
> > > dictionaries
> > > > > >>> are at the start of stream (see details in PR comments, and this
> > > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > > > > >>> > >
> > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration
> > > test
> > > > > >>> JSON files[3]
> > > > > >>> > > Java side code already checked in, other implementations
> > seems
> > > not.
> > > > > >>> > >
> > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by
> > trying
> > > > > >>> > > to load all records in one contiguous batch, fixed
> > > > > >>> by providing iterator API for iteratively reading in
> > ARROW-6219[5].
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Ji Liu
> > > > > >>> > >
> > > > > >>> > > [1]
> > > > > >>> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >>> > >
> > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > > >>> > > ric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d736678a45%7
> > > > > >>> > >
> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > > >>> > >
> > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > > >>> > > reserved=0 [2]
> > > > > >>> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >>> > > 2Farrow.apache.org
> > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > > >>> > > ardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > >>> > >
> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > > >>> > >
> > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > > >>> > > d=0 [3]
> > > > > >>> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >>> > > 2Fissues.apache.org
> > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d736678
> > > > > >>> > >
> > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > > >>> > >
> > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > > >>> > > ;reserved=0 [4]
> > > > > >>> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > >>> > > 2Fissues.apache.org
> > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d73
> > > > > >>> > >
> > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > > >>> > >
> > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > > >>> > > &reserved=0]
> > > > > >>>
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > > >>> sues.apache.org
> > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > > >>> .Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > > >>>
> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > > >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > ----------------------------------------------------------------
> > > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> > > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > > > >>> > >
> > > > > >>> > > I'm going to work some on organizing the 0.15.0 backlog some
> > > > > >>> > > this week, if anyone wants to help with grooming
> > (particularly
> > > > > >>> > > for languages other than C++/Python where I'm focusing) that
> > > > > >>> > > would be helpful. There have been almost 500 JIRA issues
> > opened
> > > > > >>> > > since the
> > > > > >>> > > 0.14.0 release, so we should make sure to check whether
> > there's
> > > > > >>> > > any regressions or other serious bugs that we should try to
> > fix
> > > > > >>> > > for 0.15.0.
> > > > > >>> > >
> > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > > > > >>> > > <we...@gmail.com>
> > > > > >>> wrote:
> > > > > >>> > > >
> > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > > > > >>> > > >
> > > > > >>> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >>> > > > F%2Fissues.apache.org
> > > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d
> > > > > >>> > > >
> > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > > >>> > > >
> > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > > >>> > > > WEWg0%3D&reserved=0
> > > > > >>> > > >
> > > > > >>> > > > I think the root cause could be the Windows changes in
> > > > > >>> > > >
> > > > > >>> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >>> > > >
> > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > > >>> > > > CEric.Erhardt%40microsoft.com
> > > %7Ccbead81a42104034a4f308d736678a
> > > > > >>> > > >
> > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > > >>> > > >
> > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > > >>> > > > bs%3D&reserved=0
> > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > > >>> > > >
> > > > > >>> > > > I would be appreciative if a volunteer would look into what
> > > > > >>> > > > was
> > > > > >>> wrong
> > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
> > > > > >>> > > > wheels will be broken, too
> > > > > >>> > > >
> > > > > >>> > > > The bad wheels can be found at
> > > > > >>> > > >
> > > > > >>> > > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > >>> > > >
> > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > > %7Ccbea
> > > > > >>> > > >
> > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > > >>> > > >
> > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > > >>> > > >
> > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > > > > >>> solipsis@pitrou.net> wrote:
> > > > > >>> > > > >
> > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> > > > > >>> > > > > <em...@gmail.com> wrote:
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > In C++ they are
> > > > > >>> > > > > > > independent, we could have 32-bit array lengths and
> > > > > >>> variable-length
> > > > > >>> > > > > > > types with 64-bit offsets if we wanted (we just
> > > wouldn't
> > > > > >>> > > > > > > be
> > > > > >>> able to
> > > > > >>> > > > > > > have a List child with more than INT32_MAX elements).
> > > > > >>> > > > > >
> > > > > >>> > > > > > I think the point is we could do this in C++ but we
> > > don't.
> > > > > >>> I'm not sure we
> > > > > >>> > > > > > would have introduced the "Large" types if we did.
> > > > > >>> > > > >
> > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> > offsets,
> > > > > >>> > > > > so if
> > > > > >>> you're
> > > > > >>> > > > > storing lots of small-ish lists or strings, 32-bit
> > offsets
> > > > > >>> > > > > are preferrable. So even with 64-bit array lengths from
> > > the
> > > > > >>> > > > > start
> > > > > >>> it would
> > > > > >>> > > > > still be beneficial to have types with 32-bit offsets.
> > > > > >>> > > > >
> > > > > >>> > > > > > Going with the limited address space in Java and
> > calling
> > > > > >>> > > > > > it a
> > > > > >>> reference
> > > > > >>> > > > > > implementation seems suboptimal. If a consumer uses a
> > > "Large"
> > > > > >>> type
> > > > > >>> > > > > > presumably it is because they need the ability to store
> > > > > >>> > > > > > more
> > > > > >>> than INT32_MAX
> > > > > >>> > > > > > child elements in a column, otherwise it is just
> > wasting
> > > > > >>> > > > > > space
> > > > > >>> [1].
> > > > > >>> > > > >
> > > > > >>> > > > > Probably. Though if the individual elements (lists or
> > > > > >>> > > > > strings)
> > > > > >>> are
> > > > > >>> > > > > large, not much space is wasted in proportion, so it may
> > be
> > > > > >>> simpler in
> > > > > >>> > > > > such a case to always create a "Large" type array.
> > > > > >>> > > > >
> > > > > >>> > > > > > [1] I suppose theoretically there might be some
> > > > > >>> > > > > > performance
> > > > > >>> benefits on
> > > > > >>> > > > > > 64-bit architectures to using the native word sizes.
> > > > > >>> > > > >
> > > > > >>> > > > > Concretely, common 64-bit architectures don't do that, as
> > > > > >>> > > > > 32-bit
> > > > > >>> is an
> > > > > >>> > > > > extremely common integer size even in high-performance
> > > code.
> > > > > >>> > > > >
> > > > > >>> > > > > Regards
> > > > > >>> > > > >
> > > > > >>> > > > > Antoine.
> > > > > >>> > > > >
> > > > > >>> > > > >
> > > > > >>> > >
> > > > > >>>
> > > > > >>
> > >
> >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
SGTM, as well.
I should have a little bit of time next week if I can help as RM but I have
a couple of concerns:
1. In the past I've had trouble downloading and validating releases. I'm a
bit worried, that I might have similar problems doing the necessary uploads.
2. My internet connection will likely be not great, I don't know if this
would make it even less likely to be successful.
Does it become problematic if somehow I would have to abandon the process
mid-release? Is there anyone who could serve as a backup? Are the steps
well documented?
Thanks,
Micah
On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson <ne...@gmail.com>
wrote:
> Sounds good to me.
>
> Do we have a release manager yet? Any volunteers?
>
> Neal
>
> On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi all,
> >
> > It looks like we're drawing close to be able to make the 0.15.0
> > release. I would suggest "pencils down" at the end of this week and
> > see if a release candidate can be produced next Monday September 23.
> > Any thoughts or objections?
> >
> > Thanks,
> > Wes
> >
> > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > hi Eric -- yes, that's correct. I'm planning to amend the Format docs
> > > today regarding the EOS issue and also update the C++ library
> > >
> > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > > <Er...@microsoft.com> wrote:
> > > >
> > > > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment
> > branch into master before the 0.15 release, correct?
> > > >
> > > > BTW - I believe the C# alignment changes are ready to be merged into
> > the alignment branch - https://github.com/apache/arrow/pull/5280/
> > > >
> > > > Eric
> > > >
> > > > -----Original Message-----
> > > > From: Micah Kornfield <em...@gmail.com>
> > > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > > To: Wes McKinney <we...@gmail.com>
> > > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > > > Subject: Re: Timeline for 0.15.0 release
> > > >
> > > > I should have a little more bandwidth to help with some of the
> > packaging starting tomorrow and going into the weekend.
> > > >
> > > > On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > With the state of nightly packaging and integration builds things
> > > > > aren't looking too good for being in release readiness by the end
> of
> > > > > this week but maybe I'm wrong. I'm planning to be working to close
> as
> > > > > many issues as I can and also to help with the ongoing alignment
> > fixes.
> > > > >
> > > > > Wes
> > > > >
> > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <
> emkornfield@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Just for reference [1] has a dashboard of the current issues:
> > > > >>
> > > > >>
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > > >> ki.apache.org
> > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034
> > > > >>
> > a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > >>
> > 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > > >> %3D&reserved=0
> > > > >>
> > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > > >>
> > > > >>> hi all,
> > > > >>>
> > > > >>> It doesn't seem like we're going to be in a position to release
> at
> > > > >>> the beginning of next week. I hope that one more week of work (or
> > > > >>> less) will be enough to get us there. Aside from merging the
> > > > >>> alignment changes, we need to make sure that our packaging jobs
> > > > >>> required for the release candidate are all working.
> > > > >>>
> > > > >>> If folks could remove issues from the 0.15.0 backlog that they
> > don't
> > > > >>> think they will finish by end of next week that would help focus
> > > > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> > > > >>> looking to tackle a few small features related to dictionaries
> > while
> > > > >>> the release window is still open.
> > > > >>>
> > > > >>> - Wes
> > > > >>>
> > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > hi,
> > > > >>> >
> > > > >>> > I think we should try to release the week of September 9, so
> > > > >>> > development work should be completed by end of next week.
> > > > >>> >
> > > > >>> > Does that seem reasonable?
> > > > >>> >
> > > > >>> > I plan to get up a patch for the protocol alignment changes for
> > > > >>> > C++ in the next couple of days -- I think that getting the
> > > > >>> > alignment work done is the main barrier to releasing.
> > > > >>> >
> > > > >>> > Thanks
> > > > >>> > Wes
> > > > >>> >
> > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > > >>> > <ni...@aliyun.com.invalid>
> > > > >>> wrote:
> > > > >>> > >
> > > > >>> > > Hi, Wes, on the java side, I can think of several bugs that
> > need
> > > > >>> > > to
> > > > >>> be fixed or reminded.
> > > > >>> > >
> > > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
> > > > >>> > > even
> > > > >>> when empty[1]
> > > > >>> > > This one is under review now, however through this PR we find
> > > > >>> > > that
> > > > >>> there seems a bug in java reading and writing dictionaries in IPC
> > > > >>> which is Inconsistent with spec[2] since it assumes all
> > dictionaries
> > > > >>> are at the start of stream (see details in PR comments, and this
> > > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > > > >>> > >
> > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration
> > test
> > > > >>> JSON files[3]
> > > > >>> > > Java side code already checked in, other implementations
> seems
> > not.
> > > > >>> > >
> > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by
> trying
> > > > >>> > > to load all records in one contiguous batch, fixed
> > > > >>> by providing iterator API for iteratively reading in
> ARROW-6219[5].
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Ji Liu
> > > > >>> > >
> > > > >>> > > [1]
> > > > >>> > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >>> > >
> > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > > >>> > > ric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d736678a45%7
> > > > >>> > >
> > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > > >>> > >
> > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > > >>> > > reserved=0 [2]
> > > > >>> > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >>> > > 2Farrow.apache.org
> > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > > >>> > > ardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > >>> > >
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > > >>> > >
> > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > > >>> > > d=0 [3]
> > > > >>> > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >>> > > 2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d736678
> > > > >>> > >
> > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > > >>> > >
> > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > > >>> > > ;reserved=0 [4]
> > > > >>> > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > >>> > > 2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d73
> > > > >>> > >
> > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > > >>> > >
> > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > > >>> > > &reserved=0]
> > > > >>>
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > > >>> sues.apache.org
> > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > > >>> .Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > > >>>
> > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > > >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > ----------------------------------------------------------------
> > > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> > > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > > >>> > >
> > > > >>> > > I'm going to work some on organizing the 0.15.0 backlog some
> > > > >>> > > this week, if anyone wants to help with grooming
> (particularly
> > > > >>> > > for languages other than C++/Python where I'm focusing) that
> > > > >>> > > would be helpful. There have been almost 500 JIRA issues
> opened
> > > > >>> > > since the
> > > > >>> > > 0.14.0 release, so we should make sure to check whether
> there's
> > > > >>> > > any regressions or other serious bugs that we should try to
> fix
> > > > >>> > > for 0.15.0.
> > > > >>> > >
> > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > > > >>> > > <we...@gmail.com>
> > > > >>> wrote:
> > > > >>> > > >
> > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > > > >>> > > >
> > > > >>> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >>> > > > F%2Fissues.apache.org
> > %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d
> > > > >>> > > >
> > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > > >>> > > >
> > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > > >>> > > > WEWg0%3D&reserved=0
> > > > >>> > > >
> > > > >>> > > > I think the root cause could be the Windows changes in
> > > > >>> > > >
> > > > >>> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >>> > > >
> > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > > >>> > > > CEric.Erhardt%40microsoft.com
> > %7Ccbead81a42104034a4f308d736678a
> > > > >>> > > >
> > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > > >>> > > >
> > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > > >>> > > > bs%3D&reserved=0
> > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > > >>> > > >
> > > > >>> > > > I would be appreciative if a volunteer would look into what
> > > > >>> > > > was
> > > > >>> wrong
> > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
> > > > >>> > > > wheels will be broken, too
> > > > >>> > > >
> > > > >>> > > > The bad wheels can be found at
> > > > >>> > > >
> > > > >>> > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > >>> > > >
> > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com
> > %7Ccbea
> > > > >>> > > >
> > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > > >>> > > >
> > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > > >>> > > >
> > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > > > >>> solipsis@pitrou.net> wrote:
> > > > >>> > > > >
> > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> > > > >>> > > > > <em...@gmail.com> wrote:
> > > > >>> > > > > > >
> > > > >>> > > > > > > In C++ they are
> > > > >>> > > > > > > independent, we could have 32-bit array lengths and
> > > > >>> variable-length
> > > > >>> > > > > > > types with 64-bit offsets if we wanted (we just
> > wouldn't
> > > > >>> > > > > > > be
> > > > >>> able to
> > > > >>> > > > > > > have a List child with more than INT32_MAX elements).
> > > > >>> > > > > >
> > > > >>> > > > > > I think the point is we could do this in C++ but we
> > don't.
> > > > >>> I'm not sure we
> > > > >>> > > > > > would have introduced the "Large" types if we did.
> > > > >>> > > > >
> > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit
> offsets,
> > > > >>> > > > > so if
> > > > >>> you're
> > > > >>> > > > > storing lots of small-ish lists or strings, 32-bit
> offsets
> > > > >>> > > > > are preferrable. So even with 64-bit array lengths from
> > the
> > > > >>> > > > > start
> > > > >>> it would
> > > > >>> > > > > still be beneficial to have types with 32-bit offsets.
> > > > >>> > > > >
> > > > >>> > > > > > Going with the limited address space in Java and
> calling
> > > > >>> > > > > > it a
> > > > >>> reference
> > > > >>> > > > > > implementation seems suboptimal. If a consumer uses a
> > "Large"
> > > > >>> type
> > > > >>> > > > > > presumably it is because they need the ability to store
> > > > >>> > > > > > more
> > > > >>> than INT32_MAX
> > > > >>> > > > > > child elements in a column, otherwise it is just
> wasting
> > > > >>> > > > > > space
> > > > >>> [1].
> > > > >>> > > > >
> > > > >>> > > > > Probably. Though if the individual elements (lists or
> > > > >>> > > > > strings)
> > > > >>> are
> > > > >>> > > > > large, not much space is wasted in proportion, so it may
> be
> > > > >>> simpler in
> > > > >>> > > > > such a case to always create a "Large" type array.
> > > > >>> > > > >
> > > > >>> > > > > > [1] I suppose theoretically there might be some
> > > > >>> > > > > > performance
> > > > >>> benefits on
> > > > >>> > > > > > 64-bit architectures to using the native word sizes.
> > > > >>> > > > >
> > > > >>> > > > > Concretely, common 64-bit architectures don't do that, as
> > > > >>> > > > > 32-bit
> > > > >>> is an
> > > > >>> > > > > extremely common integer size even in high-performance
> > code.
> > > > >>> > > > >
> > > > >>> > > > > Regards
> > > > >>> > > > >
> > > > >>> > > > > Antoine.
> > > > >>> > > > >
> > > > >>> > > > >
> > > > >>> > >
> > > > >>>
> > > > >>
> >
>
Re: Timeline for 0.15.0 release
Posted by Neal Richardson <ne...@gmail.com>.
Sounds good to me.
Do we have a release manager yet? Any volunteers?
Neal
On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney <we...@gmail.com> wrote:
> hi all,
>
> It looks like we're drawing close to be able to make the 0.15.0
> release. I would suggest "pencils down" at the end of this week and
> see if a release candidate can be produced next Monday September 23.
> Any thoughts or objections?
>
> Thanks,
> Wes
>
> On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi Eric -- yes, that's correct. I'm planning to amend the Format docs
> > today regarding the EOS issue and also update the C++ library
> >
> > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> > <Er...@microsoft.com> wrote:
> > >
> > > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment
> branch into master before the 0.15 release, correct?
> > >
> > > BTW - I believe the C# alignment changes are ready to be merged into
> the alignment branch - https://github.com/apache/arrow/pull/5280/
> > >
> > > Eric
> > >
> > > -----Original Message-----
> > > From: Micah Kornfield <em...@gmail.com>
> > > Sent: Tuesday, September 10, 2019 10:24 PM
> > > To: Wes McKinney <we...@gmail.com>
> > > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > > Subject: Re: Timeline for 0.15.0 release
> > >
> > > I should have a little more bandwidth to help with some of the
> packaging starting tomorrow and going into the weekend.
> > >
> > > On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > With the state of nightly packaging and integration builds things
> > > > aren't looking too good for being in release readiness by the end of
> > > > this week but maybe I'm wrong. I'm planning to be working to close as
> > > > many issues as I can and also to help with the ongoing alignment
> fixes.
> > > >
> > > > Wes
> > > >
> > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <emkornfield@gmail.com
> >
> > > > wrote:
> > > >
> > > >> Just for reference [1] has a dashboard of the current issues:
> > > >>
> > > >>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > > >> ki.apache.org
> %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com
> %7Ccbead81a42104034
> > > >>
> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > >>
> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > > >> %3D&reserved=0
> > > >>
> > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > >>
> > > >>> hi all,
> > > >>>
> > > >>> It doesn't seem like we're going to be in a position to release at
> > > >>> the beginning of next week. I hope that one more week of work (or
> > > >>> less) will be enough to get us there. Aside from merging the
> > > >>> alignment changes, we need to make sure that our packaging jobs
> > > >>> required for the release candidate are all working.
> > > >>>
> > > >>> If folks could remove issues from the 0.15.0 backlog that they
> don't
> > > >>> think they will finish by end of next week that would help focus
> > > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> > > >>> looking to tackle a few small features related to dictionaries
> while
> > > >>> the release window is still open.
> > > >>>
> > > >>> - Wes
> > > >>>
> > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > hi,
> > > >>> >
> > > >>> > I think we should try to release the week of September 9, so
> > > >>> > development work should be completed by end of next week.
> > > >>> >
> > > >>> > Does that seem reasonable?
> > > >>> >
> > > >>> > I plan to get up a patch for the protocol alignment changes for
> > > >>> > C++ in the next couple of days -- I think that getting the
> > > >>> > alignment work done is the main barrier to releasing.
> > > >>> >
> > > >>> > Thanks
> > > >>> > Wes
> > > >>> >
> > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > > >>> > <ni...@aliyun.com.invalid>
> > > >>> wrote:
> > > >>> > >
> > > >>> > > Hi, Wes, on the java side, I can think of several bugs that
> need
> > > >>> > > to
> > > >>> be fixed or reminded.
> > > >>> > >
> > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
> > > >>> > > even
> > > >>> when empty[1]
> > > >>> > > This one is under review now, however through this PR we find
> > > >>> > > that
> > > >>> there seems a bug in java reading and writing dictionaries in IPC
> > > >>> which is Inconsistent with spec[2] since it assumes all
> dictionaries
> > > >>> are at the start of stream (see details in PR comments, and this
> > > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > > >>> > >
> > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration
> test
> > > >>> JSON files[3]
> > > >>> > > Java side code already checked in, other implementations seems
> not.
> > > >>> > >
> > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by trying
> > > >>> > > to load all records in one contiguous batch, fixed
> > > >>> by providing iterator API for iteratively reading in ARROW-6219[5].
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Ji Liu
> > > >>> > >
> > > >>> > > [1]
> > > >>> > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >>> > >
> 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > > >>> > > ric.Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d736678a45%7
> > > >>> > >
> C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > > >>> > >
> mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > > >>> > > reserved=0 [2]
> > > >>> > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >>> > > 2Farrow.apache.org
> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > > >>> > > ardt%40microsoft.com
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > >>> > >
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > > >>> > >
> a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > > >>> > > d=0 [3]
> > > >>> > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >>> > > 2Fissues.apache.org
> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > > >>> > > 1%7CEric.Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d736678
> > > >>> > >
> a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > > >>> > >
> 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > > >>> > > ;reserved=0 [4]
> > > >>> > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > >>> > > 2Fissues.apache.org
> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d73
> > > >>> > >
> 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > > >>> > >
> 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > > >>> > > &reserved=0]
> > > >>>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > >>> sues.apache.org
> %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > > >>> .Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d736678a45%7C72f988
> > > >>>
> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > > >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> ----------------------------------------------------------------
> > > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> > > >>> > > Subject:Re: Timeline for 0.15.0 release
> > > >>> > >
> > > >>> > > I'm going to work some on organizing the 0.15.0 backlog some
> > > >>> > > this week, if anyone wants to help with grooming (particularly
> > > >>> > > for languages other than C++/Python where I'm focusing) that
> > > >>> > > would be helpful. There have been almost 500 JIRA issues opened
> > > >>> > > since the
> > > >>> > > 0.14.0 release, so we should make sure to check whether there's
> > > >>> > > any regressions or other serious bugs that we should try to fix
> > > >>> > > for 0.15.0.
> > > >>> > >
> > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > > >>> > > <we...@gmail.com>
> > > >>> wrote:
> > > >>> > > >
> > > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > > >>> > > >
> > > >>> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >>> > > > F%2Fissues.apache.org
> %2Fjira%2Fbrowse%2FARROW-6015&data=02
> > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d
> > > >>> > > >
> 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > > >>> > > >
> 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > > >>> > > > WEWg0%3D&reserved=0
> > > >>> > > >
> > > >>> > > > I think the root cause could be the Windows changes in
> > > >>> > > >
> > > >>> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >>> > > >
> F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > > >>> > > > CEric.Erhardt%40microsoft.com
> %7Ccbead81a42104034a4f308d736678a
> > > >>> > > >
> 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > > >>> > > >
> 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > > >>> > > > bs%3D&reserved=0
> > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > > >>> > > >
> > > >>> > > > I would be appreciative if a volunteer would look into what
> > > >>> > > > was
> > > >>> wrong
> > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
> > > >>> > > > wheels will be broken, too
> > > >>> > > >
> > > >>> > > > The bad wheels can be found at
> > > >>> > > >
> > > >>> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > >>> > > >
> F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com
> %7Ccbea
> > > >>> > > >
> d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > > >>> > > >
> 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > > >>> > > >
> > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > > >>> solipsis@pitrou.net> wrote:
> > > >>> > > > >
> > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> > > >>> > > > > <em...@gmail.com> wrote:
> > > >>> > > > > > >
> > > >>> > > > > > > In C++ they are
> > > >>> > > > > > > independent, we could have 32-bit array lengths and
> > > >>> variable-length
> > > >>> > > > > > > types with 64-bit offsets if we wanted (we just
> wouldn't
> > > >>> > > > > > > be
> > > >>> able to
> > > >>> > > > > > > have a List child with more than INT32_MAX elements).
> > > >>> > > > > >
> > > >>> > > > > > I think the point is we could do this in C++ but we
> don't.
> > > >>> I'm not sure we
> > > >>> > > > > > would have introduced the "Large" types if we did.
> > > >>> > > > >
> > > >>> > > > > 64-bit offsets take twice as much space as 32-bit offsets,
> > > >>> > > > > so if
> > > >>> you're
> > > >>> > > > > storing lots of small-ish lists or strings, 32-bit offsets
> > > >>> > > > > are preferrable. So even with 64-bit array lengths from
> the
> > > >>> > > > > start
> > > >>> it would
> > > >>> > > > > still be beneficial to have types with 32-bit offsets.
> > > >>> > > > >
> > > >>> > > > > > Going with the limited address space in Java and calling
> > > >>> > > > > > it a
> > > >>> reference
> > > >>> > > > > > implementation seems suboptimal. If a consumer uses a
> "Large"
> > > >>> type
> > > >>> > > > > > presumably it is because they need the ability to store
> > > >>> > > > > > more
> > > >>> than INT32_MAX
> > > >>> > > > > > child elements in a column, otherwise it is just wasting
> > > >>> > > > > > space
> > > >>> [1].
> > > >>> > > > >
> > > >>> > > > > Probably. Though if the individual elements (lists or
> > > >>> > > > > strings)
> > > >>> are
> > > >>> > > > > large, not much space is wasted in proportion, so it may be
> > > >>> simpler in
> > > >>> > > > > such a case to always create a "Large" type array.
> > > >>> > > > >
> > > >>> > > > > > [1] I suppose theoretically there might be some
> > > >>> > > > > > performance
> > > >>> benefits on
> > > >>> > > > > > 64-bit architectures to using the native word sizes.
> > > >>> > > > >
> > > >>> > > > > Concretely, common 64-bit architectures don't do that, as
> > > >>> > > > > 32-bit
> > > >>> is an
> > > >>> > > > > extremely common integer size even in high-performance
> code.
> > > >>> > > > >
> > > >>> > > > > Regards
> > > >>> > > > >
> > > >>> > > > > Antoine.
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > >
> > > >>>
> > > >>
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
hi all,
It looks like we're drawing close to be able to make the 0.15.0
release. I would suggest "pencils down" at the end of this week and
see if a release candidate can be produced next Monday September 23.
Any thoughts or objections?
Thanks,
Wes
On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney <we...@gmail.com> wrote:
>
> hi Eric -- yes, that's correct. I'm planning to amend the Format docs
> today regarding the EOS issue and also update the C++ library
>
> On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
> <Er...@microsoft.com> wrote:
> >
> > I assume the plan is to merge the ARROW-6313-flatbuffer-alignment branch into master before the 0.15 release, correct?
> >
> > BTW - I believe the C# alignment changes are ready to be merged into the alignment branch - https://github.com/apache/arrow/pull/5280/
> >
> > Eric
> >
> > -----Original Message-----
> > From: Micah Kornfield <em...@gmail.com>
> > Sent: Tuesday, September 10, 2019 10:24 PM
> > To: Wes McKinney <we...@gmail.com>
> > Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> > Subject: Re: Timeline for 0.15.0 release
> >
> > I should have a little more bandwidth to help with some of the packaging starting tomorrow and going into the weekend.
> >
> > On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > With the state of nightly packaging and integration builds things
> > > aren't looking too good for being in release readiness by the end of
> > > this week but maybe I'm wrong. I'm planning to be working to close as
> > > many issues as I can and also to help with the ongoing alignment fixes.
> > >
> > > Wes
> > >
> > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> Just for reference [1] has a dashboard of the current issues:
> > >>
> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> > >> ki.apache.org%2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034
> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> > >> %3D&reserved=0
> > >>
> > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
> > >>
> > >>> hi all,
> > >>>
> > >>> It doesn't seem like we're going to be in a position to release at
> > >>> the beginning of next week. I hope that one more week of work (or
> > >>> less) will be enough to get us there. Aside from merging the
> > >>> alignment changes, we need to make sure that our packaging jobs
> > >>> required for the release candidate are all working.
> > >>>
> > >>> If folks could remove issues from the 0.15.0 backlog that they don't
> > >>> think they will finish by end of next week that would help focus
> > >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> > >>> looking to tackle a few small features related to dictionaries while
> > >>> the release window is still open.
> > >>>
> > >>> - Wes
> > >>>
> > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > hi,
> > >>> >
> > >>> > I think we should try to release the week of September 9, so
> > >>> > development work should be completed by end of next week.
> > >>> >
> > >>> > Does that seem reasonable?
> > >>> >
> > >>> > I plan to get up a patch for the protocol alignment changes for
> > >>> > C++ in the next couple of days -- I think that getting the
> > >>> > alignment work done is the main barrier to releasing.
> > >>> >
> > >>> > Thanks
> > >>> > Wes
> > >>> >
> > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> > >>> > <ni...@aliyun.com.invalid>
> > >>> wrote:
> > >>> > >
> > >>> > > Hi, Wes, on the java side, I can think of several bugs that need
> > >>> > > to
> > >>> be fixed or reminded.
> > >>> > >
> > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
> > >>> > > even
> > >>> when empty[1]
> > >>> > > This one is under review now, however through this PR we find
> > >>> > > that
> > >>> there seems a bug in java reading and writing dictionaries in IPC
> > >>> which is Inconsistent with spec[2] since it assumes all dictionaries
> > >>> are at the start of stream (see details in PR comments, and this
> > >>> fix may not catch up with version 0.15). @Micah Kornfield
> > >>> > >
> > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test
> > >>> JSON files[3]
> > >>> > > Java side code already checked in, other implementations seems not.
> > >>> > >
> > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by trying
> > >>> > > to load all records in one contiguous batch, fixed
> > >>> by providing iterator API for iteratively reading in ARROW-6219[5].
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Ji Liu
> > >>> > >
> > >>> > > [1]
> > >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>> > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> > >>> > > ric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7
> > >>> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> > >>> > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> > >>> > > reserved=0 [2]
> > >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>> > > 2Farrow.apache.org%2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> > >>> > > ardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >>> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> > >>> > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> > >>> > > d=0 [3]
> > >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> > >>> > > 1%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678
> > >>> > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> > >>> > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> > >>> > > ;reserved=0 [4]
> > >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > >>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> > >>> > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d73
> > >>> > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> > >>> > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> > >>> > > &reserved=0]
> > >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > >>> sues.apache.org%2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> > >>> .Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
> > >>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> > >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > ----------------------------------------------------------------
> > >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> > >>> > > Subject:Re: Timeline for 0.15.0 release
> > >>> > >
> > >>> > > I'm going to work some on organizing the 0.15.0 backlog some
> > >>> > > this week, if anyone wants to help with grooming (particularly
> > >>> > > for languages other than C++/Python where I'm focusing) that
> > >>> > > would be helpful. There have been almost 500 JIRA issues opened
> > >>> > > since the
> > >>> > > 0.14.0 release, so we should make sure to check whether there's
> > >>> > > any regressions or other serious bugs that we should try to fix
> > >>> > > for 0.15.0.
> > >>> > >
> > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> > >>> > > <we...@gmail.com>
> > >>> wrote:
> > >>> > > >
> > >>> > > > The Windows wheel issue in 0.14.1 seems to be
> > >>> > > >
> > >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>> > > > F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6015&data=02
> > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d
> > >>> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> > >>> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> > >>> > > > WEWg0%3D&reserved=0
> > >>> > > >
> > >>> > > > I think the root cause could be the Windows changes in
> > >>> > > >
> > >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> > >>> > > > CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a
> > >>> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> > >>> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> > >>> > > > bs%3D&reserved=0
> > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> > >>> > > >
> > >>> > > > I would be appreciative if a volunteer would look into what
> > >>> > > > was
> > >>> wrong
> > >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
> > >>> > > > wheels will be broken, too
> > >>> > > >
> > >>> > > > The bad wheels can be found at
> > >>> > > >
> > >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > >>> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbea
> > >>> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> > >>> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> > >>> > > >
> > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> > >>> solipsis@pitrou.net> wrote:
> > >>> > > > >
> > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> > >>> > > > > <em...@gmail.com> wrote:
> > >>> > > > > > >
> > >>> > > > > > > In C++ they are
> > >>> > > > > > > independent, we could have 32-bit array lengths and
> > >>> variable-length
> > >>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't
> > >>> > > > > > > be
> > >>> able to
> > >>> > > > > > > have a List child with more than INT32_MAX elements).
> > >>> > > > > >
> > >>> > > > > > I think the point is we could do this in C++ but we don't.
> > >>> I'm not sure we
> > >>> > > > > > would have introduced the "Large" types if we did.
> > >>> > > > >
> > >>> > > > > 64-bit offsets take twice as much space as 32-bit offsets,
> > >>> > > > > so if
> > >>> you're
> > >>> > > > > storing lots of small-ish lists or strings, 32-bit offsets
> > >>> > > > > are preferrable. So even with 64-bit array lengths from the
> > >>> > > > > start
> > >>> it would
> > >>> > > > > still be beneficial to have types with 32-bit offsets.
> > >>> > > > >
> > >>> > > > > > Going with the limited address space in Java and calling
> > >>> > > > > > it a
> > >>> reference
> > >>> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
> > >>> type
> > >>> > > > > > presumably it is because they need the ability to store
> > >>> > > > > > more
> > >>> than INT32_MAX
> > >>> > > > > > child elements in a column, otherwise it is just wasting
> > >>> > > > > > space
> > >>> [1].
> > >>> > > > >
> > >>> > > > > Probably. Though if the individual elements (lists or
> > >>> > > > > strings)
> > >>> are
> > >>> > > > > large, not much space is wasted in proportion, so it may be
> > >>> simpler in
> > >>> > > > > such a case to always create a "Large" type array.
> > >>> > > > >
> > >>> > > > > > [1] I suppose theoretically there might be some
> > >>> > > > > > performance
> > >>> benefits on
> > >>> > > > > > 64-bit architectures to using the native word sizes.
> > >>> > > > >
> > >>> > > > > Concretely, common 64-bit architectures don't do that, as
> > >>> > > > > 32-bit
> > >>> is an
> > >>> > > > > extremely common integer size even in high-performance code.
> > >>> > > > >
> > >>> > > > > Regards
> > >>> > > > >
> > >>> > > > > Antoine.
> > >>> > > > >
> > >>> > > > >
> > >>> > >
> > >>>
> > >>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
hi Eric -- yes, that's correct. I'm planning to amend the Format docs
today regarding the EOS issue and also update the C++ library
On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt
<Er...@microsoft.com> wrote:
>
> I assume the plan is to merge the ARROW-6313-flatbuffer-alignment branch into master before the 0.15 release, correct?
>
> BTW - I believe the C# alignment changes are ready to be merged into the alignment branch - https://github.com/apache/arrow/pull/5280/
>
> Eric
>
> -----Original Message-----
> From: Micah Kornfield <em...@gmail.com>
> Sent: Tuesday, September 10, 2019 10:24 PM
> To: Wes McKinney <we...@gmail.com>
> Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
> Subject: Re: Timeline for 0.15.0 release
>
> I should have a little more bandwidth to help with some of the packaging starting tomorrow and going into the weekend.
>
> On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com> wrote:
>
> > Hi folks,
> >
> > With the state of nightly packaging and integration builds things
> > aren't looking too good for being in release readiness by the end of
> > this week but maybe I'm wrong. I'm planning to be working to close as
> > many issues as I can and also to help with the ongoing alignment fixes.
> >
> > Wes
> >
> > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> Just for reference [1] has a dashboard of the current issues:
> >>
> >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
> >> ki.apache.org%2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
> >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034
> >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
> >> %3D&reserved=0
> >>
> >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >>> hi all,
> >>>
> >>> It doesn't seem like we're going to be in a position to release at
> >>> the beginning of next week. I hope that one more week of work (or
> >>> less) will be enough to get us there. Aside from merging the
> >>> alignment changes, we need to make sure that our packaging jobs
> >>> required for the release candidate are all working.
> >>>
> >>> If folks could remove issues from the 0.15.0 backlog that they don't
> >>> think they will finish by end of next week that would help focus
> >>> efforts (there are currently 78 issues in 0.15.0 still). I am
> >>> looking to tackle a few small features related to dictionaries while
> >>> the release window is still open.
> >>>
> >>> - Wes
> >>>
> >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com>
> >>> wrote:
> >>> >
> >>> > hi,
> >>> >
> >>> > I think we should try to release the week of September 9, so
> >>> > development work should be completed by end of next week.
> >>> >
> >>> > Does that seem reasonable?
> >>> >
> >>> > I plan to get up a patch for the protocol alignment changes for
> >>> > C++ in the next couple of days -- I think that getting the
> >>> > alignment work done is the main barrier to releasing.
> >>> >
> >>> > Thanks
> >>> > Wes
> >>> >
> >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
> >>> > <ni...@aliyun.com.invalid>
> >>> wrote:
> >>> > >
> >>> > > Hi, Wes, on the java side, I can think of several bugs that need
> >>> > > to
> >>> be fixed or reminded.
> >>> > >
> >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
> >>> > > even
> >>> when empty[1]
> >>> > > This one is under review now, however through this PR we find
> >>> > > that
> >>> there seems a bug in java reading and writing dictionaries in IPC
> >>> which is Inconsistent with spec[2] since it assumes all dictionaries
> >>> are at the start of stream (see details in PR comments, and this
> >>> fix may not catch up with version 0.15). @Micah Kornfield
> >>> > >
> >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test
> >>> JSON files[3]
> >>> > > Java side code already checked in, other implementations seems not.
> >>> > >
> >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by trying
> >>> > > to load all records in one contiguous batch, fixed
> >>> by providing iterator API for iteratively reading in ARROW-6219[5].
> >>> > >
> >>> > > Thanks,
> >>> > > Ji Liu
> >>> > >
> >>> > > [1]
> >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>> > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
> >>> > > ric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7
> >>> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
> >>> > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
> >>> > > reserved=0 [2]
> >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>> > > 2Farrow.apache.org%2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
> >>> > > ardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
> >>> > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
> >>> > > d=0 [3]
> >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
> >>> > > 1%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678
> >>> > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
> >>> > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
> >>> > > ;reserved=0 [4]
> >>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> >>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
> >>> > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d73
> >>> > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
> >>> > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
> >>> > > &reserved=0]
> >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> >>> sues.apache.org%2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
> >>> .Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
> >>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
> >>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
> >>> > >
> >>> > >
> >>> > >
> >>> > > ----------------------------------------------------------------
> >>> > > -- From:Wes McKinney <we...@gmail.com> Send
> >>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
> >>> > > Subject:Re: Timeline for 0.15.0 release
> >>> > >
> >>> > > I'm going to work some on organizing the 0.15.0 backlog some
> >>> > > this week, if anyone wants to help with grooming (particularly
> >>> > > for languages other than C++/Python where I'm focusing) that
> >>> > > would be helpful. There have been almost 500 JIRA issues opened
> >>> > > since the
> >>> > > 0.14.0 release, so we should make sure to check whether there's
> >>> > > any regressions or other serious bugs that we should try to fix
> >>> > > for 0.15.0.
> >>> > >
> >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
> >>> > > <we...@gmail.com>
> >>> wrote:
> >>> > > >
> >>> > > > The Windows wheel issue in 0.14.1 seems to be
> >>> > > >
> >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>> > > > F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6015&data=02
> >>> > > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d
> >>> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
> >>> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
> >>> > > > WEWg0%3D&reserved=0
> >>> > > >
> >>> > > > I think the root cause could be the Windows changes in
> >>> > > >
> >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
> >>> > > > CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a
> >>> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
> >>> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
> >>> > > > bs%3D&reserved=0
> >>> 223ae744cc2a12c60cecb5db593263a03c13f85a
> >>> > > >
> >>> > > > I would be appreciative if a volunteer would look into what
> >>> > > > was
> >>> wrong
> >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
> >>> > > > wheels will be broken, too
> >>> > > >
> >>> > > > The bad wheels can be found at
> >>> > > >
> >>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> >>> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
> >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbea
> >>> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
> >>> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
> >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
> >>> > > >
> >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
> >>> solipsis@pitrou.net> wrote:
> >>> > > > >
> >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
> >>> > > > > <em...@gmail.com> wrote:
> >>> > > > > > >
> >>> > > > > > > In C++ they are
> >>> > > > > > > independent, we could have 32-bit array lengths and
> >>> variable-length
> >>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't
> >>> > > > > > > be
> >>> able to
> >>> > > > > > > have a List child with more than INT32_MAX elements).
> >>> > > > > >
> >>> > > > > > I think the point is we could do this in C++ but we don't.
> >>> I'm not sure we
> >>> > > > > > would have introduced the "Large" types if we did.
> >>> > > > >
> >>> > > > > 64-bit offsets take twice as much space as 32-bit offsets,
> >>> > > > > so if
> >>> you're
> >>> > > > > storing lots of small-ish lists or strings, 32-bit offsets
> >>> > > > > are preferrable. So even with 64-bit array lengths from the
> >>> > > > > start
> >>> it would
> >>> > > > > still be beneficial to have types with 32-bit offsets.
> >>> > > > >
> >>> > > > > > Going with the limited address space in Java and calling
> >>> > > > > > it a
> >>> reference
> >>> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
> >>> type
> >>> > > > > > presumably it is because they need the ability to store
> >>> > > > > > more
> >>> than INT32_MAX
> >>> > > > > > child elements in a column, otherwise it is just wasting
> >>> > > > > > space
> >>> [1].
> >>> > > > >
> >>> > > > > Probably. Though if the individual elements (lists or
> >>> > > > > strings)
> >>> are
> >>> > > > > large, not much space is wasted in proportion, so it may be
> >>> simpler in
> >>> > > > > such a case to always create a "Large" type array.
> >>> > > > >
> >>> > > > > > [1] I suppose theoretically there might be some
> >>> > > > > > performance
> >>> benefits on
> >>> > > > > > 64-bit architectures to using the native word sizes.
> >>> > > > >
> >>> > > > > Concretely, common 64-bit architectures don't do that, as
> >>> > > > > 32-bit
> >>> is an
> >>> > > > > extremely common integer size even in high-performance code.
> >>> > > > >
> >>> > > > > Regards
> >>> > > > >
> >>> > > > > Antoine.
> >>> > > > >
> >>> > > > >
> >>> > >
> >>>
> >>
RE: Timeline for 0.15.0 release
Posted by Eric Erhardt <Er...@microsoft.com.INVALID>.
I assume the plan is to merge the ARROW-6313-flatbuffer-alignment branch into master before the 0.15 release, correct?
BTW - I believe the C# alignment changes are ready to be merged into the alignment branch - https://github.com/apache/arrow/pull/5280/
Eric
-----Original Message-----
From: Micah Kornfield <em...@gmail.com>
Sent: Tuesday, September 10, 2019 10:24 PM
To: Wes McKinney <we...@gmail.com>
Cc: dev <de...@arrow.apache.org>; niki.lj <ni...@aliyun.com>
Subject: Re: Timeline for 0.15.0 release
I should have a little more bandwidth to help with some of the packaging starting tomorrow and going into the weekend.
On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com> wrote:
> Hi folks,
>
> With the state of nightly packaging and integration builds things
> aren't looking too good for being in release readiness by the end of
> this week but maybe I'm wrong. I'm planning to be working to close as
> many issues as I can and also to help with the ongoing alignment fixes.
>
> Wes
>
> On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Just for reference [1] has a dashboard of the current issues:
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi
>> ki.apache.org%2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea
>> se&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034
>> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4
>> %3D&reserved=0
>>
>> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
>>
>>> hi all,
>>>
>>> It doesn't seem like we're going to be in a position to release at
>>> the beginning of next week. I hope that one more week of work (or
>>> less) will be enough to get us there. Aside from merging the
>>> alignment changes, we need to make sure that our packaging jobs
>>> required for the release candidate are all working.
>>>
>>> If folks could remove issues from the 0.15.0 backlog that they don't
>>> think they will finish by end of next week that would help focus
>>> efforts (there are currently 78 issues in 0.15.0 still). I am
>>> looking to tackle a few small features related to dictionaries while
>>> the release window is still open.
>>>
>>> - Wes
>>>
>>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>> >
>>> > hi,
>>> >
>>> > I think we should try to release the week of September 9, so
>>> > development work should be completed by end of next week.
>>> >
>>> > Does that seem reasonable?
>>> >
>>> > I plan to get up a patch for the protocol alignment changes for
>>> > C++ in the next couple of days -- I think that getting the
>>> > alignment work done is the main barrier to releasing.
>>> >
>>> > Thanks
>>> > Wes
>>> >
>>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu
>>> > <ni...@aliyun.com.invalid>
>>> wrote:
>>> > >
>>> > > Hi, Wes, on the java side, I can think of several bugs that need
>>> > > to
>>> be fixed or reminded.
>>> > >
>>> > > i. ARROW-6040: Dictionary entries are required in IPC streams
>>> > > even
>>> when empty[1]
>>> > > This one is under review now, however through this PR we find
>>> > > that
>>> there seems a bug in java reading and writing dictionaries in IPC
>>> which is Inconsistent with spec[2] since it assumes all dictionaries
>>> are at the start of stream (see details in PR comments, and this
>>> fix may not catch up with version 0.15). @Micah Kornfield
>>> > >
>>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test
>>> JSON files[3]
>>> > > Java side code already checked in, other implementations seems not.
>>> > >
>>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] Caused by trying
>>> > > to load all records in one contiguous batch, fixed
>>> by providing iterator API for iteratively reading in ARROW-6219[5].
>>> > >
>>> > > Thanks,
>>> > > Ji Liu
>>> > >
>>> > > [1]
>>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE
>>> > > ric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7
>>> > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a
>>> > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D&
>>> > > reserved=0 [2]
>>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > 2Farrow.apache.org%2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh
>>> > > ardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
>>> > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat
>>> > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve
>>> > > d=0 [3]
>>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0
>>> > > 1%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678
>>> > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216
>>> > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D&
>>> > > ;reserved=0 [4]
>>> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
>>> > > 2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02
>>> > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d73
>>> > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064
>>> > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D
>>> > > &reserved=0]
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
>>> sues.apache.org%2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric
>>> .Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a45%7C72f988
>>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3
>>> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0
>>> > >
>>> > >
>>> > >
>>> > > ----------------------------------------------------------------
>>> > > -- From:Wes McKinney <we...@gmail.com> Send
>>> > > Time:2019年8月19日(星期一) 23:03 To:dev <de...@arrow.apache.org>
>>> > > Subject:Re: Timeline for 0.15.0 release
>>> > >
>>> > > I'm going to work some on organizing the 0.15.0 backlog some
>>> > > this week, if anyone wants to help with grooming (particularly
>>> > > for languages other than C++/Python where I'm focusing) that
>>> > > would be helpful. There have been almost 500 JIRA issues opened
>>> > > since the
>>> > > 0.14.0 release, so we should make sure to check whether there's
>>> > > any regressions or other serious bugs that we should try to fix
>>> > > for 0.15.0.
>>> > >
>>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney
>>> > > <we...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > The Windows wheel issue in 0.14.1 seems to be
>>> > > >
>>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-6015&data=02
>>> > > > %7C01%7CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d
>>> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376
>>> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u
>>> > > > WEWg0%3D&reserved=0
>>> > > >
>>> > > > I think the root cause could be the Windows changes in
>>> > > >
>>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7
>>> > > > CEric.Erhardt%40microsoft.com%7Ccbead81a42104034a4f308d736678a
>>> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821
>>> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl
>>> > > > bs%3D&reserved=0
>>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>>> > > >
>>> > > > I would be appreciative if a volunteer would look into what
>>> > > > was
>>> wrong
>>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows
>>> > > > wheels will be broken, too
>>> > > >
>>> > > > The bad wheels can be found at
>>> > > >
>>> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>>> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252
>>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt%40microsoft.com%7Ccbea
>>> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4
>>> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ
>>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0
>>> > > >
>>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
>>> solipsis@pitrou.net> wrote:
>>> > > > >
>>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah Kornfield
>>> > > > > <em...@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > > In C++ they are
>>> > > > > > > independent, we could have 32-bit array lengths and
>>> variable-length
>>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't
>>> > > > > > > be
>>> able to
>>> > > > > > > have a List child with more than INT32_MAX elements).
>>> > > > > >
>>> > > > > > I think the point is we could do this in C++ but we don't.
>>> I'm not sure we
>>> > > > > > would have introduced the "Large" types if we did.
>>> > > > >
>>> > > > > 64-bit offsets take twice as much space as 32-bit offsets,
>>> > > > > so if
>>> you're
>>> > > > > storing lots of small-ish lists or strings, 32-bit offsets
>>> > > > > are preferrable. So even with 64-bit array lengths from the
>>> > > > > start
>>> it would
>>> > > > > still be beneficial to have types with 32-bit offsets.
>>> > > > >
>>> > > > > > Going with the limited address space in Java and calling
>>> > > > > > it a
>>> reference
>>> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
>>> type
>>> > > > > > presumably it is because they need the ability to store
>>> > > > > > more
>>> than INT32_MAX
>>> > > > > > child elements in a column, otherwise it is just wasting
>>> > > > > > space
>>> [1].
>>> > > > >
>>> > > > > Probably. Though if the individual elements (lists or
>>> > > > > strings)
>>> are
>>> > > > > large, not much space is wasted in proportion, so it may be
>>> simpler in
>>> > > > > such a case to always create a "Large" type array.
>>> > > > >
>>> > > > > > [1] I suppose theoretically there might be some
>>> > > > > > performance
>>> benefits on
>>> > > > > > 64-bit architectures to using the native word sizes.
>>> > > > >
>>> > > > > Concretely, common 64-bit architectures don't do that, as
>>> > > > > 32-bit
>>> is an
>>> > > > > extremely common integer size even in high-performance code.
>>> > > > >
>>> > > > > Regards
>>> > > > >
>>> > > > > Antoine.
>>> > > > >
>>> > > > >
>>> > >
>>>
>>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
I should have a little more bandwidth to help with some of the packaging
starting tomorrow and going into the weekend.
On Tuesday, September 10, 2019, Wes McKinney <we...@gmail.com> wrote:
> Hi folks,
>
> With the state of nightly packaging and integration builds things aren't
> looking too good for being in release readiness by the end of this week but
> maybe I'm wrong. I'm planning to be working to close as many issues as I
> can and also to help with the ongoing alignment fixes.
>
> Wes
>
> On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Just for reference [1] has a dashboard of the current issues:
>>
>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.15.0+Release
>>
>> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
>>
>>> hi all,
>>>
>>> It doesn't seem like we're going to be in a position to release at the
>>> beginning of next week. I hope that one more week of work (or less)
>>> will be enough to get us there. Aside from merging the alignment
>>> changes, we need to make sure that our packaging jobs required for the
>>> release candidate are all working.
>>>
>>> If folks could remove issues from the 0.15.0 backlog that they don't
>>> think they will finish by end of next week that would help focus
>>> efforts (there are currently 78 issues in 0.15.0 still). I am looking
>>> to tackle a few small features related to dictionaries while the
>>> release window is still open.
>>>
>>> - Wes
>>>
>>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>> >
>>> > hi,
>>> >
>>> > I think we should try to release the week of September 9, so
>>> > development work should be completed by end of next week.
>>> >
>>> > Does that seem reasonable?
>>> >
>>> > I plan to get up a patch for the protocol alignment changes for C++ in
>>> > the next couple of days -- I think that getting the alignment work
>>> > done is the main barrier to releasing.
>>> >
>>> > Thanks
>>> > Wes
>>> >
>>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <ni...@aliyun.com.invalid>
>>> wrote:
>>> > >
>>> > > Hi, Wes, on the java side, I can think of several bugs that need to
>>> be fixed or reminded.
>>> > >
>>> > > i. ARROW-6040: Dictionary entries are required in IPC streams even
>>> when empty[1]
>>> > > This one is under review now, however through this PR we find that
>>> there seems a bug in java reading and writing dictionaries in IPC which is
>>> Inconsistent with spec[2] since it assumes all dictionaries are at the
>>> start of stream (see details in PR comments, and this fix may not catch up
>>> with version 0.15). @Micah Kornfield
>>> > >
>>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test
>>> JSON files[3]
>>> > > Java side code already checked in, other implementations seems not.
>>> > >
>>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
>>> > > Caused by trying to load all records in one contiguous batch, fixed
>>> by providing iterator API for iteratively reading in ARROW-6219[5].
>>> > >
>>> > > Thanks,
>>> > > Ji Liu
>>> > >
>>> > > [1] https://github.com/apache/arrow/pull/4960
>>> > > [2] https://arrow.apache.org/docs/ipc.html
>>> > > [3] https://issues.apache.org/jira/browse/ARROW-1875
>>> > > [4] https://issues.apache.org/jira/browse/ARROW-6202[5]
>>> https://issues.apache.org/jira/browse/ARROW-6219
>>> > >
>>> > >
>>> > >
>>> > > ------------------------------------------------------------------
>>> > > From:Wes McKinney <we...@gmail.com>
>>> > > Send Time:2019年8月19日(星期一) 23:03
>>> > > To:dev <de...@arrow.apache.org>
>>> > > Subject:Re: Timeline for 0.15.0 release
>>> > >
>>> > > I'm going to work some on organizing the 0.15.0 backlog some this
>>> > > week, if anyone wants to help with grooming (particularly for
>>> > > languages other than C++/Python where I'm focusing) that would be
>>> > > helpful. There have been almost 500 JIRA issues opened since the
>>> > > 0.14.0 release, so we should make sure to check whether there's any
>>> > > regressions or other serious bugs that we should try to fix for
>>> > > 0.15.0.
>>> > >
>>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > The Windows wheel issue in 0.14.1 seems to be
>>> > > >
>>> > > > https://issues.apache.org/jira/browse/ARROW-6015
>>> > > >
>>> > > > I think the root cause could be the Windows changes in
>>> > > >
>>> > > > https://github.com/apache/arrow/commit/
>>> 223ae744cc2a12c60cecb5db593263a03c13f85a
>>> > > >
>>> > > > I would be appreciative if a volunteer would look into what was
>>> wrong
>>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
>>> > > > will be broken, too
>>> > > >
>>> > > > The bad wheels can be found at
>>> > > >
>>> > > > https://bintray.com/apache/arrow/python#files/python%2F0.14.1
>>> > > >
>>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <
>>> solipsis@pitrou.net> wrote:
>>> > > > >
>>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
>>> > > > > Micah Kornfield <em...@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > > In C++ they are
>>> > > > > > > independent, we could have 32-bit array lengths and
>>> variable-length
>>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't be
>>> able to
>>> > > > > > > have a List child with more than INT32_MAX elements).
>>> > > > > >
>>> > > > > > I think the point is we could do this in C++ but we don't.
>>> I'm not sure we
>>> > > > > > would have introduced the "Large" types if we did.
>>> > > > >
>>> > > > > 64-bit offsets take twice as much space as 32-bit offsets, so if
>>> you're
>>> > > > > storing lots of small-ish lists or strings, 32-bit offsets are
>>> > > > > preferrable. So even with 64-bit array lengths from the start
>>> it would
>>> > > > > still be beneficial to have types with 32-bit offsets.
>>> > > > >
>>> > > > > > Going with the limited address space in Java and calling it a
>>> reference
>>> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
>>> type
>>> > > > > > presumably it is because they need the ability to store more
>>> than INT32_MAX
>>> > > > > > child elements in a column, otherwise it is just wasting space
>>> [1].
>>> > > > >
>>> > > > > Probably. Though if the individual elements (lists or strings)
>>> are
>>> > > > > large, not much space is wasted in proportion, so it may be
>>> simpler in
>>> > > > > such a case to always create a "Large" type array.
>>> > > > >
>>> > > > > > [1] I suppose theoretically there might be some performance
>>> benefits on
>>> > > > > > 64-bit architectures to using the native word sizes.
>>> > > > >
>>> > > > > Concretely, common 64-bit architectures don't do that, as 32-bit
>>> is an
>>> > > > > extremely common integer size even in high-performance code.
>>> > > > >
>>> > > > > Regards
>>> > > > >
>>> > > > > Antoine.
>>> > > > >
>>> > > > >
>>> > >
>>>
>>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
Hi folks,
With the state of nightly packaging and integration builds things aren't
looking too good for being in release readiness by the end of this week but
maybe I'm wrong. I'm planning to be working to close as many issues as I
can and also to help with the ongoing alignment fixes.
Wes
On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <em...@gmail.com> wrote:
> Just for reference [1] has a dashboard of the current issues:
>
> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.15.0+Release
>
> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
>
>> hi all,
>>
>> It doesn't seem like we're going to be in a position to release at the
>> beginning of next week. I hope that one more week of work (or less)
>> will be enough to get us there. Aside from merging the alignment
>> changes, we need to make sure that our packaging jobs required for the
>> release candidate are all working.
>>
>> If folks could remove issues from the 0.15.0 backlog that they don't
>> think they will finish by end of next week that would help focus
>> efforts (there are currently 78 issues in 0.15.0 still). I am looking
>> to tackle a few small features related to dictionaries while the
>> release window is still open.
>>
>> - Wes
>>
>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com> wrote:
>> >
>> > hi,
>> >
>> > I think we should try to release the week of September 9, so
>> > development work should be completed by end of next week.
>> >
>> > Does that seem reasonable?
>> >
>> > I plan to get up a patch for the protocol alignment changes for C++ in
>> > the next couple of days -- I think that getting the alignment work
>> > done is the main barrier to releasing.
>> >
>> > Thanks
>> > Wes
>> >
>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <ni...@aliyun.com.invalid>
>> wrote:
>> > >
>> > > Hi, Wes, on the java side, I can think of several bugs that need to
>> be fixed or reminded.
>> > >
>> > > i. ARROW-6040: Dictionary entries are required in IPC streams even
>> when empty[1]
>> > > This one is under review now, however through this PR we find that
>> there seems a bug in java reading and writing dictionaries in IPC which is
>> Inconsistent with spec[2] since it assumes all dictionaries are at the
>> start of stream (see details in PR comments, and this fix may not catch up
>> with version 0.15). @Micah Kornfield
>> > >
>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test JSON
>> files[3]
>> > > Java side code already checked in, other implementations seems not.
>> > >
>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
>> > > Caused by trying to load all records in one contiguous batch, fixed
>> by providing iterator API for iteratively reading in ARROW-6219[5].
>> > >
>> > > Thanks,
>> > > Ji Liu
>> > >
>> > > [1] https://github.com/apache/arrow/pull/4960
>> > > [2] https://arrow.apache.org/docs/ipc.html
>> > > [3] https://issues.apache.org/jira/browse/ARROW-1875
>> > > [4] https://issues.apache.org/jira/browse/ARROW-6202[5]
>> https://issues.apache.org/jira/browse/ARROW-6219
>> > >
>> > >
>> > >
>> > > ------------------------------------------------------------------
>> > > From:Wes McKinney <we...@gmail.com>
>> > > Send Time:2019年8月19日(星期一) 23:03
>> > > To:dev <de...@arrow.apache.org>
>> > > Subject:Re: Timeline for 0.15.0 release
>> > >
>> > > I'm going to work some on organizing the 0.15.0 backlog some this
>> > > week, if anyone wants to help with grooming (particularly for
>> > > languages other than C++/Python where I'm focusing) that would be
>> > > helpful. There have been almost 500 JIRA issues opened since the
>> > > 0.14.0 release, so we should make sure to check whether there's any
>> > > regressions or other serious bugs that we should try to fix for
>> > > 0.15.0.
>> > >
>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com>
>> wrote:
>> > > >
>> > > > The Windows wheel issue in 0.14.1 seems to be
>> > > >
>> > > > https://issues.apache.org/jira/browse/ARROW-6015
>> > > >
>> > > > I think the root cause could be the Windows changes in
>> > > >
>> > > >
>> https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
>> > > >
>> > > > I would be appreciative if a volunteer would look into what was
>> wrong
>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
>> > > > will be broken, too
>> > > >
>> > > > The bad wheels can be found at
>> > > >
>> > > > https://bintray.com/apache/arrow/python#files/python%2F0.14.1
>> > > >
>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net>
>> wrote:
>> > > > >
>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
>> > > > > Micah Kornfield <em...@gmail.com> wrote:
>> > > > > > >
>> > > > > > > In C++ they are
>> > > > > > > independent, we could have 32-bit array lengths and
>> variable-length
>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't be
>> able to
>> > > > > > > have a List child with more than INT32_MAX elements).
>> > > > > >
>> > > > > > I think the point is we could do this in C++ but we don't. I'm
>> not sure we
>> > > > > > would have introduced the "Large" types if we did.
>> > > > >
>> > > > > 64-bit offsets take twice as much space as 32-bit offsets, so if
>> you're
>> > > > > storing lots of small-ish lists or strings, 32-bit offsets are
>> > > > > preferrable. So even with 64-bit array lengths from the start it
>> would
>> > > > > still be beneficial to have types with 32-bit offsets.
>> > > > >
>> > > > > > Going with the limited address space in Java and calling it a
>> reference
>> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
>> type
>> > > > > > presumably it is because they need the ability to store more
>> than INT32_MAX
>> > > > > > child elements in a column, otherwise it is just wasting space
>> [1].
>> > > > >
>> > > > > Probably. Though if the individual elements (lists or strings) are
>> > > > > large, not much space is wasted in proportion, so it may be
>> simpler in
>> > > > > such a case to always create a "Large" type array.
>> > > > >
>> > > > > > [1] I suppose theoretically there might be some performance
>> benefits on
>> > > > > > 64-bit architectures to using the native word sizes.
>> > > > >
>> > > > > Concretely, common 64-bit architectures don't do that, as 32-bit
>> is an
>> > > > > extremely common integer size even in high-performance code.
>> > > > >
>> > > > > Regards
>> > > > >
>> > > > > Antoine.
>> > > > >
>> > > > >
>> > >
>>
>
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
Just for reference [1] has a dashboard of the current issues:
https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.15.0+Release
On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <we...@gmail.com> wrote:
> hi all,
>
> It doesn't seem like we're going to be in a position to release at the
> beginning of next week. I hope that one more week of work (or less)
> will be enough to get us there. Aside from merging the alignment
> changes, we need to make sure that our packaging jobs required for the
> release candidate are all working.
>
> If folks could remove issues from the 0.15.0 backlog that they don't
> think they will finish by end of next week that would help focus
> efforts (there are currently 78 issues in 0.15.0 still). I am looking
> to tackle a few small features related to dictionaries while the
> release window is still open.
>
> - Wes
>
> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi,
> >
> > I think we should try to release the week of September 9, so
> > development work should be completed by end of next week.
> >
> > Does that seem reasonable?
> >
> > I plan to get up a patch for the protocol alignment changes for C++ in
> > the next couple of days -- I think that getting the alignment work
> > done is the main barrier to releasing.
> >
> > Thanks
> > Wes
> >
> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <ni...@aliyun.com.invalid>
> wrote:
> > >
> > > Hi, Wes, on the java side, I can think of several bugs that need to be
> fixed or reminded.
> > >
> > > i. ARROW-6040: Dictionary entries are required in IPC streams even
> when empty[1]
> > > This one is under review now, however through this PR we find that
> there seems a bug in java reading and writing dictionaries in IPC which is
> Inconsistent with spec[2] since it assumes all dictionaries are at the
> start of stream (see details in PR comments, and this fix may not catch up
> with version 0.15). @Micah Kornfield
> > >
> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test JSON
> files[3]
> > > Java side code already checked in, other implementations seems not.
> > >
> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > > Caused by trying to load all records in one contiguous batch, fixed by
> providing iterator API for iteratively reading in ARROW-6219[5].
> > >
> > > Thanks,
> > > Ji Liu
> > >
> > > [1] https://github.com/apache/arrow/pull/4960
> > > [2] https://arrow.apache.org/docs/ipc.html
> > > [3] https://issues.apache.org/jira/browse/ARROW-1875
> > > [4] https://issues.apache.org/jira/browse/ARROW-6202[5]
> https://issues.apache.org/jira/browse/ARROW-6219
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Wes McKinney <we...@gmail.com>
> > > Send Time:2019年8月19日(星期一) 23:03
> > > To:dev <de...@arrow.apache.org>
> > > Subject:Re: Timeline for 0.15.0 release
> > >
> > > I'm going to work some on organizing the 0.15.0 backlog some this
> > > week, if anyone wants to help with grooming (particularly for
> > > languages other than C++/Python where I'm focusing) that would be
> > > helpful. There have been almost 500 JIRA issues opened since the
> > > 0.14.0 release, so we should make sure to check whether there's any
> > > regressions or other serious bugs that we should try to fix for
> > > 0.15.0.
> > >
> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > >
> > > > The Windows wheel issue in 0.14.1 seems to be
> > > >
> > > > https://issues.apache.org/jira/browse/ARROW-6015
> > > >
> > > > I think the root cause could be the Windows changes in
> > > >
> > > >
> https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
> > > >
> > > > I would be appreciative if a volunteer would look into what was wrong
> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
> > > > will be broken, too
> > > >
> > > > The bad wheels can be found at
> > > >
> > > > https://bintray.com/apache/arrow/python#files/python%2F0.14.1
> > > >
> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net>
> wrote:
> > > > >
> > > > > On Thu, 15 Aug 2019 11:17:07 -0700
> > > > > Micah Kornfield <em...@gmail.com> wrote:
> > > > > > >
> > > > > > > In C++ they are
> > > > > > > independent, we could have 32-bit array lengths and
> variable-length
> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't be
> able to
> > > > > > > have a List child with more than INT32_MAX elements).
> > > > > >
> > > > > > I think the point is we could do this in C++ but we don't. I'm
> not sure we
> > > > > > would have introduced the "Large" types if we did.
> > > > >
> > > > > 64-bit offsets take twice as much space as 32-bit offsets, so if
> you're
> > > > > storing lots of small-ish lists or strings, 32-bit offsets are
> > > > > preferrable. So even with 64-bit array lengths from the start it
> would
> > > > > still be beneficial to have types with 32-bit offsets.
> > > > >
> > > > > > Going with the limited address space in Java and calling it a
> reference
> > > > > > implementation seems suboptimal. If a consumer uses a "Large"
> type
> > > > > > presumably it is because they need the ability to store more
> than INT32_MAX
> > > > > > child elements in a column, otherwise it is just wasting space
> [1].
> > > > >
> > > > > Probably. Though if the individual elements (lists or strings) are
> > > > > large, not much space is wasted in proportion, so it may be
> simpler in
> > > > > such a case to always create a "Large" type array.
> > > > >
> > > > > > [1] I suppose theoretically there might be some performance
> benefits on
> > > > > > 64-bit architectures to using the native word sizes.
> > > > >
> > > > > Concretely, common 64-bit architectures don't do that, as 32-bit
> is an
> > > > > extremely common integer size even in high-performance code.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > > > >
> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
hi all,
It doesn't seem like we're going to be in a position to release at the
beginning of next week. I hope that one more week of work (or less)
will be enough to get us there. Aside from merging the alignment
changes, we need to make sure that our packaging jobs required for the
release candidate are all working.
If folks could remove issues from the 0.15.0 backlog that they don't
think they will finish by end of next week that would help focus
efforts (there are currently 78 issues in 0.15.0 still). I am looking
to tackle a few small features related to dictionaries while the
release window is still open.
- Wes
On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi,
>
> I think we should try to release the week of September 9, so
> development work should be completed by end of next week.
>
> Does that seem reasonable?
>
> I plan to get up a patch for the protocol alignment changes for C++ in
> the next couple of days -- I think that getting the alignment work
> done is the main barrier to releasing.
>
> Thanks
> Wes
>
> On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <ni...@aliyun.com.invalid> wrote:
> >
> > Hi, Wes, on the java side, I can think of several bugs that need to be fixed or reminded.
> >
> > i. ARROW-6040: Dictionary entries are required in IPC streams even when empty[1]
> > This one is under review now, however through this PR we find that there seems a bug in java reading and writing dictionaries in IPC which is Inconsistent with spec[2] since it assumes all dictionaries are at the start of stream (see details in PR comments, and this fix may not catch up with version 0.15). @Micah Kornfield
> >
> > ii. ARROW-1875: Write 64-bit ints as strings in integration test JSON files[3]
> > Java side code already checked in, other implementations seems not.
> >
> > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> > Caused by trying to load all records in one contiguous batch, fixed by providing iterator API for iteratively reading in ARROW-6219[5].
> >
> > Thanks,
> > Ji Liu
> >
> > [1] https://github.com/apache/arrow/pull/4960
> > [2] https://arrow.apache.org/docs/ipc.html
> > [3] https://issues.apache.org/jira/browse/ARROW-1875
> > [4] https://issues.apache.org/jira/browse/ARROW-6202[5] https://issues.apache.org/jira/browse/ARROW-6219
> >
> >
> >
> > ------------------------------------------------------------------
> > From:Wes McKinney <we...@gmail.com>
> > Send Time:2019年8月19日(星期一) 23:03
> > To:dev <de...@arrow.apache.org>
> > Subject:Re: Timeline for 0.15.0 release
> >
> > I'm going to work some on organizing the 0.15.0 backlog some this
> > week, if anyone wants to help with grooming (particularly for
> > languages other than C++/Python where I'm focusing) that would be
> > helpful. There have been almost 500 JIRA issues opened since the
> > 0.14.0 release, so we should make sure to check whether there's any
> > regressions or other serious bugs that we should try to fix for
> > 0.15.0.
> >
> > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > The Windows wheel issue in 0.14.1 seems to be
> > >
> > > https://issues.apache.org/jira/browse/ARROW-6015
> > >
> > > I think the root cause could be the Windows changes in
> > >
> > > https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
> > >
> > > I would be appreciative if a volunteer would look into what was wrong
> > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
> > > will be broken, too
> > >
> > > The bad wheels can be found at
> > >
> > > https://bintray.com/apache/arrow/python#files/python%2F0.14.1
> > >
> > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net> wrote:
> > > >
> > > > On Thu, 15 Aug 2019 11:17:07 -0700
> > > > Micah Kornfield <em...@gmail.com> wrote:
> > > > > >
> > > > > > In C++ they are
> > > > > > independent, we could have 32-bit array lengths and variable-length
> > > > > > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > > > > > have a List child with more than INT32_MAX elements).
> > > > >
> > > > > I think the point is we could do this in C++ but we don't. I'm not sure we
> > > > > would have introduced the "Large" types if we did.
> > > >
> > > > 64-bit offsets take twice as much space as 32-bit offsets, so if you're
> > > > storing lots of small-ish lists or strings, 32-bit offsets are
> > > > preferrable. So even with 64-bit array lengths from the start it would
> > > > still be beneficial to have types with 32-bit offsets.
> > > >
> > > > > Going with the limited address space in Java and calling it a reference
> > > > > implementation seems suboptimal. If a consumer uses a "Large" type
> > > > > presumably it is because they need the ability to store more than INT32_MAX
> > > > > child elements in a column, otherwise it is just wasting space [1].
> > > >
> > > > Probably. Though if the individual elements (lists or strings) are
> > > > large, not much space is wasted in proportion, so it may be simpler in
> > > > such a case to always create a "Large" type array.
> > > >
> > > > > [1] I suppose theoretically there might be some performance benefits on
> > > > > 64-bit architectures to using the native word sizes.
> > > >
> > > > Concretely, common 64-bit architectures don't do that, as 32-bit is an
> > > > extremely common integer size even in high-performance code.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
hi,
I think we should try to release the week of September 9, so
development work should be completed by end of next week.
Does that seem reasonable?
I plan to get up a patch for the protocol alignment changes for C++ in
the next couple of days -- I think that getting the alignment work
done is the main barrier to releasing.
Thanks
Wes
On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <ni...@aliyun.com.invalid> wrote:
>
> Hi, Wes, on the java side, I can think of several bugs that need to be fixed or reminded.
>
> i. ARROW-6040: Dictionary entries are required in IPC streams even when empty[1]
> This one is under review now, however through this PR we find that there seems a bug in java reading and writing dictionaries in IPC which is Inconsistent with spec[2] since it assumes all dictionaries are at the start of stream (see details in PR comments, and this fix may not catch up with version 0.15). @Micah Kornfield
>
> ii. ARROW-1875: Write 64-bit ints as strings in integration test JSON files[3]
> Java side code already checked in, other implementations seems not.
>
> iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
> Caused by trying to load all records in one contiguous batch, fixed by providing iterator API for iteratively reading in ARROW-6219[5].
>
> Thanks,
> Ji Liu
>
> [1] https://github.com/apache/arrow/pull/4960
> [2] https://arrow.apache.org/docs/ipc.html
> [3] https://issues.apache.org/jira/browse/ARROW-1875
> [4] https://issues.apache.org/jira/browse/ARROW-6202[5] https://issues.apache.org/jira/browse/ARROW-6219
>
>
>
> ------------------------------------------------------------------
> From:Wes McKinney <we...@gmail.com>
> Send Time:2019年8月19日(星期一) 23:03
> To:dev <de...@arrow.apache.org>
> Subject:Re: Timeline for 0.15.0 release
>
> I'm going to work some on organizing the 0.15.0 backlog some this
> week, if anyone wants to help with grooming (particularly for
> languages other than C++/Python where I'm focusing) that would be
> helpful. There have been almost 500 JIRA issues opened since the
> 0.14.0 release, so we should make sure to check whether there's any
> regressions or other serious bugs that we should try to fix for
> 0.15.0.
>
> On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > The Windows wheel issue in 0.14.1 seems to be
> >
> > https://issues.apache.org/jira/browse/ARROW-6015
> >
> > I think the root cause could be the Windows changes in
> >
> > https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
> >
> > I would be appreciative if a volunteer would look into what was wrong
> > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
> > will be broken, too
> >
> > The bad wheels can be found at
> >
> > https://bintray.com/apache/arrow/python#files/python%2F0.14.1
> >
> > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net> wrote:
> > >
> > > On Thu, 15 Aug 2019 11:17:07 -0700
> > > Micah Kornfield <em...@gmail.com> wrote:
> > > > >
> > > > > In C++ they are
> > > > > independent, we could have 32-bit array lengths and variable-length
> > > > > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > > > > have a List child with more than INT32_MAX elements).
> > > >
> > > > I think the point is we could do this in C++ but we don't. I'm not sure we
> > > > would have introduced the "Large" types if we did.
> > >
> > > 64-bit offsets take twice as much space as 32-bit offsets, so if you're
> > > storing lots of small-ish lists or strings, 32-bit offsets are
> > > preferrable. So even with 64-bit array lengths from the start it would
> > > still be beneficial to have types with 32-bit offsets.
> > >
> > > > Going with the limited address space in Java and calling it a reference
> > > > implementation seems suboptimal. If a consumer uses a "Large" type
> > > > presumably it is because they need the ability to store more than INT32_MAX
> > > > child elements in a column, otherwise it is just wasting space [1].
> > >
> > > Probably. Though if the individual elements (lists or strings) are
> > > large, not much space is wasted in proportion, so it may be simpler in
> > > such a case to always create a "Large" type array.
> > >
> > > > [1] I suppose theoretically there might be some performance benefits on
> > > > 64-bit architectures to using the native word sizes.
> > >
> > > Concretely, common 64-bit architectures don't do that, as 32-bit is an
> > > extremely common integer size even in high-performance code.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
>
Re: Timeline for 0.15.0 release
Posted by Ji Liu <ni...@aliyun.com.INVALID>.
Hi, Wes, on the java side, I can think of several bugs that need to be fixed or reminded.
i. ARROW-6040: Dictionary entries are required in IPC streams even when empty[1]
This one is under review now, however through this PR we find that there seems a bug in java reading and writing dictionaries in IPC which is Inconsistent with spec[2] since it assumes all dictionaries are at the start of stream (see details in PR comments, and this fix may not catch up with version 0.15). @Micah Kornfield
ii. ARROW-1875: Write 64-bit ints as strings in integration test JSON files[3]
Java side code already checked in, other implementations seems not.
iii. ARROW-6202: OutOfMemory in JdbcAdapter[4]
Caused by trying to load all records in one contiguous batch, fixed by providing iterator API for iteratively reading in ARROW-6219[5].
Thanks,
Ji Liu
[1] https://github.com/apache/arrow/pull/4960
[2] https://arrow.apache.org/docs/ipc.html
[3] https://issues.apache.org/jira/browse/ARROW-1875
[4] https://issues.apache.org/jira/browse/ARROW-6202[5] https://issues.apache.org/jira/browse/ARROW-6219
------------------------------------------------------------------
From:Wes McKinney <we...@gmail.com>
Send Time:2019年8月19日(星期一) 23:03
To:dev <de...@arrow.apache.org>
Subject:Re: Timeline for 0.15.0 release
I'm going to work some on organizing the 0.15.0 backlog some this
week, if anyone wants to help with grooming (particularly for
languages other than C++/Python where I'm focusing) that would be
helpful. There have been almost 500 JIRA issues opened since the
0.14.0 release, so we should make sure to check whether there's any
regressions or other serious bugs that we should try to fix for
0.15.0.
On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com> wrote:
>
> The Windows wheel issue in 0.14.1 seems to be
>
> https://issues.apache.org/jira/browse/ARROW-6015
>
> I think the root cause could be the Windows changes in
>
> https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
>
> I would be appreciative if a volunteer would look into what was wrong
> with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
> will be broken, too
>
> The bad wheels can be found at
>
> https://bintray.com/apache/arrow/python#files/python%2F0.14.1
>
> On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net> wrote:
> >
> > On Thu, 15 Aug 2019 11:17:07 -0700
> > Micah Kornfield <em...@gmail.com> wrote:
> > > >
> > > > In C++ they are
> > > > independent, we could have 32-bit array lengths and variable-length
> > > > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > > > have a List child with more than INT32_MAX elements).
> > >
> > > I think the point is we could do this in C++ but we don't. I'm not sure we
> > > would have introduced the "Large" types if we did.
> >
> > 64-bit offsets take twice as much space as 32-bit offsets, so if you're
> > storing lots of small-ish lists or strings, 32-bit offsets are
> > preferrable. So even with 64-bit array lengths from the start it would
> > still be beneficial to have types with 32-bit offsets.
> >
> > > Going with the limited address space in Java and calling it a reference
> > > implementation seems suboptimal. If a consumer uses a "Large" type
> > > presumably it is because they need the ability to store more than INT32_MAX
> > > child elements in a column, otherwise it is just wasting space [1].
> >
> > Probably. Though if the individual elements (lists or strings) are
> > large, not much space is wasted in proportion, so it may be simpler in
> > such a case to always create a "Large" type array.
> >
> > > [1] I suppose theoretically there might be some performance benefits on
> > > 64-bit architectures to using the native word sizes.
> >
> > Concretely, common 64-bit architectures don't do that, as 32-bit is an
> > extremely common integer size even in high-performance code.
> >
> > Regards
> >
> > Antoine.
> >
> >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
I'm going to work some on organizing the 0.15.0 backlog some this
week, if anyone wants to help with grooming (particularly for
languages other than C++/Python where I'm focusing) that would be
helpful. There have been almost 500 JIRA issues opened since the
0.14.0 release, so we should make sure to check whether there's any
regressions or other serious bugs that we should try to fix for
0.15.0.
On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <we...@gmail.com> wrote:
>
> The Windows wheel issue in 0.14.1 seems to be
>
> https://issues.apache.org/jira/browse/ARROW-6015
>
> I think the root cause could be the Windows changes in
>
> https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
>
> I would be appreciative if a volunteer would look into what was wrong
> with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
> will be broken, too
>
> The bad wheels can be found at
>
> https://bintray.com/apache/arrow/python#files/python%2F0.14.1
>
> On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net> wrote:
> >
> > On Thu, 15 Aug 2019 11:17:07 -0700
> > Micah Kornfield <em...@gmail.com> wrote:
> > > >
> > > > In C++ they are
> > > > independent, we could have 32-bit array lengths and variable-length
> > > > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > > > have a List child with more than INT32_MAX elements).
> > >
> > > I think the point is we could do this in C++ but we don't. I'm not sure we
> > > would have introduced the "Large" types if we did.
> >
> > 64-bit offsets take twice as much space as 32-bit offsets, so if you're
> > storing lots of small-ish lists or strings, 32-bit offsets are
> > preferrable. So even with 64-bit array lengths from the start it would
> > still be beneficial to have types with 32-bit offsets.
> >
> > > Going with the limited address space in Java and calling it a reference
> > > implementation seems suboptimal. If a consumer uses a "Large" type
> > > presumably it is because they need the ability to store more than INT32_MAX
> > > child elements in a column, otherwise it is just wasting space [1].
> >
> > Probably. Though if the individual elements (lists or strings) are
> > large, not much space is wasted in proportion, so it may be simpler in
> > such a case to always create a "Large" type array.
> >
> > > [1] I suppose theoretically there might be some performance benefits on
> > > 64-bit architectures to using the native word sizes.
> >
> > Concretely, common 64-bit architectures don't do that, as 32-bit is an
> > extremely common integer size even in high-performance code.
> >
> > Regards
> >
> > Antoine.
> >
> >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
The Windows wheel issue in 0.14.1 seems to be
https://issues.apache.org/jira/browse/ARROW-6015
I think the root cause could be the Windows changes in
https://github.com/apache/arrow/commit/223ae744cc2a12c60cecb5db593263a03c13f85a
I would be appreciative if a volunteer would look into what was wrong
with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels
will be broken, too
The bad wheels can be found at
https://bintray.com/apache/arrow/python#files/python%2F0.14.1
On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou <so...@pitrou.net> wrote:
>
> On Thu, 15 Aug 2019 11:17:07 -0700
> Micah Kornfield <em...@gmail.com> wrote:
> > >
> > > In C++ they are
> > > independent, we could have 32-bit array lengths and variable-length
> > > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > > have a List child with more than INT32_MAX elements).
> >
> > I think the point is we could do this in C++ but we don't. I'm not sure we
> > would have introduced the "Large" types if we did.
>
> 64-bit offsets take twice as much space as 32-bit offsets, so if you're
> storing lots of small-ish lists or strings, 32-bit offsets are
> preferrable. So even with 64-bit array lengths from the start it would
> still be beneficial to have types with 32-bit offsets.
>
> > Going with the limited address space in Java and calling it a reference
> > implementation seems suboptimal. If a consumer uses a "Large" type
> > presumably it is because they need the ability to store more than INT32_MAX
> > child elements in a column, otherwise it is just wasting space [1].
>
> Probably. Though if the individual elements (lists or strings) are
> large, not much space is wasted in proportion, so it may be simpler in
> such a case to always create a "Large" type array.
>
> > [1] I suppose theoretically there might be some performance benefits on
> > 64-bit architectures to using the native word sizes.
>
> Concretely, common 64-bit architectures don't do that, as 32-bit is an
> extremely common integer size even in high-performance code.
>
> Regards
>
> Antoine.
>
>
Re: Timeline for 0.15.0 release
Posted by Antoine Pitrou <so...@pitrou.net>.
On Thu, 15 Aug 2019 11:17:07 -0700
Micah Kornfield <em...@gmail.com> wrote:
> >
> > In C++ they are
> > independent, we could have 32-bit array lengths and variable-length
> > types with 64-bit offsets if we wanted (we just wouldn't be able to
> > have a List child with more than INT32_MAX elements).
>
> I think the point is we could do this in C++ but we don't. I'm not sure we
> would have introduced the "Large" types if we did.
64-bit offsets take twice as much space as 32-bit offsets, so if you're
storing lots of small-ish lists or strings, 32-bit offsets are
preferrable. So even with 64-bit array lengths from the start it would
still be beneficial to have types with 32-bit offsets.
> Going with the limited address space in Java and calling it a reference
> implementation seems suboptimal. If a consumer uses a "Large" type
> presumably it is because they need the ability to store more than INT32_MAX
> child elements in a column, otherwise it is just wasting space [1].
Probably. Though if the individual elements (lists or strings) are
large, not much space is wasted in proportion, so it may be simpler in
such a case to always create a "Large" type array.
> [1] I suppose theoretically there might be some performance benefits on
> 64-bit architectures to using the native word sizes.
Concretely, common 64-bit architectures don't do that, as 32-bit is an
extremely common integer size even in high-performance code.
Regards
Antoine.
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
>
> In C++ they are
> independent, we could have 32-bit array lengths and variable-length
> types with 64-bit offsets if we wanted (we just wouldn't be able to
> have a List child with more than INT32_MAX elements).
I think the point is we could do this in C++ but we don't. I'm not sure we
would have introduced the "Large" types if we did.
We will have to do this Java, it we don't want to convert to 64-bit
addressing.
Going with the limited address space in Java and calling it a reference
implementation seems suboptimal. If a consumer uses a "Large" type
presumably it is because they need the ability to store more than INT32_MAX
child elements in a column, otherwise it is just wasting space [1].
Let's pause until next week when Jacques is back online (and continue on
the other thread). Like I said I think there is enough time either way to
get something in along the timeline we expect for the next release.
[1] I suppose theoretically there might be some performance benefits on
64-bit architectures to using the native word sizes.
On Thu, Aug 15, 2019 at 10:59 AM Wes McKinney <we...@gmail.com> wrote:
> On Thu, Aug 15, 2019 at 12:00 AM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > Hi Wes,
> > >
> > > Do these need to be dependent on the 64-bit array length discussion?
> >
> > We could hack something that can read the lower 32-bit range, so I guess
> > not, but this leaves a bad taste in my mouth. I think there is likely
> > still enough time to have the discussion and get these implemented, one
> way
> > or another.
> >
>
> I guess I still don't understand how the array lengths and the
> List/Varchar offsets are related to each other. I probably just
> haven't looked at the Java library enough. In C++ they are
> independent, we could have 32-bit array lengths and variable-length
> types with 64-bit offsets if we wanted (we just wouldn't be able to
> have a List child with more than INT32_MAX elements). We would have to
> do a limited amount of boundschecking at IPC boundary points (like
> Java is checking presumably now for vectors exceeding INT32_MAX).
>
> > For the record, I don't think we should hold a major release hostage
> > > if we aren't able to complete various feature milestones in time.
> > > Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> > > desired 8-10 week timeline for major releases, so if we need to have
> > > 0.16.0 prior to 1.0.0, I think that is OK also.
> >
> > I agree with the time based milestones in practice, but we are
> backpedaling
> > on the intent to keep type parity between the two reference
> > implementations. At least the way I read the previous threads on the
> > topic, I thought there was lazy consensus that in lieu of requiring
> working
> > implementations in Java and C++ be checked in at the same time, we would
> > rely on the release as a mechanism to forcing function for parity.
> >
>
> I agree with the intent and spirit of the idea, but it seems we have a
> can of worms on our hands now and so I don't think we should keep from
> releasing the work that has been completed if consensus about Java
> changes is not reached in time.
>
> > Thanks,
> > Micah
> >
> > On Wed, Aug 14, 2019 at 11:32 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Agreed with Wes.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 14/08/2019 à 20:30, Wes McKinney a écrit :
> > > > For the record, I don't think we should hold a major release hostage
> > > > if we aren't able to complete various feature milestones in time.
> > > > Since it's been about 5-6 weeks since 0.14.0 we're coming close to
> the
> > > > desired 8-10 week timeline for major releases, so if we need to have
> > > > 0.16.0 prior to 1.0.0, I think that is OK also.
> > > >
> > > > On Wed, Aug 14, 2019 at 11:45 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >>
> > > >> On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <
> emkornfield@gmail.com>
> > > wrote:
> > > >>>
> > > >>>>
> > > >>>> is there anything else that has come up that
> > > >>>> definitely needs to happen before we can release again?
> > > >>>
> > > >>> We need to decide on a way forward for LargeList, LargeBinary, etc,
> > > types...
> > > >>>
> > > >>
> > > >> Do these need to be dependent on the 64-bit array length discussion?
> > > >> They seem somewhat orthogonal to me. If we have to release 0.15.0
> > > >> without the Java side of these, that's OK with me, since reaching
> > > >> format implementation completeness is more of a 1.0.0 concern
> > > >>
> > > >>> On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >>>
> > > >>>> hi folks,
> > > >>>>
> > > >>>> Since there have been a number of fairly serious issues (e.g.
> > > >>>> ARROW-6060) since 0.14.1 that have been fixed I think we should
> start
> > > >>>> planning of the next major release. Note that we still have some
> > > >>>> format-related work (the Flatbuffers alignment issue) that ought
> to be
> > > >>>> resolved (not a small task since it affects 4 or 5
> implementations),
> > > >>>> but aside from that, is there anything else that has come up that
> > > >>>> definitely needs to happen before we can release again?
> > > >>>>
> > > >>>> I would say cutting a release somewhere around the US Labor Day
> > > >>>> holiday (~the week after or so) would be called for.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Wes
> > > >>>>
> > >
>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
On Thu, Aug 15, 2019 at 12:00 AM Micah Kornfield <em...@gmail.com> wrote:
>
> Hi Wes,
> >
> > Do these need to be dependent on the 64-bit array length discussion?
>
> We could hack something that can read the lower 32-bit range, so I guess
> not, but this leaves a bad taste in my mouth. I think there is likely
> still enough time to have the discussion and get these implemented, one way
> or another.
>
I guess I still don't understand how the array lengths and the
List/Varchar offsets are related to each other. I probably just
haven't looked at the Java library enough. In C++ they are
independent, we could have 32-bit array lengths and variable-length
types with 64-bit offsets if we wanted (we just wouldn't be able to
have a List child with more than INT32_MAX elements). We would have to
do a limited amount of boundschecking at IPC boundary points (like
Java is checking presumably now for vectors exceeding INT32_MAX).
> For the record, I don't think we should hold a major release hostage
> > if we aren't able to complete various feature milestones in time.
> > Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> > desired 8-10 week timeline for major releases, so if we need to have
> > 0.16.0 prior to 1.0.0, I think that is OK also.
>
> I agree with the time based milestones in practice, but we are backpedaling
> on the intent to keep type parity between the two reference
> implementations. At least the way I read the previous threads on the
> topic, I thought there was lazy consensus that in lieu of requiring working
> implementations in Java and C++ be checked in at the same time, we would
> rely on the release as a mechanism to forcing function for parity.
>
I agree with the intent and spirit of the idea, but it seems we have a
can of worms on our hands now and so I don't think we should keep from
releasing the work that has been completed if consensus about Java
changes is not reached in time.
> Thanks,
> Micah
>
> On Wed, Aug 14, 2019 at 11:32 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Agreed with Wes.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 14/08/2019 à 20:30, Wes McKinney a écrit :
> > > For the record, I don't think we should hold a major release hostage
> > > if we aren't able to complete various feature milestones in time.
> > > Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> > > desired 8-10 week timeline for major releases, so if we need to have
> > > 0.16.0 prior to 1.0.0, I think that is OK also.
> > >
> > > On Wed, Aug 14, 2019 at 11:45 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>
> > >> On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> > >>>
> > >>>>
> > >>>> is there anything else that has come up that
> > >>>> definitely needs to happen before we can release again?
> > >>>
> > >>> We need to decide on a way forward for LargeList, LargeBinary, etc,
> > types...
> > >>>
> > >>
> > >> Do these need to be dependent on the 64-bit array length discussion?
> > >> They seem somewhat orthogonal to me. If we have to release 0.15.0
> > >> without the Java side of these, that's OK with me, since reaching
> > >> format implementation completeness is more of a 1.0.0 concern
> > >>
> > >>> On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>>
> > >>>> hi folks,
> > >>>>
> > >>>> Since there have been a number of fairly serious issues (e.g.
> > >>>> ARROW-6060) since 0.14.1 that have been fixed I think we should start
> > >>>> planning of the next major release. Note that we still have some
> > >>>> format-related work (the Flatbuffers alignment issue) that ought to be
> > >>>> resolved (not a small task since it affects 4 or 5 implementations),
> > >>>> but aside from that, is there anything else that has come up that
> > >>>> definitely needs to happen before we can release again?
> > >>>>
> > >>>> I would say cutting a release somewhere around the US Labor Day
> > >>>> holiday (~the week after or so) would be called for.
> > >>>>
> > >>>> Thanks,
> > >>>> Wes
> > >>>>
> >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
Hi Wes,
>
> Do these need to be dependent on the 64-bit array length discussion?
We could hack something that can read the lower 32-bit range, so I guess
not, but this leaves a bad taste in my mouth. I think there is likely
still enough time to have the discussion and get these implemented, one way
or another.
For the record, I don't think we should hold a major release hostage
> if we aren't able to complete various feature milestones in time.
> Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> desired 8-10 week timeline for major releases, so if we need to have
> 0.16.0 prior to 1.0.0, I think that is OK also.
I agree with the time based milestones in practice, but we are backpedaling
on the intent to keep type parity between the two reference
implementations. At least the way I read the previous threads on the
topic, I thought there was lazy consensus that in lieu of requiring working
implementations in Java and C++ be checked in at the same time, we would
rely on the release as a mechanism to forcing function for parity.
Thanks,
Micah
On Wed, Aug 14, 2019 at 11:32 AM Antoine Pitrou <an...@python.org> wrote:
>
> Agreed with Wes.
>
> Regards
>
> Antoine.
>
>
> Le 14/08/2019 à 20:30, Wes McKinney a écrit :
> > For the record, I don't think we should hold a major release hostage
> > if we aren't able to complete various feature milestones in time.
> > Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> > desired 8-10 week timeline for major releases, so if we need to have
> > 0.16.0 prior to 1.0.0, I think that is OK also.
> >
> > On Wed, Aug 14, 2019 at 11:45 AM Wes McKinney <we...@gmail.com>
> wrote:
> >>
> >> On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <em...@gmail.com>
> wrote:
> >>>
> >>>>
> >>>> is there anything else that has come up that
> >>>> definitely needs to happen before we can release again?
> >>>
> >>> We need to decide on a way forward for LargeList, LargeBinary, etc,
> types...
> >>>
> >>
> >> Do these need to be dependent on the 64-bit array length discussion?
> >> They seem somewhat orthogonal to me. If we have to release 0.15.0
> >> without the Java side of these, that's OK with me, since reaching
> >> format implementation completeness is more of a 1.0.0 concern
> >>
> >>> On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com>
> wrote:
> >>>
> >>>> hi folks,
> >>>>
> >>>> Since there have been a number of fairly serious issues (e.g.
> >>>> ARROW-6060) since 0.14.1 that have been fixed I think we should start
> >>>> planning of the next major release. Note that we still have some
> >>>> format-related work (the Flatbuffers alignment issue) that ought to be
> >>>> resolved (not a small task since it affects 4 or 5 implementations),
> >>>> but aside from that, is there anything else that has come up that
> >>>> definitely needs to happen before we can release again?
> >>>>
> >>>> I would say cutting a release somewhere around the US Labor Day
> >>>> holiday (~the week after or so) would be called for.
> >>>>
> >>>> Thanks,
> >>>> Wes
> >>>>
>
Re: Timeline for 0.15.0 release
Posted by Antoine Pitrou <an...@python.org>.
Agreed with Wes.
Regards
Antoine.
Le 14/08/2019 à 20:30, Wes McKinney a écrit :
> For the record, I don't think we should hold a major release hostage
> if we aren't able to complete various feature milestones in time.
> Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
> desired 8-10 week timeline for major releases, so if we need to have
> 0.16.0 prior to 1.0.0, I think that is OK also.
>
> On Wed, Aug 14, 2019 at 11:45 AM Wes McKinney <we...@gmail.com> wrote:
>>
>> On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <em...@gmail.com> wrote:
>>>
>>>>
>>>> is there anything else that has come up that
>>>> definitely needs to happen before we can release again?
>>>
>>> We need to decide on a way forward for LargeList, LargeBinary, etc, types...
>>>
>>
>> Do these need to be dependent on the 64-bit array length discussion?
>> They seem somewhat orthogonal to me. If we have to release 0.15.0
>> without the Java side of these, that's OK with me, since reaching
>> format implementation completeness is more of a 1.0.0 concern
>>
>>> On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com> wrote:
>>>
>>>> hi folks,
>>>>
>>>> Since there have been a number of fairly serious issues (e.g.
>>>> ARROW-6060) since 0.14.1 that have been fixed I think we should start
>>>> planning of the next major release. Note that we still have some
>>>> format-related work (the Flatbuffers alignment issue) that ought to be
>>>> resolved (not a small task since it affects 4 or 5 implementations),
>>>> but aside from that, is there anything else that has come up that
>>>> definitely needs to happen before we can release again?
>>>>
>>>> I would say cutting a release somewhere around the US Labor Day
>>>> holiday (~the week after or so) would be called for.
>>>>
>>>> Thanks,
>>>> Wes
>>>>
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
For the record, I don't think we should hold a major release hostage
if we aren't able to complete various feature milestones in time.
Since it's been about 5-6 weeks since 0.14.0 we're coming close to the
desired 8-10 week timeline for major releases, so if we need to have
0.16.0 prior to 1.0.0, I think that is OK also.
On Wed, Aug 14, 2019 at 11:45 AM Wes McKinney <we...@gmail.com> wrote:
>
> On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <em...@gmail.com> wrote:
> >
> > >
> > > is there anything else that has come up that
> > > definitely needs to happen before we can release again?
> >
> > We need to decide on a way forward for LargeList, LargeBinary, etc, types...
> >
>
> Do these need to be dependent on the 64-bit array length discussion?
> They seem somewhat orthogonal to me. If we have to release 0.15.0
> without the Java side of these, that's OK with me, since reaching
> format implementation completeness is more of a 1.0.0 concern
>
> > On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi folks,
> > >
> > > Since there have been a number of fairly serious issues (e.g.
> > > ARROW-6060) since 0.14.1 that have been fixed I think we should start
> > > planning of the next major release. Note that we still have some
> > > format-related work (the Flatbuffers alignment issue) that ought to be
> > > resolved (not a small task since it affects 4 or 5 implementations),
> > > but aside from that, is there anything else that has come up that
> > > definitely needs to happen before we can release again?
> > >
> > > I would say cutting a release somewhere around the US Labor Day
> > > holiday (~the week after or so) would be called for.
> > >
> > > Thanks,
> > > Wes
> > >
Re: Timeline for 0.15.0 release
Posted by Wes McKinney <we...@gmail.com>.
On Wed, Aug 14, 2019 at 11:43 AM Micah Kornfield <em...@gmail.com> wrote:
>
> >
> > is there anything else that has come up that
> > definitely needs to happen before we can release again?
>
> We need to decide on a way forward for LargeList, LargeBinary, etc, types...
>
Do these need to be dependent on the 64-bit array length discussion?
They seem somewhat orthogonal to me. If we have to release 0.15.0
without the Java side of these, that's OK with me, since reaching
format implementation completeness is more of a 1.0.0 concern
> On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi folks,
> >
> > Since there have been a number of fairly serious issues (e.g.
> > ARROW-6060) since 0.14.1 that have been fixed I think we should start
> > planning of the next major release. Note that we still have some
> > format-related work (the Flatbuffers alignment issue) that ought to be
> > resolved (not a small task since it affects 4 or 5 implementations),
> > but aside from that, is there anything else that has come up that
> > definitely needs to happen before we can release again?
> >
> > I would say cutting a release somewhere around the US Labor Day
> > holiday (~the week after or so) would be called for.
> >
> > Thanks,
> > Wes
> >
Re: Timeline for 0.15.0 release
Posted by Micah Kornfield <em...@gmail.com>.
>
> is there anything else that has come up that
> definitely needs to happen before we can release again?
We need to decide on a way forward for LargeList, LargeBinary, etc, types...
On Tue, Aug 13, 2019 at 8:27 PM Wes McKinney <we...@gmail.com> wrote:
> hi folks,
>
> Since there have been a number of fairly serious issues (e.g.
> ARROW-6060) since 0.14.1 that have been fixed I think we should start
> planning of the next major release. Note that we still have some
> format-related work (the Flatbuffers alignment issue) that ought to be
> resolved (not a small task since it affects 4 or 5 implementations),
> but aside from that, is there anything else that has come up that
> definitely needs to happen before we can release again?
>
> I would say cutting a release somewhere around the US Labor Day
> holiday (~the week after or so) would be called for.
>
> Thanks,
> Wes
>