You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Siddharth Teotia <si...@dremio.com> on 2018/02/25 01:09:14 UTC

Working towards getting 0.9.0 release candidate up next week

Hi All,

We currently have 10 issues in progress and PRs are available for 8 of
them. In interest of getting a release candidate next week, I would request
people to review PRs as soon as they can to help make progress and close
out as many JIRAs as we can.

There are 32 issues in TODO list and 25 of them are not yet assigned. I am
planning to defer some of the unassigned ones later today or tomorrow. It
would be good to soon grab/assign the issues that people want to be fixed
for 0.9.0.

Here is the link to backlog:
https://issues.apache.org/jira/projects/ARROW/versions/12341707

Thanks,
Sidd

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Philipp Moritz <pc...@gmail.com>.

Thanks, I'm trying to reproduce it now!

On Thu, Mar 15, 2018 at 1:28 PM, Wes McKinney <we...@gmail.com> wrote:

> Unfortunately, we aren't quite out of the woods yet.
>
> https://issues.apache.org/jira/browse/ARROW-2318
>
> Robert or Philipp, could you please advise on this issue?
>
> I think if we're learning anything from this experience, we need to
> work on our continuous integration and delivery. We shouldn't be
> finding out about all these problems at the least minute -- the master
> branch should be kept in a releasable state every day (i.e. the
> release verification script passes on a snapshot of master)
>
> On Thu, Mar 15, 2018 at 2:06 AM, Kouhei Sutou <ko...@clear-code.com> wrote:
> > https://github.com/apache/arrow/pull/1752 will solve this.
> >
> > --
> > kou
> >
> > In <20...@clear-code.com>
> >   "Re: Working towards getting 0.9.0 release candidate up next week" on
> Thu, 15 Mar 2018 13:51:46 +0900 (JST),
> >   Kouhei Sutou <ko...@clear-code.com> wrote:
> >
> >> Sure. Please wait for a while.
> >>
> >> --
> >> kou
> >>
> >> In <CA...@mail.gmail.com>
> >>   "Re: Working towards getting 0.9.0 release candidate up next week" on
> Thu, 15 Mar 2018 00:49:02 -0400,
> >>   Wes McKinney <we...@gmail.com> wrote:
> >>
> >>> I ran into another problem:
> >>>
> >>> https://issues.apache.org/jira/browse/ARROW-2313
> >>>
> >>> @Kou, could you have a look at this? Thanks
> >>>
> >>> On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <we...@gmail.com>
> wrote:
> >>>> Thanks Paul! This seems to have done the trick. I will get the RC out
> >>>> later this evening
> >>>>
> >>>> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org>
> wrote:
> >>>>> This should fix it: https://github.com/apache/arrow/pull/1751 <
> https://github.com/apache/arrow/pull/1751>
> >>>>>
> >>>>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Last item https://issues.apache.org/jira/browse/ARROW-2312
> >>>>>>
> >>>>>> I can start the release vote as soon as we have the release
> >>>>>> verification script working again
> >>>>>>
> >>>>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com>
> wrote:
> >>>>>>> OK, patch is up for ARROW-2307:
> >>>>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed
> and
> >>>>>>> merged I will start the release vote
> >>>>>>>
> >>>>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy
> fix. If
> >>>>>>>> not, I will go ahead with the RC
> >>>>>>>>
> >>>>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com>
> wrote:
> >>>>>>>>> The mentioned bugfixes were merged and I also tested the
> Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both
> seem to work fine. From my side it looks like we're ready to make an RC.
> >>>>>>>>>
> >>>>>>>>> Uwe
> >>>>>>>>>
> >>>>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
> >>>>>>>>>> I fixed these bugs
> >>>>>>>>>>
> >>>>>>>>>> https://github.com/apache/arrow/pull/1742
> >>>>>>>>>> https://github.com/apache/arrow/pull/1743
> >>>>>>>>>>
> >>>>>>>>>> As soon as these patches go in, we can cut the RC0. I can do
> this as
> >>>>>>>>>> soon as tomorrow (Wednesday) morning
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing
> some final
> >>>>>>>>>>> testing of master (stuff we haven't been testing in CI -- we
> _really_
> >>>>>>>>>>> need to set up nightly CI jobs for more time consuming tests,
> like
> >>>>>>>>>>> HDFS, that we want to test periodically but perhaps not on
> every
> >>>>>>>>>>> commit). I'm going to see if these are easy to fix
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/
> arrow/pull/1740, so if
> >>>>>>>>>>>> someone could review that would be great.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am going to make sure the C++/Python HDFS tests run
> locally, then
> >>>>>>>>>>>> after the patch above is merged we should be good to make the
> RC.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note to other onlookers -- please feel free to keep writing
> new
> >>>>>>>>>>>> patches not mentioned here. They may just have to go in to
> the next
> >>>>>>>>>>>> release
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>> Wes
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for
> ARROW-1643
> >>>>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that
> we ought to
> >>>>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the
> vote after
> >>>>>>>>>>>>> that if nothing else comes up
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I created ARROW-2300 in the course of trying to test
> ARROW-1643. I
> >>>>>>>>>>>>> have an alternative way to run the HDFS tests (since we
> aren't running
> >>>>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS
> tests and
> >>>>>>>>>>>>> then move ARROW-2300 to the next release milestone.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>> Wes
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're
> going to
> >>>>>>>>>>>>>> need at least another full day to get things buttoned up,
> so I would
> >>>>>>>>>>>>>> say we're looking at an RC on Tuesday.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In progress:
> >>>>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to
> 0.10.0
> >>>>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
> >>>>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may
> want to
> >>>>>>>>>>>>>> move this JIRA to the Parquet project
> >>>>>>>>>>>>>> - ARROW-2122: Patch to be reviewed
> >>>>>>>>>>>>>> - ARROW-2135: Patch to be merged
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> TODO
> >>>>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may
> be moved to 0.10.0
> >>>>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
> >>>>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in
> Table.from_pandas
> >>>>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are
> small
> >>>>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be
> fixable
> >>>>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in
> case the
> >>>>>>>>>>>>>> fix is easy -- multiple users hit this bug.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Wes
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <
> kou@clear-code.com> wrote:
> >>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> kou
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In <CAJPUwMBsxXW9-8VRJ3xWUd_P3Y7soSDMC6xc53PKrJuSeWmxdw@
> mail.gmail.com>
> >>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up
> next week" on Thu, 8 Mar 2018 20:44:14 -0500,
> >>>>>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will
> require a little
> >>>>>>>>>>>>>>>> bit of work to be able to produce a viable standalone
> source tarball.
> >>>>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able
> to come up
> >>>>>>>>>>>>>>>> with a plan to do this
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Wes
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <
> kou@clear-code.com> wrote:
> >>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
> >>>>>>>>>>>>>>>>> 0.9.0?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by
> .deb
> >>>>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
> >>>>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
> >>>>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
> >>>>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
> >>>>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a
> source
> >>>>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
> >>>>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
> >>>>>>>>>>>>>>>>> pyarrow".
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or
> .rpm.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> kou
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In <CAKHW1aqGv7SN70JewE5QmW00yeVX
> _Xg_ZjinywgBo00+c4Khxw@mail.gmail.com>
> >>>>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up
> next week" on Thu, 8 Mar 2018 11:25:32 -0800,
> >>>>>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> All,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday.
> Is that fine with
> >>>>>>>>>>>>>>>>>> everybody?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have 6 items in progress --
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/
> 12341707#release-report-tab-body.
> >>>>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by
> tomorrow? I am
> >>>>>>>>>>>>>>>>>> completely fine with deferring the RC to early next
> week (Mon/Tue/Wed) if
> >>>>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest
> that we defer the
> >>>>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless
> I hear otherwise.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I was wondering if anyone else is interested in
> collaborating for the
> >>>>>>>>>>>>>>>>>> post-release tasks. As per
> >>>>>>>>>>>>>>>>>> https://github.com/apache/
> arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
> >>>>>>>>>>>>>>>>>> following are the high level post-release tasks. Please
> let me know if you
> >>>>>>>>>>>>>>>>>> would like to take up something. I have written my name
> against some of
> >>>>>>>>>>>>>>>>>> them.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
> >>>>>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like
> PMC karma is needed
> >>>>>>>>>>>>>>>>>>   to do this
> >>>>>>>>>>>>>>>>>>   - Announcing release (Sidd)
> >>>>>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
> >>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
> >>>>>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
> >>>>>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central
> (Sidd)
> >>>>>>>>>>>>>>>>>>   - Release blog post
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If anything is missing, please add to the above list.
> It will be helpful
> >>>>>>>>>>>>>>>>>> for tracking.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Sidd
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> hey Sidd,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd
> like to see if
> >>>>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to
> Monday/Tuesday
> >>>>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much
> as possible out
> >>>>>>>>>>>>>>>>>>> of the current backlog to get things down to the
> essential
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - Wes
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <
> siddharth@dremio.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> Sounds good.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>> Sidd
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <
> wesmckinn@gmail.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Sidd,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this
> coming week. I suggest
> >>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> defer to the following week.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>> Wes
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <
> siddharth@dremio.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are
> available for 8 of
> >>>>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next
> week, I would
> >>>>>>>>>>>>>>>>>>> request
> >>>>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make
> progress and close
> >>>>>>>>>>>>>>>>>>>> out as many JIRAs as we can.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are
> not yet assigned. I
> >>>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later
> today or tomorrow. It
> >>>>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that
> people want to be fixed
> >>>>>>>>>>>>>>>>>>>> for 0.9.0.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Here is the link to backlog:
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/projects/ARROW/versions/12341707
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> Sidd
> >>>>>>>>>>>>>>>>>>>
> >>>>>
>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Unfortunately, we aren't quite out of the woods yet.

https://issues.apache.org/jira/browse/ARROW-2318

Robert or Philipp, could you please advise on this issue?

I think if we're learning anything from this experience, we need to
work on our continuous integration and delivery. We shouldn't be
finding out about all these problems at the least minute -- the master
branch should be kept in a releasable state every day (i.e. the
release verification script passes on a snapshot of master)

On Thu, Mar 15, 2018 at 2:06 AM, Kouhei Sutou <ko...@clear-code.com> wrote:
> https://github.com/apache/arrow/pull/1752 will solve this.
>
> --
> kou
>
> In <20...@clear-code.com>
>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 15 Mar 2018 13:51:46 +0900 (JST),
>   Kouhei Sutou <ko...@clear-code.com> wrote:
>
>> Sure. Please wait for a while.
>>
>> --
>> kou
>>
>> In <CA...@mail.gmail.com>
>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 15 Mar 2018 00:49:02 -0400,
>>   Wes McKinney <we...@gmail.com> wrote:
>>
>>> I ran into another problem:
>>>
>>> https://issues.apache.org/jira/browse/ARROW-2313
>>>
>>> @Kou, could you have a look at this? Thanks
>>>
>>> On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <we...@gmail.com> wrote:
>>>> Thanks Paul! This seems to have done the trick. I will get the RC out
>>>> later this evening
>>>>
>>>> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org> wrote:
>>>>> This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>
>>>>>
>>>>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>
>>>>>> Last item https://issues.apache.org/jira/browse/ARROW-2312
>>>>>>
>>>>>> I can start the release vote as soon as we have the release
>>>>>> verification script working again
>>>>>>
>>>>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>> OK, patch is up for ARROW-2307:
>>>>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>>>>>>> merged I will start the release vote
>>>>>>>
>>>>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>>>>>>> not, I will go ahead with the RC
>>>>>>>>
>>>>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>>>>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>>>>>>>
>>>>>>>>> Uwe
>>>>>>>>>
>>>>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>>>>>>> I fixed these bugs
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/arrow/pull/1742
>>>>>>>>>> https://github.com/apache/arrow/pull/1743
>>>>>>>>>>
>>>>>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>>>>>>> soon as tomorrow (Wednesday) morning
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>>>>>>> commit). I'm going to see if these are easy to fix
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>>>>>>> someone could review that would be great.
>>>>>>>>>>>>
>>>>>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>>>>>>>
>>>>>>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>>>>>>> release
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Wes
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>>>>>>> that if nothing else comes up
>>>>>>>>>>>>>
>>>>>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In progress:
>>>>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TODO
>>>>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>>>>>>> with a plan to do this
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>>
>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Kouhei Sutou <ko...@clear-code.com>.

https://github.com/apache/arrow/pull/1752 will solve this.

--
kou

In <20...@clear-code.com>
  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 15 Mar 2018 13:51:46 +0900 (JST),
  Kouhei Sutou <ko...@clear-code.com> wrote:

> Sure. Please wait for a while.
> 
> --
> kou
> 
> In <CA...@mail.gmail.com>
>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 15 Mar 2018 00:49:02 -0400,
>   Wes McKinney <we...@gmail.com> wrote:
> 
>> I ran into another problem:
>> 
>> https://issues.apache.org/jira/browse/ARROW-2313
>> 
>> @Kou, could you have a look at this? Thanks
>> 
>> On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <we...@gmail.com> wrote:
>>> Thanks Paul! This seems to have done the trick. I will get the RC out
>>> later this evening
>>>
>>> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org> wrote:
>>>> This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>
>>>>
>>>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>
>>>>> Last item https://issues.apache.org/jira/browse/ARROW-2312
>>>>>
>>>>> I can start the release vote as soon as we have the release
>>>>> verification script working again
>>>>>
>>>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>> OK, patch is up for ARROW-2307:
>>>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>>>>>> merged I will start the release vote
>>>>>>
>>>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>>>>>> not, I will go ahead with the RC
>>>>>>>
>>>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>>>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>>>>>>
>>>>>>>> Uwe
>>>>>>>>
>>>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>>>>>> I fixed these bugs
>>>>>>>>>
>>>>>>>>> https://github.com/apache/arrow/pull/1742
>>>>>>>>> https://github.com/apache/arrow/pull/1743
>>>>>>>>>
>>>>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>>>>>> soon as tomorrow (Wednesday) morning
>>>>>>>>>
>>>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>>>>>> commit). I'm going to see if these are easy to fix
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>>>>>> someone could review that would be great.
>>>>>>>>>>>
>>>>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>>>>>>
>>>>>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>>>>>> release
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Wes
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>>>>>> that if nothing else comes up
>>>>>>>>>>>>
>>>>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Wes
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In progress:
>>>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>>>>>>
>>>>>>>>>>>>> TODO
>>>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>>>>>>
>>>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>>>>>> with a plan to do this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>
>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Kouhei Sutou <ko...@clear-code.com>.

Sure. Please wait for a while.

--
kou

In <CA...@mail.gmail.com>
  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 15 Mar 2018 00:49:02 -0400,
  Wes McKinney <we...@gmail.com> wrote:

> I ran into another problem:
> 
> https://issues.apache.org/jira/browse/ARROW-2313
> 
> @Kou, could you have a look at this? Thanks
> 
> On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <we...@gmail.com> wrote:
>> Thanks Paul! This seems to have done the trick. I will get the RC out
>> later this evening
>>
>> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org> wrote:
>>> This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>
>>>
>>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> Last item https://issues.apache.org/jira/browse/ARROW-2312
>>>>
>>>> I can start the release vote as soon as we have the release
>>>> verification script working again
>>>>
>>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>> OK, patch is up for ARROW-2307:
>>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>>>>> merged I will start the release vote
>>>>>
>>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>>>>> not, I will go ahead with the RC
>>>>>>
>>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>>>>>
>>>>>>> Uwe
>>>>>>>
>>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>>>>> I fixed these bugs
>>>>>>>>
>>>>>>>> https://github.com/apache/arrow/pull/1742
>>>>>>>> https://github.com/apache/arrow/pull/1743
>>>>>>>>
>>>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>>>>> soon as tomorrow (Wednesday) morning
>>>>>>>>
>>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>>>>> commit). I'm going to see if these are easy to fix
>>>>>>>>>
>>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>>>>> someone could review that would be great.
>>>>>>>>>>
>>>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>>>>>
>>>>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>>>>> release
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Wes
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>>>>> that if nothing else comes up
>>>>>>>>>>>
>>>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Wes
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>>>>>
>>>>>>>>>>>> In progress:
>>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>>>>>
>>>>>>>>>>>> TODO
>>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>>>>>
>>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Wes
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> kou
>>>>>>>>>>>>>
>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>>>>> with a plan to do this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>
>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I ran into another problem:

https://issues.apache.org/jira/browse/ARROW-2313

@Kou, could you have a look at this? Thanks

On Wed, Mar 14, 2018 at 10:36 PM, Wes McKinney <we...@gmail.com> wrote:
> Thanks Paul! This seems to have done the trick. I will get the RC out
> later this evening
>
> On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org> wrote:
>> This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>
>>
>>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
>>>
>>> Last item https://issues.apache.org/jira/browse/ARROW-2312
>>>
>>> I can start the release vote as soon as we have the release
>>> verification script working again
>>>
>>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>>>> OK, patch is up for ARROW-2307:
>>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>>>> merged I will start the release vote
>>>>
>>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>>>> not, I will go ahead with the RC
>>>>>
>>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>>>> I fixed these bugs
>>>>>>>
>>>>>>> https://github.com/apache/arrow/pull/1742
>>>>>>> https://github.com/apache/arrow/pull/1743
>>>>>>>
>>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>>>> soon as tomorrow (Wednesday) morning
>>>>>>>
>>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>>>> commit). I'm going to see if these are easy to fix
>>>>>>>>
>>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>>>> someone could review that would be great.
>>>>>>>>>
>>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>>>>
>>>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>>>> release
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Wes
>>>>>>>>>
>>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>>>> that if nothing else comes up
>>>>>>>>>>
>>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Wes
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>>>>
>>>>>>>>>>> In progress:
>>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>>>>
>>>>>>>>>>> TODO
>>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>>>>
>>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Wes
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> kou
>>>>>>>>>>>>
>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>>>> with a plan to do this
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> kou
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>
>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Thanks Paul! This seems to have done the trick. I will get the RC out
later this evening

On Wed, Mar 14, 2018 at 9:50 PM, Paul Taylor <pt...@apache.org> wrote:
> This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>
>
>> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
>>
>> Last item https://issues.apache.org/jira/browse/ARROW-2312
>>
>> I can start the release vote as soon as we have the release
>> verification script working again
>>
>> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>>> OK, patch is up for ARROW-2307:
>>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>>> merged I will start the release vote
>>>
>>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>>> not, I will go ahead with the RC
>>>>
>>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>>>
>>>>> Uwe
>>>>>
>>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>>> I fixed these bugs
>>>>>>
>>>>>> https://github.com/apache/arrow/pull/1742
>>>>>> https://github.com/apache/arrow/pull/1743
>>>>>>
>>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>>> soon as tomorrow (Wednesday) morning
>>>>>>
>>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>>> commit). I'm going to see if these are easy to fix
>>>>>>>
>>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>>> someone could review that would be great.
>>>>>>>>
>>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>>>
>>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>>> release
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Wes
>>>>>>>>
>>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>>> that if nothing else comes up
>>>>>>>>>
>>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Wes
>>>>>>>>>
>>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>>>
>>>>>>>>>> In progress:
>>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>>>
>>>>>>>>>> TODO
>>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>>>
>>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Wes
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> kou
>>>>>>>>>>>
>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>>> with a plan to do this
>>>>>>>>>>>>
>>>>>>>>>>>> - Wes
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --
>>>>>>>>>>>>> kou
>>>>>>>>>>>>>
>>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>>
>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Paul Taylor <pt...@apache.org>.

This should fix it: https://github.com/apache/arrow/pull/1751 <https://github.com/apache/arrow/pull/1751>

> On Mar 14, 2018, at 6:42 PM, Wes McKinney <we...@gmail.com> wrote:
> 
> Last item https://issues.apache.org/jira/browse/ARROW-2312
> 
> I can start the release vote as soon as we have the release
> verification script working again
> 
> On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
>> OK, patch is up for ARROW-2307:
>> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
>> merged I will start the release vote
>> 
>> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>>> not, I will go ahead with the RC
>>> 
>>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>> 
>>>> Uwe
>>>> 
>>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>>> I fixed these bugs
>>>>> 
>>>>> https://github.com/apache/arrow/pull/1742
>>>>> https://github.com/apache/arrow/pull/1743
>>>>> 
>>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>>> soon as tomorrow (Wednesday) morning
>>>>> 
>>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>>>> testing of master (stuff we haven't been testing in CI -- we _really_
>>>>>> need to set up nightly CI jobs for more time consuming tests, like
>>>>>> HDFS, that we want to test periodically but perhaps not on every
>>>>>> commit). I'm going to see if these are easy to fix
>>>>>> 
>>>>>> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>>>>> someone could review that would be great.
>>>>>>> 
>>>>>>> I am going to make sure the C++/Python HDFS tests run locally, then
>>>>>>> after the patch above is merged we should be good to make the RC.
>>>>>>> 
>>>>>>> Note to other onlookers -- please feel free to keep writing new
>>>>>>> patches not mentioned here. They may just have to go in to the next
>>>>>>> release
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Wes
>>>>>>> 
>>>>>>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>>>>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>>>>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>>>>>> that if nothing else comes up
>>>>>>>> 
>>>>>>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>>>>>> have an alternative way to run the HDFS tests (since we aren't running
>>>>>>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>>>>>> then move ARROW-2300 to the next release milestone.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Wes
>>>>>>>> 
>>>>>>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>>>>>>> need at least another full day to get things buttoned up, so I would
>>>>>>>>> say we're looking at an RC on Tuesday.
>>>>>>>>> 
>>>>>>>>> In progress:
>>>>>>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>>>>>>> - ARROW-2282: Patch up, needs some test cases
>>>>>>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>>>>>>> move this JIRA to the Parquet project
>>>>>>>>> - ARROW-2122: Patch to be reviewed
>>>>>>>>> - ARROW-2135: Patch to be merged
>>>>>>>>> 
>>>>>>>>> TODO
>>>>>>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>>>>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>>>>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>>>>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>>>>>> 
>>>>>>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>>>>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>>>>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>>>>>>> fix is easy -- multiple users hit this bug.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Wes
>>>>>>>>> 
>>>>>>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>> Thanks!
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> kou
>>>>>>>>>> 
>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>>>>>>  Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>>>>>>> with a plan to do this
>>>>>>>>>>> 
>>>>>>>>>>> - Wes
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>> 
>>>>>>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>>>>>>> 0.9.0?
>>>>>>>>>>>> 
>>>>>>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>>>>>> 
>>>>>>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>>>>>>> pyarrow".
>>>>>>>>>>>> 
>>>>>>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --
>>>>>>>>>>>> kou
>>>>>>>>>>>> 
>>>>>>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>>>>>>  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>>>>>>  Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>>>>>>> everybody?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We have 6 items in progress --
>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>>>>>>> post-release tasks. As per
>>>>>>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>>>>>>> them.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>   - Updating the Arrow Website (Sidd)
>>>>>>>>>>>>>   - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>>>>>>   to do this
>>>>>>>>>>>>>   - Announcing release (Sidd)
>>>>>>>>>>>>>   - Updating website with new API documentation (Sidd)
>>>>>>>>>>>>>   - Updating pip packages for C++ and Python
>>>>>>>>>>>>>   - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>>>>>>   - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>>>>>>   - Release blog post
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>>>>>>> for tracking.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> hey Sidd,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - Wes
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Sounds good.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Sidd,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> defer to the following week.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Wes
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>>>>>>>> them. In interest of getting a release candidate next week, I would
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>> people to review PRs as soon as they can to help make progress and close
>>>>>>>>>>>>>>> out as many JIRAs as we can.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>> planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>>>>>>>> would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>>>>>>>> for 0.9.0.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here is the link to backlog:
>>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Sidd
>>>>>>>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Last item https://issues.apache.org/jira/browse/ARROW-2312

I can start the release vote as soon as we have the release
verification script working again

On Wed, Mar 14, 2018 at 2:28 PM, Wes McKinney <we...@gmail.com> wrote:
> OK, patch is up for ARROW-2307:
> https://github.com/apache/arrow/pull/1747. Once that is reviewed and
> merged I will start the release vote
>
> On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
>> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
>> not, I will go ahead with the RC
>>
>> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>>
>>> Uwe
>>>
>>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>>> I fixed these bugs
>>>>
>>>> https://github.com/apache/arrow/pull/1742
>>>> https://github.com/apache/arrow/pull/1743
>>>>
>>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>>> soon as tomorrow (Wednesday) morning
>>>>
>>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>>> > I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>>> > testing of master (stuff we haven't been testing in CI -- we _really_
>>>> > need to set up nightly CI jobs for more time consuming tests, like
>>>> > HDFS, that we want to test periodically but perhaps not on every
>>>> > commit). I'm going to see if these are easy to fix
>>>> >
>>>> > On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>>> >> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>>> >> someone could review that would be great.
>>>> >>
>>>> >> I am going to make sure the C++/Python HDFS tests run locally, then
>>>> >> after the patch above is merged we should be good to make the RC.
>>>> >>
>>>> >> Note to other onlookers -- please feel free to keep writing new
>>>> >> patches not mentioned here. They may just have to go in to the next
>>>> >> release
>>>> >>
>>>> >> Thanks
>>>> >> Wes
>>>> >>
>>>> >> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>>> >>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>>> >>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>>> >>> try to fix tomorrow before cutting the RC. I can start the vote after
>>>> >>> that if nothing else comes up
>>>> >>>
>>>> >>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>>> >>> have an alternative way to run the HDFS tests (since we aren't running
>>>> >>> them in Travis CI) so most likely I will try out the HDFS tests and
>>>> >>> then move ARROW-2300 to the next release milestone.
>>>> >>>
>>>> >>> Thanks
>>>> >>> Wes
>>>> >>>
>>>> >>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>> >>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>> >>>> need at least another full day to get things buttoned up, so I would
>>>> >>>> say we're looking at an RC on Tuesday.
>>>> >>>>
>>>> >>>> In progress:
>>>> >>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>> >>>> - ARROW-2282: Patch up, needs some test cases
>>>> >>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>> >>>> move this JIRA to the Parquet project
>>>> >>>> - ARROW-2122: Patch to be reviewed
>>>> >>>> - ARROW-2135: Patch to be merged
>>>> >>>>
>>>> >>>> TODO
>>>> >>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>> >>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>> >>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>> >>>> - ARROW-2292: Deprecation / renaming a Python method
>>>> >>>>
>>>> >>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>> >>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>> >>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>> >>>> fix is easy -- multiple users hit this bug.
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Wes
>>>> >>>>
>>>> >>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>> >>>>> Thanks!
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> kou
>>>> >>>>>
>>>> >>>>> In <CA...@mail.gmail.com>
>>>> >>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>> >>>>>   Wes McKinney <we...@gmail.com> wrote:
>>>> >>>>>
>>>> >>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>> >>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>> >>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>> >>>>>> with a plan to do this
>>>> >>>>>>
>>>> >>>>>> - Wes
>>>> >>>>>>
>>>> >>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>> >>>>>>> Hi,
>>>> >>>>>>>
>>>> >>>>>>>>    - Updating pip packages for C++ and Python
>>>> >>>>>>>
>>>> >>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>> >>>>>>> 0.9.0?
>>>> >>>>>>>
>>>> >>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>> >>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>> >>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>> >>>>>>> to integrate with Python). In the case, I need to use the
>>>> >>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>> >>>>>>>
>>>> >>>>>>> Now, there are only binary packages for PyArrow at
>>>> >>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>> >>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>> >>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>> >>>>>>> pyarrow".
>>>> >>>>>>>
>>>> >>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Thanks,
>>>> >>>>>>> --
>>>> >>>>>>> kou
>>>> >>>>>>>
>>>> >>>>>>> In <CA...@mail.gmail.com>
>>>> >>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>> >>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>>> All,
>>>> >>>>>>>>
>>>> >>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>> >>>>>>>> everybody?
>>>> >>>>>>>>
>>>> >>>>>>>> We have 6 items in progress --
>>>> >>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>> >>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>> >>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>> >>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>> >>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>> >>>>>>>>
>>>> >>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>> >>>>>>>> post-release tasks. As per
>>>> >>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>> >>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>> >>>>>>>> would like to take up something. I have written my name against some of
>>>> >>>>>>>> them.
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>    - Updating the Arrow Website (Sidd)
>>>> >>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>> >>>>>>>>    to do this
>>>> >>>>>>>>    - Announcing release (Sidd)
>>>> >>>>>>>>    - Updating website with new API documentation (Sidd)
>>>> >>>>>>>>    - Updating pip packages for C++ and Python
>>>> >>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>> >>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>> >>>>>>>>    - Release blog post
>>>> >>>>>>>>
>>>> >>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>> >>>>>>>> for tracking.
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks,
>>>> >>>>>>>> Sidd
>>>> >>>>>>>>
>>>> >>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> hey Sidd,
>>>> >>>>>>>>>
>>>> >>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>> >>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>> >>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>> >>>>>>>>> of the current backlog to get things down to the essential
>>>> >>>>>>>>>
>>>> >>>>>>>>> - Wes
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>> >>>>>>>>> wrote:
>>>> >>>>>>>>> > Sounds good.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Thanks
>>>> >>>>>>>>> > Sidd
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Hi Sidd,
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>> >>>>>>>>> we
>>>> >>>>>>>>> > defer to the following week.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Thanks
>>>> >>>>>>>>> > Wes
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>> >>>>>>>>> wrote:
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Hi All,
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>> >>>>>>>>> > them. In interest of getting a release candidate next week, I would
>>>> >>>>>>>>> request
>>>> >>>>>>>>> > people to review PRs as soon as they can to help make progress and close
>>>> >>>>>>>>> > out as many JIRAs as we can.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>> >>>>>>>>> am
>>>> >>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>> >>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>> >>>>>>>>> > for 0.9.0.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Here is the link to backlog:
>>>> >>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Thanks,
>>>> >>>>>>>>> > Sidd
>>>> >>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

OK, patch is up for ARROW-2307:
https://github.com/apache/arrow/pull/1747. Once that is reviewed and
merged I will start the release vote

On Wed, Mar 14, 2018 at 10:18 AM, Wes McKinney <we...@gmail.com> wrote:
> I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
> not, I will go ahead with the RC
>
> On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>>
>> Uwe
>>
>> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>>> I fixed these bugs
>>>
>>> https://github.com/apache/arrow/pull/1742
>>> https://github.com/apache/arrow/pull/1743
>>>
>>> As soon as these patches go in, we can cut the RC0. I can do this as
>>> soon as tomorrow (Wednesday) morning
>>>
>>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>>> > I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>>> > testing of master (stuff we haven't been testing in CI -- we _really_
>>> > need to set up nightly CI jobs for more time consuming tests, like
>>> > HDFS, that we want to test periodically but perhaps not on every
>>> > commit). I'm going to see if these are easy to fix
>>> >
>>> > On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>>> >> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>>> >> someone could review that would be great.
>>> >>
>>> >> I am going to make sure the C++/Python HDFS tests run locally, then
>>> >> after the patch above is merged we should be good to make the RC.
>>> >>
>>> >> Note to other onlookers -- please feel free to keep writing new
>>> >> patches not mentioned here. They may just have to go in to the next
>>> >> release
>>> >>
>>> >> Thanks
>>> >> Wes
>>> >>
>>> >> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>> >>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>> >>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>> >>> try to fix tomorrow before cutting the RC. I can start the vote after
>>> >>> that if nothing else comes up
>>> >>>
>>> >>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>> >>> have an alternative way to run the HDFS tests (since we aren't running
>>> >>> them in Travis CI) so most likely I will try out the HDFS tests and
>>> >>> then move ARROW-2300 to the next release milestone.
>>> >>>
>>> >>> Thanks
>>> >>> Wes
>>> >>>
>>> >>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>> >>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>> >>>> need at least another full day to get things buttoned up, so I would
>>> >>>> say we're looking at an RC on Tuesday.
>>> >>>>
>>> >>>> In progress:
>>> >>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>> >>>> - ARROW-2282: Patch up, needs some test cases
>>> >>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>> >>>> move this JIRA to the Parquet project
>>> >>>> - ARROW-2122: Patch to be reviewed
>>> >>>> - ARROW-2135: Patch to be merged
>>> >>>>
>>> >>>> TODO
>>> >>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>> >>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>> >>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>> >>>> - ARROW-2292: Deprecation / renaming a Python method
>>> >>>>
>>> >>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>> >>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>> >>>> within ~1 day but it would be useful to have a diagnosis in case the
>>> >>>> fix is easy -- multiple users hit this bug.
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Wes
>>> >>>>
>>> >>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>> >>>>> Thanks!
>>> >>>>>
>>> >>>>> --
>>> >>>>> kou
>>> >>>>>
>>> >>>>> In <CA...@mail.gmail.com>
>>> >>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>> >>>>>   Wes McKinney <we...@gmail.com> wrote:
>>> >>>>>
>>> >>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>> >>>>>> bit of work to be able to produce a viable standalone source tarball.
>>> >>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>> >>>>>> with a plan to do this
>>> >>>>>>
>>> >>>>>> - Wes
>>> >>>>>>
>>> >>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>> >>>>>>> Hi,
>>> >>>>>>>
>>> >>>>>>>>    - Updating pip packages for C++ and Python
>>> >>>>>>>
>>> >>>>>>> Can we try adding PyArrow source package to PyPI at the
>>> >>>>>>> 0.9.0?
>>> >>>>>>>
>>> >>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>> >>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>> >>>>>>> PyArrow in the same process via PyCall (Ruby library
>>> >>>>>>> to integrate with Python). In the case, I need to use the
>>> >>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>> >>>>>>>
>>> >>>>>>> Now, there are only binary packages for PyArrow at
>>> >>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>> >>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>> >>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>> >>>>>>> pyarrow".
>>> >>>>>>>
>>> >>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> --
>>> >>>>>>> kou
>>> >>>>>>>
>>> >>>>>>> In <CA...@mail.gmail.com>
>>> >>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>> >>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>> >>>>>>>
>>> >>>>>>>> All,
>>> >>>>>>>>
>>> >>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>> >>>>>>>> everybody?
>>> >>>>>>>>
>>> >>>>>>>> We have 6 items in progress --
>>> >>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>> >>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>> >>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>> >>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>> >>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>> >>>>>>>>
>>> >>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>> >>>>>>>> post-release tasks. As per
>>> >>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>> >>>>>>>> following are the high level post-release tasks. Please let me know if you
>>> >>>>>>>> would like to take up something. I have written my name against some of
>>> >>>>>>>> them.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>    - Updating the Arrow Website (Sidd)
>>> >>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>> >>>>>>>>    to do this
>>> >>>>>>>>    - Announcing release (Sidd)
>>> >>>>>>>>    - Updating website with new API documentation (Sidd)
>>> >>>>>>>>    - Updating pip packages for C++ and Python
>>> >>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>> >>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>> >>>>>>>>    - Release blog post
>>> >>>>>>>>
>>> >>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>> >>>>>>>> for tracking.
>>> >>>>>>>>
>>> >>>>>>>> Thanks,
>>> >>>>>>>> Sidd
>>> >>>>>>>>
>>> >>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> hey Sidd,
>>> >>>>>>>>>
>>> >>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>> >>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>> >>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>> >>>>>>>>> of the current backlog to get things down to the essential
>>> >>>>>>>>>
>>> >>>>>>>>> - Wes
>>> >>>>>>>>>
>>> >>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>> > Sounds good.
>>> >>>>>>>>> >
>>> >>>>>>>>> > Thanks
>>> >>>>>>>>> > Sidd
>>> >>>>>>>>> >
>>> >>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>> >>>>>>>>> >
>>> >>>>>>>>> > Hi Sidd,
>>> >>>>>>>>> >
>>> >>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>> >>>>>>>>> we
>>> >>>>>>>>> > defer to the following week.
>>> >>>>>>>>> >
>>> >>>>>>>>> > Thanks
>>> >>>>>>>>> > Wes
>>> >>>>>>>>> >
>>> >>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>> >
>>> >>>>>>>>> > Hi All,
>>> >>>>>>>>> >
>>> >>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>> >>>>>>>>> > them. In interest of getting a release candidate next week, I would
>>> >>>>>>>>> request
>>> >>>>>>>>> > people to review PRs as soon as they can to help make progress and close
>>> >>>>>>>>> > out as many JIRAs as we can.
>>> >>>>>>>>> >
>>> >>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>> >>>>>>>>> am
>>> >>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>> >>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>> >>>>>>>>> > for 0.9.0.
>>> >>>>>>>>> >
>>> >>>>>>>>> > Here is the link to backlog:
>>> >>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>> >>>>>>>>> >
>>> >>>>>>>>> > Thanks,
>>> >>>>>>>>> > Sidd
>>> >>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I'm going to have a look at ARROW-2307 to see if it's an easy fix. If
not, I will go ahead with the RC

On Wed, Mar 14, 2018 at 7:36 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.
>
> Uwe
>
> On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
>> I fixed these bugs
>>
>> https://github.com/apache/arrow/pull/1742
>> https://github.com/apache/arrow/pull/1743
>>
>> As soon as these patches go in, we can cut the RC0. I can do this as
>> soon as tomorrow (Wednesday) morning
>>
>> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
>> > I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
>> > testing of master (stuff we haven't been testing in CI -- we _really_
>> > need to set up nightly CI jobs for more time consuming tests, like
>> > HDFS, that we want to test periodically but perhaps not on every
>> > commit). I'm going to see if these are easy to fix
>> >
>> > On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>> >> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>> >> someone could review that would be great.
>> >>
>> >> I am going to make sure the C++/Python HDFS tests run locally, then
>> >> after the patch above is merged we should be good to make the RC.
>> >>
>> >> Note to other onlookers -- please feel free to keep writing new
>> >> patches not mentioned here. They may just have to go in to the next
>> >> release
>> >>
>> >> Thanks
>> >> Wes
>> >>
>> >> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>> >>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>> >>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>> >>> try to fix tomorrow before cutting the RC. I can start the vote after
>> >>> that if nothing else comes up
>> >>>
>> >>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>> >>> have an alternative way to run the HDFS tests (since we aren't running
>> >>> them in Travis CI) so most likely I will try out the HDFS tests and
>> >>> then move ARROW-2300 to the next release milestone.
>> >>>
>> >>> Thanks
>> >>> Wes
>> >>>
>> >>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>> >>>> I've done a pass over the remaining JIRAs -- I think we're going to
>> >>>> need at least another full day to get things buttoned up, so I would
>> >>>> say we're looking at an RC on Tuesday.
>> >>>>
>> >>>> In progress:
>> >>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>> >>>> - ARROW-2282: Patch up, needs some test cases
>> >>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>> >>>> move this JIRA to the Parquet project
>> >>>> - ARROW-2122: Patch to be reviewed
>> >>>> - ARROW-2135: Patch to be merged
>> >>>>
>> >>>> TODO
>> >>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>> >>>> - ARROW-2118: Fix rough edge with reading length-0 files
>> >>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>> >>>> - ARROW-2292: Deprecation / renaming a Python method
>> >>>>
>> >>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>> >>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>> >>>> within ~1 day but it would be useful to have a diagnosis in case the
>> >>>> fix is easy -- multiple users hit this bug.
>> >>>>
>> >>>> Thanks,
>> >>>> Wes
>> >>>>
>> >>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>> >>>>> Thanks!
>> >>>>>
>> >>>>> --
>> >>>>> kou
>> >>>>>
>> >>>>> In <CA...@mail.gmail.com>
>> >>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>> >>>>>   Wes McKinney <we...@gmail.com> wrote:
>> >>>>>
>> >>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>> >>>>>> bit of work to be able to produce a viable standalone source tarball.
>> >>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>> >>>>>> with a plan to do this
>> >>>>>>
>> >>>>>> - Wes
>> >>>>>>
>> >>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>>>    - Updating pip packages for C++ and Python
>> >>>>>>>
>> >>>>>>> Can we try adding PyArrow source package to PyPI at the
>> >>>>>>> 0.9.0?
>> >>>>>>>
>> >>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>> >>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>> >>>>>>> PyArrow in the same process via PyCall (Ruby library
>> >>>>>>> to integrate with Python). In the case, I need to use the
>> >>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>> >>>>>>>
>> >>>>>>> Now, there are only binary packages for PyArrow at
>> >>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>> >>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>> >>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>> >>>>>>> pyarrow".
>> >>>>>>>
>> >>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> --
>> >>>>>>> kou
>> >>>>>>>
>> >>>>>>> In <CA...@mail.gmail.com>
>> >>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>> >>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>> >>>>>>>
>> >>>>>>>> All,
>> >>>>>>>>
>> >>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>> >>>>>>>> everybody?
>> >>>>>>>>
>> >>>>>>>> We have 6 items in progress --
>> >>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>> >>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>> >>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>> >>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>> >>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>> >>>>>>>>
>> >>>>>>>> I was wondering if anyone else is interested in collaborating for the
>> >>>>>>>> post-release tasks. As per
>> >>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>> >>>>>>>> following are the high level post-release tasks. Please let me know if you
>> >>>>>>>> would like to take up something. I have written my name against some of
>> >>>>>>>> them.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>    - Updating the Arrow Website (Sidd)
>> >>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>> >>>>>>>>    to do this
>> >>>>>>>>    - Announcing release (Sidd)
>> >>>>>>>>    - Updating website with new API documentation (Sidd)
>> >>>>>>>>    - Updating pip packages for C++ and Python
>> >>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>> >>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>> >>>>>>>>    - Release blog post
>> >>>>>>>>
>> >>>>>>>> If anything is missing, please add to the above list. It will be helpful
>> >>>>>>>> for tracking.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Sidd
>> >>>>>>>>
>> >>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> hey Sidd,
>> >>>>>>>>>
>> >>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>> >>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>> >>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>> >>>>>>>>> of the current backlog to get things down to the essential
>> >>>>>>>>>
>> >>>>>>>>> - Wes
>> >>>>>>>>>
>> >>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>> > Sounds good.
>> >>>>>>>>> >
>> >>>>>>>>> > Thanks
>> >>>>>>>>> > Sidd
>> >>>>>>>>> >
>> >>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>> >>>>>>>>> >
>> >>>>>>>>> > Hi Sidd,
>> >>>>>>>>> >
>> >>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>> >>>>>>>>> we
>> >>>>>>>>> > defer to the following week.
>> >>>>>>>>> >
>> >>>>>>>>> > Thanks
>> >>>>>>>>> > Wes
>> >>>>>>>>> >
>> >>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>> >
>> >>>>>>>>> > Hi All,
>> >>>>>>>>> >
>> >>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>> >>>>>>>>> > them. In interest of getting a release candidate next week, I would
>> >>>>>>>>> request
>> >>>>>>>>> > people to review PRs as soon as they can to help make progress and close
>> >>>>>>>>> > out as many JIRAs as we can.
>> >>>>>>>>> >
>> >>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>> >>>>>>>>> am
>> >>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>> >>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>> >>>>>>>>> > for 0.9.0.
>> >>>>>>>>> >
>> >>>>>>>>> > Here is the link to backlog:
>> >>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>> >>>>>>>>> >
>> >>>>>>>>> > Thanks,
>> >>>>>>>>> > Sidd
>> >>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by "Uwe L. Korn" <uw...@xhochy.com>.

The mentioned bugfixes were merged and I also tested the Arrow<->Parquet-cpp as well as the Arrow<->Dask(parquet) integration. Both seem to work fine. From my side it looks like we're ready to make an RC.

Uwe

On Wed, Mar 14, 2018, at 5:07 AM, Wes McKinney wrote:
> I fixed these bugs
> 
> https://github.com/apache/arrow/pull/1742
> https://github.com/apache/arrow/pull/1743
> 
> As soon as these patches go in, we can cut the RC0. I can do this as
> soon as tomorrow (Wednesday) morning
> 
> On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
> > I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
> > testing of master (stuff we haven't been testing in CI -- we _really_
> > need to set up nightly CI jobs for more time consuming tests, like
> > HDFS, that we want to test periodically but perhaps not on every
> > commit). I'm going to see if these are easy to fix
> >
> > On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
> >> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
> >> someone could review that would be great.
> >>
> >> I am going to make sure the C++/Python HDFS tests run locally, then
> >> after the patch above is merged we should be good to make the RC.
> >>
> >> Note to other onlookers -- please feel free to keep writing new
> >> patches not mentioned here. They may just have to go in to the next
> >> release
> >>
> >> Thanks
> >> Wes
> >>
> >> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
> >>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
> >>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
> >>> try to fix tomorrow before cutting the RC. I can start the vote after
> >>> that if nothing else comes up
> >>>
> >>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
> >>> have an alternative way to run the HDFS tests (since we aren't running
> >>> them in Travis CI) so most likely I will try out the HDFS tests and
> >>> then move ARROW-2300 to the next release milestone.
> >>>
> >>> Thanks
> >>> Wes
> >>>
> >>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
> >>>> I've done a pass over the remaining JIRAs -- I think we're going to
> >>>> need at least another full day to get things buttoned up, so I would
> >>>> say we're looking at an RC on Tuesday.
> >>>>
> >>>> In progress:
> >>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
> >>>> - ARROW-2282: Patch up, needs some test cases
> >>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
> >>>> move this JIRA to the Parquet project
> >>>> - ARROW-2122: Patch to be reviewed
> >>>> - ARROW-2135: Patch to be merged
> >>>>
> >>>> TODO
> >>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
> >>>> - ARROW-2118: Fix rough edge with reading length-0 files
> >>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
> >>>> - ARROW-2292: Deprecation / renaming a Python method
> >>>>
> >>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
> >>>> matters, so I will take care of them. ARROW-2227 may not be fixable
> >>>> within ~1 day but it would be useful to have a diagnosis in case the
> >>>> fix is easy -- multiple users hit this bug.
> >>>>
> >>>> Thanks,
> >>>> Wes
> >>>>
> >>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
> >>>>> Thanks!
> >>>>>
> >>>>> --
> >>>>> kou
> >>>>>
> >>>>> In <CA...@mail.gmail.com>
> >>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
> >>>>>   Wes McKinney <we...@gmail.com> wrote:
> >>>>>
> >>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
> >>>>>> bit of work to be able to produce a viable standalone source tarball.
> >>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
> >>>>>> with a plan to do this
> >>>>>>
> >>>>>> - Wes
> >>>>>>
> >>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>>    - Updating pip packages for C++ and Python
> >>>>>>>
> >>>>>>> Can we try adding PyArrow source package to PyPI at the
> >>>>>>> 0.9.0?
> >>>>>>>
> >>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
> >>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
> >>>>>>> PyArrow in the same process via PyCall (Ruby library
> >>>>>>> to integrate with Python). In the case, I need to use the
> >>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
> >>>>>>>
> >>>>>>> Now, there are only binary packages for PyArrow at
> >>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
> >>>>>>> package for PyArrow at PyPI, I can install PyArrow with
> >>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
> >>>>>>> pyarrow".
> >>>>>>>
> >>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> --
> >>>>>>> kou
> >>>>>>>
> >>>>>>> In <CA...@mail.gmail.com>
> >>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
> >>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
> >>>>>>>
> >>>>>>>> All,
> >>>>>>>>
> >>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
> >>>>>>>> everybody?
> >>>>>>>>
> >>>>>>>> We have 6 items in progress --
> >>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
> >>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
> >>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
> >>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
> >>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
> >>>>>>>>
> >>>>>>>> I was wondering if anyone else is interested in collaborating for the
> >>>>>>>> post-release tasks. As per
> >>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
> >>>>>>>> following are the high level post-release tasks. Please let me know if you
> >>>>>>>> would like to take up something. I have written my name against some of
> >>>>>>>> them.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>    - Updating the Arrow Website (Sidd)
> >>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
> >>>>>>>>    to do this
> >>>>>>>>    - Announcing release (Sidd)
> >>>>>>>>    - Updating website with new API documentation (Sidd)
> >>>>>>>>    - Updating pip packages for C++ and Python
> >>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
> >>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
> >>>>>>>>    - Release blog post
> >>>>>>>>
> >>>>>>>> If anything is missing, please add to the above list. It will be helpful
> >>>>>>>> for tracking.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Sidd
> >>>>>>>>
> >>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> hey Sidd,
> >>>>>>>>>
> >>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
> >>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
> >>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
> >>>>>>>>> of the current backlog to get things down to the essential
> >>>>>>>>>
> >>>>>>>>> - Wes
> >>>>>>>>>
> >>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
> >>>>>>>>> wrote:
> >>>>>>>>> > Sounds good.
> >>>>>>>>> >
> >>>>>>>>> > Thanks
> >>>>>>>>> > Sidd
> >>>>>>>>> >
> >>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
> >>>>>>>>> >
> >>>>>>>>> > Hi Sidd,
> >>>>>>>>> >
> >>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
> >>>>>>>>> we
> >>>>>>>>> > defer to the following week.
> >>>>>>>>> >
> >>>>>>>>> > Thanks
> >>>>>>>>> > Wes
> >>>>>>>>> >
> >>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
> >>>>>>>>> wrote:
> >>>>>>>>> >
> >>>>>>>>> > Hi All,
> >>>>>>>>> >
> >>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
> >>>>>>>>> > them. In interest of getting a release candidate next week, I would
> >>>>>>>>> request
> >>>>>>>>> > people to review PRs as soon as they can to help make progress and close
> >>>>>>>>> > out as many JIRAs as we can.
> >>>>>>>>> >
> >>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
> >>>>>>>>> am
> >>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
> >>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
> >>>>>>>>> > for 0.9.0.
> >>>>>>>>> >
> >>>>>>>>> > Here is the link to backlog:
> >>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
> >>>>>>>>> >
> >>>>>>>>> > Thanks,
> >>>>>>>>> > Sidd
> >>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I fixed these bugs

https://github.com/apache/arrow/pull/1742
https://github.com/apache/arrow/pull/1743

As soon as these patches go in, we can cut the RC0. I can do this as
soon as tomorrow (Wednesday) morning

On Tue, Mar 13, 2018 at 5:00 PM, Wes McKinney <we...@gmail.com> wrote:
> I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
> testing of master (stuff we haven't been testing in CI -- we _really_
> need to set up nightly CI jobs for more time consuming tests, like
> HDFS, that we want to test periodically but perhaps not on every
> commit). I'm going to see if these are easy to fix
>
> On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
>> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
>> someone could review that would be great.
>>
>> I am going to make sure the C++/Python HDFS tests run locally, then
>> after the patch above is merged we should be good to make the RC.
>>
>> Note to other onlookers -- please feel free to keep writing new
>> patches not mentioned here. They may just have to go in to the next
>> release
>>
>> Thanks
>> Wes
>>
>> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>>> try to fix tomorrow before cutting the RC. I can start the vote after
>>> that if nothing else comes up
>>>
>>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>>> have an alternative way to run the HDFS tests (since we aren't running
>>> them in Travis CI) so most likely I will try out the HDFS tests and
>>> then move ARROW-2300 to the next release milestone.
>>>
>>> Thanks
>>> Wes
>>>
>>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>>> need at least another full day to get things buttoned up, so I would
>>>> say we're looking at an RC on Tuesday.
>>>>
>>>> In progress:
>>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>>> - ARROW-2282: Patch up, needs some test cases
>>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>>> move this JIRA to the Parquet project
>>>> - ARROW-2122: Patch to be reviewed
>>>> - ARROW-2135: Patch to be merged
>>>>
>>>> TODO
>>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>>> - ARROW-2292: Deprecation / renaming a Python method
>>>>
>>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>>> within ~1 day but it would be useful to have a diagnosis in case the
>>>> fix is easy -- multiple users hit this bug.
>>>>
>>>> Thanks,
>>>> Wes
>>>>
>>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> kou
>>>>>
>>>>> In <CA...@mail.gmail.com>
>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>>   Wes McKinney <we...@gmail.com> wrote:
>>>>>
>>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>>> with a plan to do this
>>>>>>
>>>>>> - Wes
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>>    - Updating pip packages for C++ and Python
>>>>>>>
>>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>>> 0.9.0?
>>>>>>>
>>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>>> to integrate with Python). In the case, I need to use the
>>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>>
>>>>>>> Now, there are only binary packages for PyArrow at
>>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>>> pyarrow".
>>>>>>>
>>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --
>>>>>>> kou
>>>>>>>
>>>>>>> In <CA...@mail.gmail.com>
>>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>>> everybody?
>>>>>>>>
>>>>>>>> We have 6 items in progress --
>>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>>
>>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>>> post-release tasks. As per
>>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>>> would like to take up something. I have written my name against some of
>>>>>>>> them.
>>>>>>>>
>>>>>>>>
>>>>>>>>    - Updating the Arrow Website (Sidd)
>>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>>    to do this
>>>>>>>>    - Announcing release (Sidd)
>>>>>>>>    - Updating website with new API documentation (Sidd)
>>>>>>>>    - Updating pip packages for C++ and Python
>>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>>    - Release blog post
>>>>>>>>
>>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>>> for tracking.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sidd
>>>>>>>>
>>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> hey Sidd,
>>>>>>>>>
>>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>>
>>>>>>>>> - Wes
>>>>>>>>>
>>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>>> wrote:
>>>>>>>>> > Sounds good.
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>> > Sidd
>>>>>>>>> >
>>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi Sidd,
>>>>>>>>> >
>>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>>> we
>>>>>>>>> > defer to the following week.
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>> > Wes
>>>>>>>>> >
>>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi All,
>>>>>>>>> >
>>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>>> > them. In interest of getting a release candidate next week, I would
>>>>>>>>> request
>>>>>>>>> > people to review PRs as soon as they can to help make progress and close
>>>>>>>>> > out as many JIRAs as we can.
>>>>>>>>> >
>>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>>> am
>>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>>> > for 0.9.0.
>>>>>>>>> >
>>>>>>>>> > Here is the link to backlog:
>>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>>> >
>>>>>>>>> > Thanks,
>>>>>>>>> > Sidd
>>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I found 2 bugs -- ARROW-2304 and ARROW-2306 -- while doing some final
testing of master (stuff we haven't been testing in CI -- we _really_
need to set up nightly CI jobs for more time consuming tests, like
HDFS, that we want to test periodically but perhaps not on every
commit). I'm going to see if these are easy to fix

On Tue, Mar 13, 2018 at 11:21 AM, Wes McKinney <we...@gmail.com> wrote:
> I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
> someone could review that would be great.
>
> I am going to make sure the C++/Python HDFS tests run locally, then
> after the patch above is merged we should be good to make the RC.
>
> Note to other onlookers -- please feel free to keep writing new
> patches not mentioned here. They may just have to go in to the next
> release
>
> Thanks
> Wes
>
> On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
>> Things are looking pretty good. I'm waiting on a build for ARROW-1643
>> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
>> try to fix tomorrow before cutting the RC. I can start the vote after
>> that if nothing else comes up
>>
>> I created ARROW-2300 in the course of trying to test ARROW-1643. I
>> have an alternative way to run the HDFS tests (since we aren't running
>> them in Travis CI) so most likely I will try out the HDFS tests and
>> then move ARROW-2300 to the next release milestone.
>>
>> Thanks
>> Wes
>>
>> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>>> I've done a pass over the remaining JIRAs -- I think we're going to
>>> need at least another full day to get things buttoned up, so I would
>>> say we're looking at an RC on Tuesday.
>>>
>>> In progress:
>>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>>> - ARROW-2282: Patch up, needs some test cases
>>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>>> move this JIRA to the Parquet project
>>> - ARROW-2122: Patch to be reviewed
>>> - ARROW-2135: Patch to be merged
>>>
>>> TODO
>>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>>> - ARROW-2118: Fix rough edge with reading length-0 files
>>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>>> - ARROW-2292: Deprecation / renaming a Python method
>>>
>>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>>> matters, so I will take care of them. ARROW-2227 may not be fixable
>>> within ~1 day but it would be useful to have a diagnosis in case the
>>> fix is easy -- multiple users hit this bug.
>>>
>>> Thanks,
>>> Wes
>>>
>>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>> Thanks!
>>>>
>>>> --
>>>> kou
>>>>
>>>> In <CA...@mail.gmail.com>
>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>>   Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>>> bit of work to be able to produce a viable standalone source tarball.
>>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>>> with a plan to do this
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>>>    - Updating pip packages for C++ and Python
>>>>>>
>>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>>> 0.9.0?
>>>>>>
>>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>>> to integrate with Python). In the case, I need to use the
>>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>>
>>>>>> Now, there are only binary packages for PyArrow at
>>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>>> pyarrow".
>>>>>>
>>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>> kou
>>>>>>
>>>>>> In <CA...@mail.gmail.com>
>>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>>> everybody?
>>>>>>>
>>>>>>> We have 6 items in progress --
>>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>>
>>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>>> post-release tasks. As per
>>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>>> would like to take up something. I have written my name against some of
>>>>>>> them.
>>>>>>>
>>>>>>>
>>>>>>>    - Updating the Arrow Website (Sidd)
>>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>>    to do this
>>>>>>>    - Announcing release (Sidd)
>>>>>>>    - Updating website with new API documentation (Sidd)
>>>>>>>    - Updating pip packages for C++ and Python
>>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>>    - Release blog post
>>>>>>>
>>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>>> for tracking.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sidd
>>>>>>>
>>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>>
>>>>>>>> hey Sidd,
>>>>>>>>
>>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>>> of the current backlog to get things down to the essential
>>>>>>>>
>>>>>>>> - Wes
>>>>>>>>
>>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>>> wrote:
>>>>>>>> > Sounds good.
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>> > Sidd
>>>>>>>> >
>>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > Hi Sidd,
>>>>>>>> >
>>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>>> we
>>>>>>>> > defer to the following week.
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>> > Wes
>>>>>>>> >
>>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Hi All,
>>>>>>>> >
>>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>>> > them. In interest of getting a release candidate next week, I would
>>>>>>>> request
>>>>>>>> > people to review PRs as soon as they can to help make progress and close
>>>>>>>> > out as many JIRAs as we can.
>>>>>>>> >
>>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>>> am
>>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>>>>>> > for 0.9.0.
>>>>>>>> >
>>>>>>>> > Here is the link to backlog:
>>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>>> >
>>>>>>>> > Thanks,
>>>>>>>> > Sidd
>>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I fixed ARROW-2227 in https://github.com/apache/arrow/pull/1740, so if
someone could review that would be great.

I am going to make sure the C++/Python HDFS tests run locally, then
after the patch above is merged we should be good to make the RC.

Note to other onlookers -- please feel free to keep writing new
patches not mentioned here. They may just have to go in to the next
release

Thanks
Wes

On Tue, Mar 13, 2018 at 12:56 AM, Wes McKinney <we...@gmail.com> wrote:
> Things are looking pretty good. I'm waiting on a build for ARROW-1643
> to go in, and there's a last blocker bug (ARROW-2227) that we ought to
> try to fix tomorrow before cutting the RC. I can start the vote after
> that if nothing else comes up
>
> I created ARROW-2300 in the course of trying to test ARROW-1643. I
> have an alternative way to run the HDFS tests (since we aren't running
> them in Travis CI) so most likely I will try out the HDFS tests and
> then move ARROW-2300 to the next release milestone.
>
> Thanks
> Wes
>
> On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
>> I've done a pass over the remaining JIRAs -- I think we're going to
>> need at least another full day to get things buttoned up, so I would
>> say we're looking at an RC on Tuesday.
>>
>> In progress:
>> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
>> - ARROW-2282: Patch up, needs some test cases
>> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
>> move this JIRA to the Parquet project
>> - ARROW-2122: Patch to be reviewed
>> - ARROW-2135: Patch to be merged
>>
>> TODO
>> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
>> - ARROW-2118: Fix rough edge with reading length-0 files
>> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
>> - ARROW-2292: Deprecation / renaming a Python method
>>
>> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
>> matters, so I will take care of them. ARROW-2227 may not be fixable
>> within ~1 day but it would be useful to have a diagnosis in case the
>> fix is easy -- multiple users hit this bug.
>>
>> Thanks,
>> Wes
>>
>> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>> Thanks!
>>>
>>> --
>>> kou
>>>
>>> In <CA...@mail.gmail.com>
>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>>   Wes McKinney <we...@gmail.com> wrote:
>>>
>>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>>> bit of work to be able to produce a viable standalone source tarball.
>>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>>> with a plan to do this
>>>>
>>>> - Wes
>>>>
>>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>>> Hi,
>>>>>
>>>>>>    - Updating pip packages for C++ and Python
>>>>>
>>>>> Can we try adding PyArrow source package to PyPI at the
>>>>> 0.9.0?
>>>>>
>>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>>> PyArrow in the same process via PyCall (Ruby library
>>>>> to integrate with Python). In the case, I need to use the
>>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>>
>>>>> Now, there are only binary packages for PyArrow at
>>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>>> pyarrow".
>>>>>
>>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> kou
>>>>>
>>>>> In <CA...@mail.gmail.com>
>>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>>>
>>>>>> All,
>>>>>>
>>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>>> everybody?
>>>>>>
>>>>>> We have 6 items in progress --
>>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>>
>>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>>> post-release tasks. As per
>>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>>> following are the high level post-release tasks. Please let me know if you
>>>>>> would like to take up something. I have written my name against some of
>>>>>> them.
>>>>>>
>>>>>>
>>>>>>    - Updating the Arrow Website (Sidd)
>>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>>    to do this
>>>>>>    - Announcing release (Sidd)
>>>>>>    - Updating website with new API documentation (Sidd)
>>>>>>    - Updating pip packages for C++ and Python
>>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>>    - Release blog post
>>>>>>
>>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>>> for tracking.
>>>>>>
>>>>>> Thanks,
>>>>>> Sidd
>>>>>>
>>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>
>>>>>>> hey Sidd,
>>>>>>>
>>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>>> of the current backlog to get things down to the essential
>>>>>>>
>>>>>>> - Wes
>>>>>>>
>>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>>> wrote:
>>>>>>> > Sounds good.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Sidd
>>>>>>> >
>>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hi Sidd,
>>>>>>> >
>>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>>>>> we
>>>>>>> > defer to the following week.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Wes
>>>>>>> >
>>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi All,
>>>>>>> >
>>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>>>>> > them. In interest of getting a release candidate next week, I would
>>>>>>> request
>>>>>>> > people to review PRs as soon as they can to help make progress and close
>>>>>>> > out as many JIRAs as we can.
>>>>>>> >
>>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>>> am
>>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>>>>> > for 0.9.0.
>>>>>>> >
>>>>>>> > Here is the link to backlog:
>>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > Sidd
>>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Things are looking pretty good. I'm waiting on a build for ARROW-1643
to go in, and there's a last blocker bug (ARROW-2227) that we ought to
try to fix tomorrow before cutting the RC. I can start the vote after
that if nothing else comes up

I created ARROW-2300 in the course of trying to test ARROW-1643. I
have an alternative way to run the HDFS tests (since we aren't running
them in Travis CI) so most likely I will try out the HDFS tests and
then move ARROW-2300 to the next release milestone.

Thanks
Wes

On Mon, Mar 12, 2018 at 12:03 AM, Wes McKinney <we...@gmail.com> wrote:
> I've done a pass over the remaining JIRAs -- I think we're going to
> need at least another full day to get things buttoned up, so I would
> say we're looking at an RC on Tuesday.
>
> In progress:
> - ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
> - ARROW-2282: Patch up, needs some test cases
> - ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
> move this JIRA to the Parquet project
> - ARROW-2122: Patch to be reviewed
> - ARROW-2135: Patch to be merged
>
> TODO
> - ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
> - ARROW-2118: Fix rough edge with reading length-0 files
> - ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
> - ARROW-2292: Deprecation / renaming a Python method
>
> Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
> matters, so I will take care of them. ARROW-2227 may not be fixable
> within ~1 day but it would be useful to have a diagnosis in case the
> fix is easy -- multiple users hit this bug.
>
> Thanks,
> Wes
>
> On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>> Thanks!
>>
>> --
>> kou
>>
>> In <CA...@mail.gmail.com>
>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>>   Wes McKinney <we...@gmail.com> wrote:
>>
>>> hi Kou -- yes, I think this is a good idea. It will require a little
>>> bit of work to be able to produce a viable standalone source tarball.
>>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>>> with a plan to do this
>>>
>>> - Wes
>>>
>>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>>> Hi,
>>>>
>>>>>    - Updating pip packages for C++ and Python
>>>>
>>>> Can we try adding PyArrow source package to PyPI at the
>>>> 0.9.0?
>>>>
>>>> I want to install PyArrow with Arrow C++ installed by .deb
>>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>>> PyArrow in the same process via PyCall (Ruby library
>>>> to integrate with Python). In the case, I need to use the
>>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>>
>>>> Now, there are only binary packages for PyArrow at
>>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>>> package for PyArrow at PyPI, I can install PyArrow with
>>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>>> pyarrow".
>>>>
>>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>>
>>>>
>>>> Thanks,
>>>> --
>>>> kou
>>>>
>>>> In <CA...@mail.gmail.com>
>>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>>
>>>>> All,
>>>>>
>>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>>> everybody?
>>>>>
>>>>> We have 6 items in progress --
>>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>>
>>>>> I was wondering if anyone else is interested in collaborating for the
>>>>> post-release tasks. As per
>>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>>> following are the high level post-release tasks. Please let me know if you
>>>>> would like to take up something. I have written my name against some of
>>>>> them.
>>>>>
>>>>>
>>>>>    - Updating the Arrow Website (Sidd)
>>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>>    to do this
>>>>>    - Announcing release (Sidd)
>>>>>    - Updating website with new API documentation (Sidd)
>>>>>    - Updating pip packages for C++ and Python
>>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>>>    - Release blog post
>>>>>
>>>>> If anything is missing, please add to the above list. It will be helpful
>>>>> for tracking.
>>>>>
>>>>> Thanks,
>>>>> Sidd
>>>>>
>>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>
>>>>>> hey Sidd,
>>>>>>
>>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>>> of the current backlog to get things down to the essential
>>>>>>
>>>>>> - Wes
>>>>>>
>>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>>> wrote:
>>>>>> > Sounds good.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Sidd
>>>>>> >
>>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi Sidd,
>>>>>> >
>>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>>>> we
>>>>>> > defer to the following week.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Wes
>>>>>> >
>>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hi All,
>>>>>> >
>>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>>>> > them. In interest of getting a release candidate next week, I would
>>>>>> request
>>>>>> > people to review PRs as soon as they can to help make progress and close
>>>>>> > out as many JIRAs as we can.
>>>>>> >
>>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>>> am
>>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>>>> > for 0.9.0.
>>>>>> >
>>>>>> > Here is the link to backlog:
>>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>>> >
>>>>>> > Thanks,
>>>>>> > Sidd
>>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

I've done a pass over the remaining JIRAs -- I think we're going to
need at least another full day to get things buttoned up, so I would
say we're looking at an RC on Tuesday.

In progress:
- ARROW-1425: Patch up, needs some editing, may be moved to 0.10.0
- ARROW-2282: Patch up, needs some test cases
- ARROW-1974: Patch in review in apache/parquet-cpp -- may want to
move this JIRA to the Parquet project
- ARROW-2122: Patch to be reviewed
- ARROW-2135: Patch to be merged

TODO
- ARROW-2082: Parquet segfault <- to be investigated, may be moved to 0.10.0
- ARROW-2118: Fix rough edge with reading length-0 files
- ARROW-2227: Bug with creating chunked arrays in Table.from_pandas
- ARROW-2292: Deprecation / renaming a Python method

Out of the two items in TODO, ARROW-2118 and ARROW-2292 are small
matters, so I will take care of them. ARROW-2227 may not be fixable
within ~1 day but it would be useful to have a diagnosis in case the
fix is easy -- multiple users hit this bug.

Thanks,
Wes

On Thu, Mar 8, 2018 at 8:47 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
> Thanks!
>
> --
> kou
>
> In <CA...@mail.gmail.com>
>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
>   Wes McKinney <we...@gmail.com> wrote:
>
>> hi Kou -- yes, I think this is a good idea. It will require a little
>> bit of work to be able to produce a viable standalone source tarball.
>> Between Uwe, Phillip, Antoine, and I, we should be able to come up
>> with a plan to do this
>>
>> - Wes
>>
>> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>>> Hi,
>>>
>>>>    - Updating pip packages for C++ and Python
>>>
>>> Can we try adding PyArrow source package to PyPI at the
>>> 0.9.0?
>>>
>>> I want to install PyArrow with Arrow C++ installed by .deb
>>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>>> PyArrow in the same process via PyCall (Ruby library
>>> to integrate with Python). In the case, I need to use the
>>> same Arrow C++ in both Red Arrow and PyArrow.
>>>
>>> Now, there are only binary packages for PyArrow at
>>> https://pypi.python.org/pypi/pyarrow . If there is a source
>>> package for PyArrow at PyPI, I can install PyArrow with
>>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>>> pyarrow".
>>>
>>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <CA...@mail.gmail.com>
>>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>>   Siddharth Teotia <si...@dremio.com> wrote:
>>>
>>>> All,
>>>>
>>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>>> everybody?
>>>>
>>>> We have 6 items in progress --
>>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>>> How do people feel about completing these JIRAs by tomorrow? I am
>>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>>
>>>> I was wondering if anyone else is interested in collaborating for the
>>>> post-release tasks. As per
>>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>>> following are the high level post-release tasks. Please let me know if you
>>>> would like to take up something. I have written my name against some of
>>>> them.
>>>>
>>>>
>>>>    - Updating the Arrow Website (Sidd)
>>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>>    to do this
>>>>    - Announcing release (Sidd)
>>>>    - Updating website with new API documentation (Sidd)
>>>>    - Updating pip packages for C++ and Python
>>>>    - Updating conda packages for C++ and Python (Sidd)
>>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>>    - Release blog post
>>>>
>>>> If anything is missing, please add to the above list. It will be helpful
>>>> for tracking.
>>>>
>>>> Thanks,
>>>> Sidd
>>>>
>>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>>> hey Sidd,
>>>>>
>>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>>> of the current backlog to get things down to the essential
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>>> wrote:
>>>>> > Sounds good.
>>>>> >
>>>>> > Thanks
>>>>> > Sidd
>>>>> >
>>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>>> >
>>>>> > Hi Sidd,
>>>>> >
>>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>>> we
>>>>> > defer to the following week.
>>>>> >
>>>>> > Thanks
>>>>> > Wes
>>>>> >
>>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi All,
>>>>> >
>>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>>> > them. In interest of getting a release candidate next week, I would
>>>>> request
>>>>> > people to review PRs as soon as they can to help make progress and close
>>>>> > out as many JIRAs as we can.
>>>>> >
>>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>>> am
>>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>>> > for 0.9.0.
>>>>> >
>>>>> > Here is the link to backlog:
>>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>>> >
>>>>> > Thanks,
>>>>> > Sidd
>>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Kouhei Sutou <ko...@clear-code.com>.

Thanks!

--
kou

In <CA...@mail.gmail.com>
  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 20:44:14 -0500,
  Wes McKinney <we...@gmail.com> wrote:

> hi Kou -- yes, I think this is a good idea. It will require a little
> bit of work to be able to produce a viable standalone source tarball.
> Between Uwe, Phillip, Antoine, and I, we should be able to come up
> with a plan to do this
> 
> - Wes
> 
> On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
>> Hi,
>>
>>>    - Updating pip packages for C++ and Python
>>
>> Can we try adding PyArrow source package to PyPI at the
>> 0.9.0?
>>
>> I want to install PyArrow with Arrow C++ installed by .deb
>> or .rpm. I want to use both Red Arrow (Ruby bindings) and
>> PyArrow in the same process via PyCall (Ruby library
>> to integrate with Python). In the case, I need to use the
>> same Arrow C++ in both Red Arrow and PyArrow.
>>
>> Now, there are only binary packages for PyArrow at
>> https://pypi.python.org/pypi/pyarrow . If there is a source
>> package for PyArrow at PyPI, I can install PyArrow with
>> Arrow C++ installed by .deb or .rpm by "pip --no-binary
>> pyarrow".
>>
>> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In <CA...@mail.gmail.com>
>>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>>   Siddharth Teotia <si...@dremio.com> wrote:
>>
>>> All,
>>>
>>> I plan to get RC out over the weekend or early Monday. Is that fine with
>>> everybody?
>>>
>>> We have 6 items in progress --
>>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>>> How do people feel about completing these JIRAs by tomorrow? I am
>>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>>> necessary. Just looking for consensus. Also, I suggest that we defer the
>>> ones with TODO status. I will do it later today unless I hear otherwise.
>>>
>>> I was wondering if anyone else is interested in collaborating for the
>>> post-release tasks. As per
>>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>>> following are the high level post-release tasks. Please let me know if you
>>> would like to take up something. I have written my name against some of
>>> them.
>>>
>>>
>>>    - Updating the Arrow Website (Sidd)
>>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>>    to do this
>>>    - Announcing release (Sidd)
>>>    - Updating website with new API documentation (Sidd)
>>>    - Updating pip packages for C++ and Python
>>>    - Updating conda packages for C++ and Python (Sidd)
>>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>>    - Release blog post
>>>
>>> If anything is missing, please add to the above list. It will be helpful
>>> for tracking.
>>>
>>> Thanks,
>>> Sidd
>>>
>>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>>
>>>> hey Sidd,
>>>>
>>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>>> the following week (3/12 or 13). I will trim as much as possible out
>>>> of the current backlog to get things down to the essential
>>>>
>>>> - Wes
>>>>
>>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>>> wrote:
>>>> > Sounds good.
>>>> >
>>>> > Thanks
>>>> > Sidd
>>>> >
>>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>>> >
>>>> > Hi Sidd,
>>>> >
>>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>>> we
>>>> > defer to the following week.
>>>> >
>>>> > Thanks
>>>> > Wes
>>>> >
>>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>>> wrote:
>>>> >
>>>> > Hi All,
>>>> >
>>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>>> > them. In interest of getting a release candidate next week, I would
>>>> request
>>>> > people to review PRs as soon as they can to help make progress and close
>>>> > out as many JIRAs as we can.
>>>> >
>>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>>> am
>>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>>> > would be good to soon grab/assign the issues that people want to be fixed
>>>> > for 0.9.0.
>>>> >
>>>> > Here is the link to backlog:
>>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>>> >
>>>> > Thanks,
>>>> > Sidd
>>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

hi Kou -- yes, I think this is a good idea. It will require a little
bit of work to be able to produce a viable standalone source tarball.
Between Uwe, Phillip, Antoine, and I, we should be able to come up
with a plan to do this

- Wes

On Thu, Mar 8, 2018 at 8:33 PM, Kouhei Sutou <ko...@clear-code.com> wrote:
> Hi,
>
>>    - Updating pip packages for C++ and Python
>
> Can we try adding PyArrow source package to PyPI at the
> 0.9.0?
>
> I want to install PyArrow with Arrow C++ installed by .deb
> or .rpm. I want to use both Red Arrow (Ruby bindings) and
> PyArrow in the same process via PyCall (Ruby library
> to integrate with Python). In the case, I need to use the
> same Arrow C++ in both Red Arrow and PyArrow.
>
> Now, there are only binary packages for PyArrow at
> https://pypi.python.org/pypi/pyarrow . If there is a source
> package for PyArrow at PyPI, I can install PyArrow with
> Arrow C++ installed by .deb or .rpm by "pip --no-binary
> pyarrow".
>
> Red Arrow can also use Arrow C++ installed by .deb or .rpm.
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
>   "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
>   Siddharth Teotia <si...@dremio.com> wrote:
>
>> All,
>>
>> I plan to get RC out over the weekend or early Monday. Is that fine with
>> everybody?
>>
>> We have 6 items in progress --
>> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
>> How do people feel about completing these JIRAs by tomorrow? I am
>> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
>> necessary. Just looking for consensus. Also, I suggest that we defer the
>> ones with TODO status. I will do it later today unless I hear otherwise.
>>
>> I was wondering if anyone else is interested in collaborating for the
>> post-release tasks. As per
>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
>> following are the high level post-release tasks. Please let me know if you
>> would like to take up something. I have written my name against some of
>> them.
>>
>>
>>    - Updating the Arrow Website (Sidd)
>>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>>    to do this
>>    - Announcing release (Sidd)
>>    - Updating website with new API documentation (Sidd)
>>    - Updating pip packages for C++ and Python
>>    - Updating conda packages for C++ and Python (Sidd)
>>    - Updating Java Maven artifacts in Maven central (Sidd)
>>    - Release blog post
>>
>> If anything is missing, please add to the above list. It will be helpful
>> for tracking.
>>
>> Thanks,
>> Sidd
>>
>> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>>
>>> hey Sidd,
>>>
>>> The Python backlog is still in pretty rough shape. I'd like to see if
>>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>>> the following week (3/12 or 13). I will trim as much as possible out
>>> of the current backlog to get things down to the essential
>>>
>>> - Wes
>>>
>>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>>> wrote:
>>> > Sounds good.
>>> >
>>> > Thanks
>>> > Sidd
>>> >
>>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>>> >
>>> > Hi Sidd,
>>> >
>>> > I think we have too many bugs to make an RC this coming week. I suggest
>>> we
>>> > defer to the following week.
>>> >
>>> > Thanks
>>> > Wes
>>> >
>>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>>> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > We currently have 10 issues in progress and PRs are available for 8 of
>>> > them. In interest of getting a release candidate next week, I would
>>> request
>>> > people to review PRs as soon as they can to help make progress and close
>>> > out as many JIRAs as we can.
>>> >
>>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>>> am
>>> > planning to defer some of the unassigned ones later today or tomorrow. It
>>> > would be good to soon grab/assign the issues that people want to be fixed
>>> > for 0.9.0.
>>> >
>>> > Here is the link to backlog:
>>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>>> >
>>> > Thanks,
>>> > Sidd
>>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Kouhei Sutou <ko...@clear-code.com>.

Hi,

>    - Updating pip packages for C++ and Python

Can we try adding PyArrow source package to PyPI at the
0.9.0?

I want to install PyArrow with Arrow C++ installed by .deb
or .rpm. I want to use both Red Arrow (Ruby bindings) and
PyArrow in the same process via PyCall (Ruby library
to integrate with Python). In the case, I need to use the
same Arrow C++ in both Red Arrow and PyArrow.

Now, there are only binary packages for PyArrow at
https://pypi.python.org/pypi/pyarrow . If there is a source
package for PyArrow at PyPI, I can install PyArrow with
Arrow C++ installed by .deb or .rpm by "pip --no-binary
pyarrow".

Red Arrow can also use Arrow C++ installed by .deb or .rpm.


Thanks,
--
kou

In <CA...@mail.gmail.com>
  "Re: Working towards getting 0.9.0 release candidate up next week" on Thu, 8 Mar 2018 11:25:32 -0800,
  Siddharth Teotia <si...@dremio.com> wrote:

> All,
> 
> I plan to get RC out over the weekend or early Monday. Is that fine with
> everybody?
> 
> We have 6 items in progress --
> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
> How do people feel about completing these JIRAs by tomorrow? I am
> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
> necessary. Just looking for consensus. Also, I suggest that we defer the
> ones with TODO status. I will do it later today unless I hear otherwise.
> 
> I was wondering if anyone else is interested in collaborating for the
> post-release tasks. As per
> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
> following are the high level post-release tasks. Please let me know if you
> would like to take up something. I have written my name against some of
> them.
> 
> 
>    - Updating the Arrow Website (Sidd)
>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>    to do this
>    - Announcing release (Sidd)
>    - Updating website with new API documentation (Sidd)
>    - Updating pip packages for C++ and Python
>    - Updating conda packages for C++ and Python (Sidd)
>    - Updating Java Maven artifacts in Maven central (Sidd)
>    - Release blog post
> 
> If anything is missing, please add to the above list. It will be helpful
> for tracking.
> 
> Thanks,
> Sidd
> 
> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
> 
>> hey Sidd,
>>
>> The Python backlog is still in pretty rough shape. I'd like to see if
>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>> the following week (3/12 or 13). I will trim as much as possible out
>> of the current backlog to get things down to the essential
>>
>> - Wes
>>
>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>> wrote:
>> > Sounds good.
>> >
>> > Thanks
>> > Sidd
>> >
>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>> >
>> > Hi Sidd,
>> >
>> > I think we have too many bugs to make an RC this coming week. I suggest
>> we
>> > defer to the following week.
>> >
>> > Thanks
>> > Wes
>> >
>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > We currently have 10 issues in progress and PRs are available for 8 of
>> > them. In interest of getting a release candidate next week, I would
>> request
>> > people to review PRs as soon as they can to help make progress and close
>> > out as many JIRAs as we can.
>> >
>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>> am
>> > planning to defer some of the unassigned ones later today or tomorrow. It
>> > would be good to soon grab/assign the issues that people want to be fixed
>> > for 0.9.0.
>> >
>> > Here is the link to backlog:
>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>> >
>> > Thanks,
>> > Sidd
>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Siddharth Teotia <si...@dremio.com>.

Thanks, Wes. Let's shoot for Monday.

On Thu, Mar 8, 2018 at 11:31 AM, Wes McKinney <we...@gmail.com> wrote:

> Since almost all of the items in TODO are C++ or Python issues, I can
> do a final review today to remove anything that isn't absolutely
> necessary for 0.9.0. We have a couple of nasty bugs still in TODO that
> we should try to fix -- in the event that they cannot be fixed, we may
> need to do a 0.9.1 in a week or two. I would suggest we wait to cut
> the RC until Monday to give enough time for these last items to get
> fixes in.
>
> There are some other things that need doing, like updates per changes
> to the ASF checksum policy ARROW-2268.
>
> I can write by EOD today with a status report on the issues in TODO.
>
> I believe you need to be a PMC to undertake the source release process
> prior to the vote -- I am happy to help with this on Monday.
>
> - Wes
>
> On Thu, Mar 8, 2018 at 2:25 PM, Siddharth Teotia <si...@dremio.com>
> wrote:
> > All,
> >
> > I plan to get RC out over the weekend or early Monday. Is that fine with
> > everybody?
> >
> > We have 6 items in progress --
> > https://issues.apache.org/jira/projects/ARROW/versions/
> 12341707#release-report-tab-body.
> > How do people feel about completing these JIRAs by tomorrow? I am
> > completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
> > necessary. Just looking for consensus. Also, I suggest that we defer the
> > ones with TODO status. I will do it later today unless I hear otherwise.
> >
> > I was wondering if anyone else is interested in collaborating for the
> > post-release tasks. As per
> > https://github.com/apache/arrow/blob/master/dev/release/
> RELEASE_MANAGEMENT.md,
> > following are the high level post-release tasks. Please let me know if
> you
> > would like to take up something. I have written my name against some of
> > them.
> >
> >
> >    - Updating the Arrow Website (Sidd)
> >    - Uploading release artifacts to SVN -- looks like PMC karma is needed
> >    to do this
> >    - Announcing release (Sidd)
> >    - Updating website with new API documentation (Sidd)
> >    - Updating pip packages for C++ and Python
> >    - Updating conda packages for C++ and Python (Sidd)
> >    - Updating Java Maven artifacts in Maven central (Sidd)
> >    - Release blog post
> >
> > If anything is missing, please add to the above list. It will be helpful
> > for tracking.
> >
> > Thanks,
> > Sidd
> >
> > On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com>
> wrote:
> >
> >> hey Sidd,
> >>
> >> The Python backlog is still in pretty rough shape. I'd like to see if
> >> we can make an RC by Friday but if not we can defer to Monday/Tuesday
> >> the following week (3/12 or 13). I will trim as much as possible out
> >> of the current backlog to get things down to the essential
> >>
> >> - Wes
> >>
> >> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <
> siddharth@dremio.com>
> >> wrote:
> >> > Sounds good.
> >> >
> >> > Thanks
> >> > Sidd
> >> >
> >> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
> >> >
> >> > Hi Sidd,
> >> >
> >> > I think we have too many bugs to make an RC this coming week. I
> suggest
> >> we
> >> > defer to the following week.
> >> >
> >> > Thanks
> >> > Wes
> >> >
> >> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
> >> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > We currently have 10 issues in progress and PRs are available for 8 of
> >> > them. In interest of getting a release candidate next week, I would
> >> request
> >> > people to review PRs as soon as they can to help make progress and
> close
> >> > out as many JIRAs as we can.
> >> >
> >> > There are 32 issues in TODO list and 25 of them are not yet assigned.
> I
> >> am
> >> > planning to defer some of the unassigned ones later today or
> tomorrow. It
> >> > would be good to soon grab/assign the issues that people want to be
> fixed
> >> > for 0.9.0.
> >> >
> >> > Here is the link to backlog:
> >> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
> >> >
> >> > Thanks,
> >> > Sidd
> >>
>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Since almost all of the items in TODO are C++ or Python issues, I can
do a final review today to remove anything that isn't absolutely
necessary for 0.9.0. We have a couple of nasty bugs still in TODO that
we should try to fix -- in the event that they cannot be fixed, we may
need to do a 0.9.1 in a week or two. I would suggest we wait to cut
the RC until Monday to give enough time for these last items to get
fixes in.

There are some other things that need doing, like updates per changes
to the ASF checksum policy ARROW-2268.

I can write by EOD today with a status report on the issues in TODO.

I believe you need to be a PMC to undertake the source release process
prior to the vote -- I am happy to help with this on Monday.

- Wes

On Thu, Mar 8, 2018 at 2:25 PM, Siddharth Teotia <si...@dremio.com> wrote:
> All,
>
> I plan to get RC out over the weekend or early Monday. Is that fine with
> everybody?
>
> We have 6 items in progress --
> https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
> How do people feel about completing these JIRAs by tomorrow? I am
> completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
> necessary. Just looking for consensus. Also, I suggest that we defer the
> ones with TODO status. I will do it later today unless I hear otherwise.
>
> I was wondering if anyone else is interested in collaborating for the
> post-release tasks. As per
> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
> following are the high level post-release tasks. Please let me know if you
> would like to take up something. I have written my name against some of
> them.
>
>
>    - Updating the Arrow Website (Sidd)
>    - Uploading release artifacts to SVN -- looks like PMC karma is needed
>    to do this
>    - Announcing release (Sidd)
>    - Updating website with new API documentation (Sidd)
>    - Updating pip packages for C++ and Python
>    - Updating conda packages for C++ and Python (Sidd)
>    - Updating Java Maven artifacts in Maven central (Sidd)
>    - Release blog post
>
> If anything is missing, please add to the above list. It will be helpful
> for tracking.
>
> Thanks,
> Sidd
>
> On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:
>
>> hey Sidd,
>>
>> The Python backlog is still in pretty rough shape. I'd like to see if
>> we can make an RC by Friday but if not we can defer to Monday/Tuesday
>> the following week (3/12 or 13). I will trim as much as possible out
>> of the current backlog to get things down to the essential
>>
>> - Wes
>>
>> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
>> wrote:
>> > Sounds good.
>> >
>> > Thanks
>> > Sidd
>> >
>> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>> >
>> > Hi Sidd,
>> >
>> > I think we have too many bugs to make an RC this coming week. I suggest
>> we
>> > defer to the following week.
>> >
>> > Thanks
>> > Wes
>> >
>> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > We currently have 10 issues in progress and PRs are available for 8 of
>> > them. In interest of getting a release candidate next week, I would
>> request
>> > people to review PRs as soon as they can to help make progress and close
>> > out as many JIRAs as we can.
>> >
>> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
>> am
>> > planning to defer some of the unassigned ones later today or tomorrow. It
>> > would be good to soon grab/assign the issues that people want to be fixed
>> > for 0.9.0.
>> >
>> > Here is the link to backlog:
>> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
>> >
>> > Thanks,
>> > Sidd
>>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Siddharth Teotia <si...@dremio.com>.

All,

I plan to get RC out over the weekend or early Monday. Is that fine with
everybody?

We have 6 items in progress --
https://issues.apache.org/jira/projects/ARROW/versions/12341707#release-report-tab-body.
How do people feel about completing these JIRAs by tomorrow? I am
completely fine with deferring the RC to early next week (Mon/Tue/Wed) if
necessary. Just looking for consensus. Also, I suggest that we defer the
ones with TODO status. I will do it later today unless I hear otherwise.

I was wondering if anyone else is interested in collaborating for the
post-release tasks. As per
https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md,
following are the high level post-release tasks. Please let me know if you
would like to take up something. I have written my name against some of
them.

   - Updating the Arrow Website (Sidd)
   - Uploading release artifacts to SVN -- looks like PMC karma is needed
   to do this
   - Announcing release (Sidd)
   - Updating website with new API documentation (Sidd)
   - Updating pip packages for C++ and Python
   - Updating conda packages for C++ and Python (Sidd)
   - Updating Java Maven artifacts in Maven central (Sidd)
   - Release blog post

If anything is missing, please add to the above list. It will be helpful
for tracking.

Thanks,
Sidd

On Sun, Mar 4, 2018 at 12:34 PM, Wes McKinney <we...@gmail.com> wrote:

> hey Sidd,
>
> The Python backlog is still in pretty rough shape. I'd like to see if
> we can make an RC by Friday but if not we can defer to Monday/Tuesday
> the following week (3/12 or 13). I will trim as much as possible out
> of the current backlog to get things down to the essential
>
> - Wes
>
> On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com>
> wrote:
> > Sounds good.
> >
> > Thanks
> > Sidd
> >
> > On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
> >
> > Hi Sidd,
> >
> > I think we have too many bugs to make an RC this coming week. I suggest
> we
> > defer to the following week.
> >
> > Thanks
> > Wes
> >
> > On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com>
> wrote:
> >
> > Hi All,
> >
> > We currently have 10 issues in progress and PRs are available for 8 of
> > them. In interest of getting a release candidate next week, I would
> request
> > people to review PRs as soon as they can to help make progress and close
> > out as many JIRAs as we can.
> >
> > There are 32 issues in TODO list and 25 of them are not yet assigned. I
> am
> > planning to defer some of the unassigned ones later today or tomorrow. It
> > would be good to soon grab/assign the issues that people want to be fixed
> > for 0.9.0.
> >
> > Here is the link to backlog:
> > https://issues.apache.org/jira/projects/ARROW/versions/12341707
> >
> > Thanks,
> > Sidd
>

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

hey Sidd,

The Python backlog is still in pretty rough shape. I'd like to see if
we can make an RC by Friday but if not we can defer to Monday/Tuesday
the following week (3/12 or 13). I will trim as much as possible out
of the current backlog to get things down to the essential

- Wes

On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia <si...@dremio.com> wrote:
> Sounds good.
>
> Thanks
> Sidd
>
> On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:
>
> Hi Sidd,
>
> I think we have too many bugs to make an RC this coming week. I suggest we
> defer to the following week.
>
> Thanks
> Wes
>
> On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com> wrote:
>
> Hi All,
>
> We currently have 10 issues in progress and PRs are available for 8 of
> them. In interest of getting a release candidate next week, I would request
> people to review PRs as soon as they can to help make progress and close
> out as many JIRAs as we can.
>
> There are 32 issues in TODO list and 25 of them are not yet assigned. I am
> planning to defer some of the unassigned ones later today or tomorrow. It
> would be good to soon grab/assign the issues that people want to be fixed
> for 0.9.0.
>
> Here is the link to backlog:
> https://issues.apache.org/jira/projects/ARROW/versions/12341707
>
> Thanks,
> Sidd

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Siddharth Teotia <si...@dremio.com>.

Sounds good.

Thanks
Sidd

On Feb 24, 2018 6:24 PM, "Wes McKinney" <we...@gmail.com> wrote:

Hi Sidd,

I think we have too many bugs to make an RC this coming week. I suggest we
defer to the following week.

Thanks
Wes

On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com> wrote:

Hi All,

We currently have 10 issues in progress and PRs are available for 8 of
them. In interest of getting a release candidate next week, I would request
people to review PRs as soon as they can to help make progress and close
out as many JIRAs as we can.

There are 32 issues in TODO list and 25 of them are not yet assigned. I am
planning to defer some of the unassigned ones later today or tomorrow. It
would be good to soon grab/assign the issues that people want to be fixed
for 0.9.0.

Here is the link to backlog:
https://issues.apache.org/jira/projects/ARROW/versions/12341707

Thanks,
Sidd

Re: Working towards getting 0.9.0 release candidate up next week

Posted by Wes McKinney <we...@gmail.com>.

Hi Sidd,

I think we have too many bugs to make an RC this coming week. I suggest we
defer to the following week.

Thanks
Wes

On Feb 24, 2018 7:09 PM, "Siddharth Teotia" <si...@dremio.com> wrote:

Hi All,

We currently have 10 issues in progress and PRs are available for 8 of
them. In interest of getting a release candidate next week, I would request
people to review PRs as soon as they can to help make progress and close
out as many JIRAs as we can.

There are 32 issues in TODO list and 25 of them are not yet assigned. I am
planning to defer some of the unassigned ones later today or tomorrow. It
would be good to soon grab/assign the issues that people want to be fixed
for 0.9.0.

Here is the link to backlog:
https://issues.apache.org/jira/projects/ARROW/versions/12341707

Thanks,
Sidd