Arrow 0.8.0 release timeline

[Wes McKinney <we...@gmail.com>]

hi folks,

With the way things are looking, I think we should aim to have all
0.8.0 work wrapped up by the end of next week, and hopefully be
positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
14.

There are still 53 JIRAs in TODO in the release milestone and several
in progress

https://issues.apache.org/jira/projects/ARROW/versions/12341352

It's likely we will have to slightly trim scope as the release nears,
but I would like for as much as possible of this work to get done in
time, so I will personally be pushing very hard throughout this week
and next week.

We have quite a lot of loose ends at the moment and many PRs in
various states of review. There is also the in-progress Java vector
hierarchy refactor. Since the Java refactor is important for
downstream users like Spark 2.3.0 it would be great to get this into
0.8.0.

Thanks,
Wes

Re: Arrow 0.8.0 release timeline

[Li Jin <ic...@gmail.com>]

+1
On Mon, Nov 20, 2017 at 7:05 PM Bryan Cutler <cu...@gmail.com> wrote:

> Sounds good to me!
>
> On Mon, Nov 20, 2017 at 11:30 AM, Wes McKinney <we...@gmail.com>
> wrote:
>
> > It seems we have quite a bit of work still in flight for 0.8.0 and
> > follow up matters from the Java refactor. In light of the Thankgiving
> > holiday, I suggest that we instead wrap up the release next week so we
> > can't scrambling later this week or over the weekend. The objective
> > then would be to cut a release candidate on 12/4. Since Spark 2.3.0 is
> > delayed as well, we can hopefully have a patch ready to drop there as
> > soon as the Arrow 0.8.0 release lands.
> >
> > For my part, I'm going to push to wrap up as much C++ and Python work
> > this week as possible.
> >
> > Any other thoughts on this?
> >
> > - Wes
> >
> > On Sat, Nov 11, 2017 at 11:01 AM, Wes McKinney <we...@gmail.com>
> > wrote:
> > > I suggest we extend development through to November 24 to make sure we
> > > have enough time to settle the Java refactor and deal with other
> > > follow up patches that are accumulating. Since this is putting us
> > > quite close to Spark 2.3.0 branching, we'll want to make sure that we
> > > are able to get Spark updated for the 0.8.0 API changes as soon as
> > > practical after 0.8.0 gets out.
> > >
> > > This will also give us enough time to resolve various things on the
> > > C++/Python side. I will have more time next week and the following
> > > week to write patches.
> > >
> > > Does this sound good?
> > >
> > > - Wes
> > >
> > > On Tue, Nov 7, 2017 at 8:29 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> > >> +1, the timeline sounds good.
> > >>
> > >> I will try to pick some of the lingering tickets on the Python/C++
> side.
> > >>
> > >> Uwe
> > >>
> > >> On Mon, Nov 6, 2017, at 11:17 PM, Li Jin wrote:
> > >>> +1.
> > >>>
> > >>> I think this is a reasonable target time line for Java refactoring.
> > >>>
> > >>> On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > hi everyone,
> > >>> >
> > >>> > With the number of items that are still up in the air for 0.8.0,
> and
> > >>> > the fact that it would be great to get the Java vector refactor
> into
> > >>> > the release, I would propose we continue pushing development until
> > the
> > >>> > end of next week, i.e. through ~ Friday November 17. I have also
> been
> > >>> > traveling and have not been able to write as many patches as I
> would
> > >>> > like.
> > >>> >
> > >>> > It might also be good during this time frame to make sure that the
> > >>> > Spark-Arrow integration is able to migrate to the new API after
> > 0.8.0.
> > >>> > There is an open JIRA about adding a Dockerized setup for
> integration
> > >>> > testing with Spark, so this might be good motivation to set up a
> > >>> > head-to-head integration tests between Spark master and Arrow
> master.
> > >>> >
> > >>> > Since the following week of 11/20 has the US Thanksgiving Holiday,
> > >>> > it's likely to be a slow development week anyway, so we could put
> out
> > >>> > an RC on Monday of that week assuming all is going well.
> > >>> >
> > >>> > best
> > >>> > Wes
> > >>> >
> > >>> > On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <wesmckinn@gmail.com
> >
> > wrote:
> > >>> > > hi folks,
> > >>> > >
> > >>> > > With the way things are looking, I think we should aim to have
> all
> > >>> > > 0.8.0 work wrapped up by the end of next week, and hopefully be
> > >>> > > positioned to make a release candidate on Monday Nov 13 or
> Tuesday
> > Nov
> > >>> > > 14.
> > >>> > >
> > >>> > > There are still 53 JIRAs in TODO in the release milestone and
> > several
> > >>> > > in progress
> > >>> > >
> > >>> > > https://issues.apache.org/jira/projects/ARROW/versions/12341352
> > >>> > >
> > >>> > > It's likely we will have to slightly trim scope as the release
> > nears,
> > >>> > > but I would like for as much as possible of this work to get done
> > in
> > >>> > > time, so I will personally be pushing very hard throughout this
> > week
> > >>> > > and next week.
> > >>> > >
> > >>> > > We have quite a lot of loose ends at the moment and many PRs in
> > >>> > > various states of review. There is also the in-progress Java
> vector
> > >>> > > hierarchy refactor. Since the Java refactor is important for
> > >>> > > downstream users like Spark 2.3.0 it would be great to get this
> > into
> > >>> > > 0.8.0.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Wes
> > >>> >
> >
>

Re: Arrow 0.8.0 release timeline

[Bryan Cutler <cu...@gmail.com>]

Sounds good to me!

On Mon, Nov 20, 2017 at 11:30 AM, Wes McKinney <we...@gmail.com> wrote:

> It seems we have quite a bit of work still in flight for 0.8.0 and
> follow up matters from the Java refactor. In light of the Thankgiving
> holiday, I suggest that we instead wrap up the release next week so we
> can't scrambling later this week or over the weekend. The objective
> then would be to cut a release candidate on 12/4. Since Spark 2.3.0 is
> delayed as well, we can hopefully have a patch ready to drop there as
> soon as the Arrow 0.8.0 release lands.
>
> For my part, I'm going to push to wrap up as much C++ and Python work
> this week as possible.
>
> Any other thoughts on this?
>
> - Wes
>
> On Sat, Nov 11, 2017 at 11:01 AM, Wes McKinney <we...@gmail.com>
> wrote:
> > I suggest we extend development through to November 24 to make sure we
> > have enough time to settle the Java refactor and deal with other
> > follow up patches that are accumulating. Since this is putting us
> > quite close to Spark 2.3.0 branching, we'll want to make sure that we
> > are able to get Spark updated for the 0.8.0 API changes as soon as
> > practical after 0.8.0 gets out.
> >
> > This will also give us enough time to resolve various things on the
> > C++/Python side. I will have more time next week and the following
> > week to write patches.
> >
> > Does this sound good?
> >
> > - Wes
> >
> > On Tue, Nov 7, 2017 at 8:29 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> >> +1, the timeline sounds good.
> >>
> >> I will try to pick some of the lingering tickets on the Python/C++ side.
> >>
> >> Uwe
> >>
> >> On Mon, Nov 6, 2017, at 11:17 PM, Li Jin wrote:
> >>> +1.
> >>>
> >>> I think this is a reasonable target time line for Java refactoring.
> >>>
> >>> On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com>
> >>> wrote:
> >>>
> >>> > hi everyone,
> >>> >
> >>> > With the number of items that are still up in the air for 0.8.0, and
> >>> > the fact that it would be great to get the Java vector refactor into
> >>> > the release, I would propose we continue pushing development until
> the
> >>> > end of next week, i.e. through ~ Friday November 17. I have also been
> >>> > traveling and have not been able to write as many patches as I would
> >>> > like.
> >>> >
> >>> > It might also be good during this time frame to make sure that the
> >>> > Spark-Arrow integration is able to migrate to the new API after
> 0.8.0.
> >>> > There is an open JIRA about adding a Dockerized setup for integration
> >>> > testing with Spark, so this might be good motivation to set up a
> >>> > head-to-head integration tests between Spark master and Arrow master.
> >>> >
> >>> > Since the following week of 11/20 has the US Thanksgiving Holiday,
> >>> > it's likely to be a slow development week anyway, so we could put out
> >>> > an RC on Monday of that week assuming all is going well.
> >>> >
> >>> > best
> >>> > Wes
> >>> >
> >>> > On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com>
> wrote:
> >>> > > hi folks,
> >>> > >
> >>> > > With the way things are looking, I think we should aim to have all
> >>> > > 0.8.0 work wrapped up by the end of next week, and hopefully be
> >>> > > positioned to make a release candidate on Monday Nov 13 or Tuesday
> Nov
> >>> > > 14.
> >>> > >
> >>> > > There are still 53 JIRAs in TODO in the release milestone and
> several
> >>> > > in progress
> >>> > >
> >>> > > https://issues.apache.org/jira/projects/ARROW/versions/12341352
> >>> > >
> >>> > > It's likely we will have to slightly trim scope as the release
> nears,
> >>> > > but I would like for as much as possible of this work to get done
> in
> >>> > > time, so I will personally be pushing very hard throughout this
> week
> >>> > > and next week.
> >>> > >
> >>> > > We have quite a lot of loose ends at the moment and many PRs in
> >>> > > various states of review. There is also the in-progress Java vector
> >>> > > hierarchy refactor. Since the Java refactor is important for
> >>> > > downstream users like Spark 2.3.0 it would be great to get this
> into
> >>> > > 0.8.0.
> >>> > >
> >>> > > Thanks,
> >>> > > Wes
> >>> >
>

Re: Arrow 0.8.0 release timeline

[Wes McKinney <we...@gmail.com>]

It seems we have quite a bit of work still in flight for 0.8.0 and
follow up matters from the Java refactor. In light of the Thankgiving
holiday, I suggest that we instead wrap up the release next week so we
can't scrambling later this week or over the weekend. The objective
then would be to cut a release candidate on 12/4. Since Spark 2.3.0 is
delayed as well, we can hopefully have a patch ready to drop there as
soon as the Arrow 0.8.0 release lands.

For my part, I'm going to push to wrap up as much C++ and Python work
this week as possible.

Any other thoughts on this?

- Wes

On Sat, Nov 11, 2017 at 11:01 AM, Wes McKinney <we...@gmail.com> wrote:
> I suggest we extend development through to November 24 to make sure we
> have enough time to settle the Java refactor and deal with other
> follow up patches that are accumulating. Since this is putting us
> quite close to Spark 2.3.0 branching, we'll want to make sure that we
> are able to get Spark updated for the 0.8.0 API changes as soon as
> practical after 0.8.0 gets out.
>
> This will also give us enough time to resolve various things on the
> C++/Python side. I will have more time next week and the following
> week to write patches.
>
> Does this sound good?
>
> - Wes
>
> On Tue, Nov 7, 2017 at 8:29 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>> +1, the timeline sounds good.
>>
>> I will try to pick some of the lingering tickets on the Python/C++ side.
>>
>> Uwe
>>
>> On Mon, Nov 6, 2017, at 11:17 PM, Li Jin wrote:
>>> +1.
>>>
>>> I think this is a reasonable target time line for Java refactoring.
>>>
>>> On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com>
>>> wrote:
>>>
>>> > hi everyone,
>>> >
>>> > With the number of items that are still up in the air for 0.8.0, and
>>> > the fact that it would be great to get the Java vector refactor into
>>> > the release, I would propose we continue pushing development until the
>>> > end of next week, i.e. through ~ Friday November 17. I have also been
>>> > traveling and have not been able to write as many patches as I would
>>> > like.
>>> >
>>> > It might also be good during this time frame to make sure that the
>>> > Spark-Arrow integration is able to migrate to the new API after 0.8.0.
>>> > There is an open JIRA about adding a Dockerized setup for integration
>>> > testing with Spark, so this might be good motivation to set up a
>>> > head-to-head integration tests between Spark master and Arrow master.
>>> >
>>> > Since the following week of 11/20 has the US Thanksgiving Holiday,
>>> > it's likely to be a slow development week anyway, so we could put out
>>> > an RC on Monday of that week assuming all is going well.
>>> >
>>> > best
>>> > Wes
>>> >
>>> > On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com> wrote:
>>> > > hi folks,
>>> > >
>>> > > With the way things are looking, I think we should aim to have all
>>> > > 0.8.0 work wrapped up by the end of next week, and hopefully be
>>> > > positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
>>> > > 14.
>>> > >
>>> > > There are still 53 JIRAs in TODO in the release milestone and several
>>> > > in progress
>>> > >
>>> > > https://issues.apache.org/jira/projects/ARROW/versions/12341352
>>> > >
>>> > > It's likely we will have to slightly trim scope as the release nears,
>>> > > but I would like for as much as possible of this work to get done in
>>> > > time, so I will personally be pushing very hard throughout this week
>>> > > and next week.
>>> > >
>>> > > We have quite a lot of loose ends at the moment and many PRs in
>>> > > various states of review. There is also the in-progress Java vector
>>> > > hierarchy refactor. Since the Java refactor is important for
>>> > > downstream users like Spark 2.3.0 it would be great to get this into
>>> > > 0.8.0.
>>> > >
>>> > > Thanks,
>>> > > Wes
>>> >

Re: Arrow 0.8.0 release timeline

[Wes McKinney <we...@gmail.com>]

I suggest we extend development through to November 24 to make sure we
have enough time to settle the Java refactor and deal with other
follow up patches that are accumulating. Since this is putting us
quite close to Spark 2.3.0 branching, we'll want to make sure that we
are able to get Spark updated for the 0.8.0 API changes as soon as
practical after 0.8.0 gets out.

This will also give us enough time to resolve various things on the
C++/Python side. I will have more time next week and the following
week to write patches.

Does this sound good?

- Wes

On Tue, Nov 7, 2017 at 8:29 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> +1, the timeline sounds good.
>
> I will try to pick some of the lingering tickets on the Python/C++ side.
>
> Uwe
>
> On Mon, Nov 6, 2017, at 11:17 PM, Li Jin wrote:
>> +1.
>>
>> I think this is a reasonable target time line for Java refactoring.
>>
>> On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com>
>> wrote:
>>
>> > hi everyone,
>> >
>> > With the number of items that are still up in the air for 0.8.0, and
>> > the fact that it would be great to get the Java vector refactor into
>> > the release, I would propose we continue pushing development until the
>> > end of next week, i.e. through ~ Friday November 17. I have also been
>> > traveling and have not been able to write as many patches as I would
>> > like.
>> >
>> > It might also be good during this time frame to make sure that the
>> > Spark-Arrow integration is able to migrate to the new API after 0.8.0.
>> > There is an open JIRA about adding a Dockerized setup for integration
>> > testing with Spark, so this might be good motivation to set up a
>> > head-to-head integration tests between Spark master and Arrow master.
>> >
>> > Since the following week of 11/20 has the US Thanksgiving Holiday,
>> > it's likely to be a slow development week anyway, so we could put out
>> > an RC on Monday of that week assuming all is going well.
>> >
>> > best
>> > Wes
>> >
>> > On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com> wrote:
>> > > hi folks,
>> > >
>> > > With the way things are looking, I think we should aim to have all
>> > > 0.8.0 work wrapped up by the end of next week, and hopefully be
>> > > positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
>> > > 14.
>> > >
>> > > There are still 53 JIRAs in TODO in the release milestone and several
>> > > in progress
>> > >
>> > > https://issues.apache.org/jira/projects/ARROW/versions/12341352
>> > >
>> > > It's likely we will have to slightly trim scope as the release nears,
>> > > but I would like for as much as possible of this work to get done in
>> > > time, so I will personally be pushing very hard throughout this week
>> > > and next week.
>> > >
>> > > We have quite a lot of loose ends at the moment and many PRs in
>> > > various states of review. There is also the in-progress Java vector
>> > > hierarchy refactor. Since the Java refactor is important for
>> > > downstream users like Spark 2.3.0 it would be great to get this into
>> > > 0.8.0.
>> > >
>> > > Thanks,
>> > > Wes
>> >

Re: Arrow 0.8.0 release timeline

["Uwe L. Korn" <uw...@xhochy.com>]

+1, the timeline sounds good. 

I will try to pick some of the lingering tickets on the Python/C++ side.

Uwe

On Mon, Nov 6, 2017, at 11:17 PM, Li Jin wrote:
> +1.
> 
> I think this is a reasonable target time line for Java refactoring.
> 
> On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com>
> wrote:
> 
> > hi everyone,
> >
> > With the number of items that are still up in the air for 0.8.0, and
> > the fact that it would be great to get the Java vector refactor into
> > the release, I would propose we continue pushing development until the
> > end of next week, i.e. through ~ Friday November 17. I have also been
> > traveling and have not been able to write as many patches as I would
> > like.
> >
> > It might also be good during this time frame to make sure that the
> > Spark-Arrow integration is able to migrate to the new API after 0.8.0.
> > There is an open JIRA about adding a Dockerized setup for integration
> > testing with Spark, so this might be good motivation to set up a
> > head-to-head integration tests between Spark master and Arrow master.
> >
> > Since the following week of 11/20 has the US Thanksgiving Holiday,
> > it's likely to be a slow development week anyway, so we could put out
> > an RC on Monday of that week assuming all is going well.
> >
> > best
> > Wes
> >
> > On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com> wrote:
> > > hi folks,
> > >
> > > With the way things are looking, I think we should aim to have all
> > > 0.8.0 work wrapped up by the end of next week, and hopefully be
> > > positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
> > > 14.
> > >
> > > There are still 53 JIRAs in TODO in the release milestone and several
> > > in progress
> > >
> > > https://issues.apache.org/jira/projects/ARROW/versions/12341352
> > >
> > > It's likely we will have to slightly trim scope as the release nears,
> > > but I would like for as much as possible of this work to get done in
> > > time, so I will personally be pushing very hard throughout this week
> > > and next week.
> > >
> > > We have quite a lot of loose ends at the moment and many PRs in
> > > various states of review. There is also the in-progress Java vector
> > > hierarchy refactor. Since the Java refactor is important for
> > > downstream users like Spark 2.3.0 it would be great to get this into
> > > 0.8.0.
> > >
> > > Thanks,
> > > Wes
> >

Re: Arrow 0.8.0 release timeline

[Li Jin <ic...@gmail.com>]

+1.

I think this is a reasonable target time line for Java refactoring.

On Mon, Nov 6, 2017 at 12:51 PM, Wes McKinney <we...@gmail.com> wrote:

> hi everyone,
>
> With the number of items that are still up in the air for 0.8.0, and
> the fact that it would be great to get the Java vector refactor into
> the release, I would propose we continue pushing development until the
> end of next week, i.e. through ~ Friday November 17. I have also been
> traveling and have not been able to write as many patches as I would
> like.
>
> It might also be good during this time frame to make sure that the
> Spark-Arrow integration is able to migrate to the new API after 0.8.0.
> There is an open JIRA about adding a Dockerized setup for integration
> testing with Spark, so this might be good motivation to set up a
> head-to-head integration tests between Spark master and Arrow master.
>
> Since the following week of 11/20 has the US Thanksgiving Holiday,
> it's likely to be a slow development week anyway, so we could put out
> an RC on Monday of that week assuming all is going well.
>
> best
> Wes
>
> On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com> wrote:
> > hi folks,
> >
> > With the way things are looking, I think we should aim to have all
> > 0.8.0 work wrapped up by the end of next week, and hopefully be
> > positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
> > 14.
> >
> > There are still 53 JIRAs in TODO in the release milestone and several
> > in progress
> >
> > https://issues.apache.org/jira/projects/ARROW/versions/12341352
> >
> > It's likely we will have to slightly trim scope as the release nears,
> > but I would like for as much as possible of this work to get done in
> > time, so I will personally be pushing very hard throughout this week
> > and next week.
> >
> > We have quite a lot of loose ends at the moment and many PRs in
> > various states of review. There is also the in-progress Java vector
> > hierarchy refactor. Since the Java refactor is important for
> > downstream users like Spark 2.3.0 it would be great to get this into
> > 0.8.0.
> >
> > Thanks,
> > Wes
>

Re: Arrow 0.8.0 release timeline

[Wes McKinney <we...@gmail.com>]

hi everyone,

With the number of items that are still up in the air for 0.8.0, and
the fact that it would be great to get the Java vector refactor into
the release, I would propose we continue pushing development until the
end of next week, i.e. through ~ Friday November 17. I have also been
traveling and have not been able to write as many patches as I would
like.

It might also be good during this time frame to make sure that the
Spark-Arrow integration is able to migrate to the new API after 0.8.0.
There is an open JIRA about adding a Dockerized setup for integration
testing with Spark, so this might be good motivation to set up a
head-to-head integration tests between Spark master and Arrow master.

Since the following week of 11/20 has the US Thanksgiving Holiday,
it's likely to be a slow development week anyway, so we could put out
an RC on Monday of that week assuming all is going well.

best
Wes

On Mon, Oct 30, 2017 at 2:52 PM, Wes McKinney <we...@gmail.com> wrote:
> hi folks,
>
> With the way things are looking, I think we should aim to have all
> 0.8.0 work wrapped up by the end of next week, and hopefully be
> positioned to make a release candidate on Monday Nov 13 or Tuesday Nov
> 14.
>
> There are still 53 JIRAs in TODO in the release milestone and several
> in progress
>
> https://issues.apache.org/jira/projects/ARROW/versions/12341352
>
> It's likely we will have to slightly trim scope as the release nears,
> but I would like for as much as possible of this work to get done in
> time, so I will personally be pushing very hard throughout this week
> and next week.
>
> We have quite a lot of loose ends at the moment and many PRs in
> various states of review. There is also the in-progress Java vector
> hierarchy refactor. Since the Java refactor is important for
> downstream users like Spark 2.3.0 it would be great to get this into
> 0.8.0.
>
> Thanks,
> Wes