You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Siddharth Teotia <si...@dremio.com> on 2017/11/13 22:31:49 UTC
Update on ARROW-1463 - Request for merging java-vector-refactor into master
Functional and Performance testing has been completed with Dremio. We have
seen overall improvement in TPCH numbers. We had about 8000 regression
tests and 12000 unit tests.
I would like to start the process of merging java-vector-refactor branch
into master. The branch has 2 patches with 95% of code changes and a third
patch will be available in couple of hours -- minor bug fixes needed as
part of testing in Dremio.
I would also like to request to not merge any orthogonal set of changes to
refactor branch at this point. It will require grabbing changes again, and
testing downstream and thus increase the timeline of merging this into
master. It is become slightly difficult to maintain local branches with
such volume of changes and work across patches.
Follow-up items in the order of priority:
- buffer consolidation (sooner but probably not this Arrow release) --
possibly the highest priority item in the effort to reduce heap usage. This
was touched upon in design spec but it wasn't feasible to squeeze it into
the work already done on refactor branch.
- fate of non-nullable fixed and var width vectors
Thanks,
Siddharth
Re: Update on ARROW-1463 - Request for merging java-vector-refactor
into master
Posted by Siddharth Teotia <si...@dremio.com>.
Sure, no sweat.
On Mon, Nov 13, 2017 at 3:50 PM, Bryan Cutler <cu...@gmail.com> wrote:
> Thanks Siddharth, great work! I'd like to run through it a final time and
> possibly try out with Spark before merging to master to make sure there
> won't be any big issues. Could we set tomorrow EOD as the time as the time
> to merge, unless anything comes up?
>
> Thanks,
> Bryan
>
> On Mon, Nov 13, 2017 at 2:39 PM, Li Jin <ic...@gmail.com> wrote:
>
> > Thanks Siddharth.
> >
> > I agree we should start merging refactor branch to master - this will
> > allows us to make further Java changes easier. There are still left items
> > on https://issues.apache.org/jira/browse/ARROW-1463, we can work on
> those
> > after merging to master.
> >
> >
> > On Mon, Nov 13, 2017 at 5:31 PM, Siddharth Teotia <si...@dremio.com>
> > wrote:
> >
> > > Functional and Performance testing has been completed with Dremio. We
> > have
> > > seen overall improvement in TPCH numbers. We had about 8000 regression
> > > tests and 12000 unit tests.
> > >
> > > I would like to start the process of merging java-vector-refactor
> branch
> > > into master. The branch has 2 patches with 95% of code changes and a
> > third
> > > patch will be available in couple of hours -- minor bug fixes needed as
> > > part of testing in Dremio.
> > >
> > > I would also like to request to not merge any orthogonal set of changes
> > to
> > > refactor branch at this point. It will require grabbing changes again,
> > and
> > > testing downstream and thus increase the timeline of merging this into
> > > master. It is become slightly difficult to maintain local branches with
> > > such volume of changes and work across patches.
> > >
> > > Follow-up items in the order of priority:
> > >
> > > - buffer consolidation (sooner but probably not this Arrow release)
> --
> > > possibly the highest priority item in the effort to reduce heap
> usage.
> > > This
> > > was touched upon in design spec but it wasn't feasible to squeeze it
> > > into
> > > the work already done on refactor branch.
> > >
> > >
> > > - fate of non-nullable fixed and var width vectors
> > >
> > > Thanks,
> > > Siddharth
> > >
> >
>
Re: Update on ARROW-1463 - Request for merging java-vector-refactor
into master
Posted by Bryan Cutler <cu...@gmail.com>.
Thanks Siddharth, great work! I'd like to run through it a final time and
possibly try out with Spark before merging to master to make sure there
won't be any big issues. Could we set tomorrow EOD as the time as the time
to merge, unless anything comes up?
Thanks,
Bryan
On Mon, Nov 13, 2017 at 2:39 PM, Li Jin <ic...@gmail.com> wrote:
> Thanks Siddharth.
>
> I agree we should start merging refactor branch to master - this will
> allows us to make further Java changes easier. There are still left items
> on https://issues.apache.org/jira/browse/ARROW-1463, we can work on those
> after merging to master.
>
>
> On Mon, Nov 13, 2017 at 5:31 PM, Siddharth Teotia <si...@dremio.com>
> wrote:
>
> > Functional and Performance testing has been completed with Dremio. We
> have
> > seen overall improvement in TPCH numbers. We had about 8000 regression
> > tests and 12000 unit tests.
> >
> > I would like to start the process of merging java-vector-refactor branch
> > into master. The branch has 2 patches with 95% of code changes and a
> third
> > patch will be available in couple of hours -- minor bug fixes needed as
> > part of testing in Dremio.
> >
> > I would also like to request to not merge any orthogonal set of changes
> to
> > refactor branch at this point. It will require grabbing changes again,
> and
> > testing downstream and thus increase the timeline of merging this into
> > master. It is become slightly difficult to maintain local branches with
> > such volume of changes and work across patches.
> >
> > Follow-up items in the order of priority:
> >
> > - buffer consolidation (sooner but probably not this Arrow release) --
> > possibly the highest priority item in the effort to reduce heap usage.
> > This
> > was touched upon in design spec but it wasn't feasible to squeeze it
> > into
> > the work already done on refactor branch.
> >
> >
> > - fate of non-nullable fixed and var width vectors
> >
> > Thanks,
> > Siddharth
> >
>
Re: Update on ARROW-1463 - Request for merging java-vector-refactor
into master
Posted by Li Jin <ic...@gmail.com>.
Thanks Siddharth.
I agree we should start merging refactor branch to master - this will
allows us to make further Java changes easier. There are still left items
on https://issues.apache.org/jira/browse/ARROW-1463, we can work on those
after merging to master.
On Mon, Nov 13, 2017 at 5:31 PM, Siddharth Teotia <si...@dremio.com>
wrote:
> Functional and Performance testing has been completed with Dremio. We have
> seen overall improvement in TPCH numbers. We had about 8000 regression
> tests and 12000 unit tests.
>
> I would like to start the process of merging java-vector-refactor branch
> into master. The branch has 2 patches with 95% of code changes and a third
> patch will be available in couple of hours -- minor bug fixes needed as
> part of testing in Dremio.
>
> I would also like to request to not merge any orthogonal set of changes to
> refactor branch at this point. It will require grabbing changes again, and
> testing downstream and thus increase the timeline of merging this into
> master. It is become slightly difficult to maintain local branches with
> such volume of changes and work across patches.
>
> Follow-up items in the order of priority:
>
> - buffer consolidation (sooner but probably not this Arrow release) --
> possibly the highest priority item in the effort to reduce heap usage.
> This
> was touched upon in design spec but it wasn't feasible to squeeze it
> into
> the work already done on refactor branch.
>
>
> - fate of non-nullable fixed and var width vectors
>
> Thanks,
> Siddharth
>