You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Julian Hyde <jh...@apache.org> on 2017/08/31 18:43:54 UTC
Re: Working on upgrade Drill Calcite version
+dev@calcite.
Excellent work, Roman.
Your changes to Calcite don’t seem to be very major, and I ran the test suite on your https://github.com/KulykRoman/incubator-calcite/commits/DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are ready, I think we can expedite getting these changes back into Calcite.
Calcite will release 1.14 probably in the next two weeks. So, when you’re on to 1.13 let’s continue the momentum and get up to the latest release. I would love to get to the situation that we can ask Drill to verify a Calcite release before we make it. There are possibly breaking changes in the area of schema and grouping-sets aggregate but it should be mostly straightforward.
Julian
> On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
>
> It's great to hear that someone is working on moving Drill on new Calcite
> release! As someone who did similar job in last two rebase work, I could
> see that there would be many regressions to fix. Please let me know if I
> can provide any help.
>
> I thought we have pushed almost Drill specific commits to Calcite. I'll
> take a look at your Calcite branch. Ideally, we want to get rid of all
> Drill specific commits, after the rebase work this time.
>
> Thanks,
>
> Jinfeng
>
>
>
>
> On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org> wrote:
>
>> Yes, this is long overdue ! Thanks for working on it Roman. If needed,
>> we can do a separate hangout with a few selected folks who have worked on
>> Drill+Calcite to provide feedback on how to resolve the test failures.
>> Although, at this stage you probably know a lot more about the integration
>> than many of us.
>>
>> -Aman
>>
>> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com> wrote:
>>
>>> Hi, team!
>>>
>>> I want to inform that I am working on updating Drill Calcite version to
>>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
>>> - DRILL-1455: Add return type-inference strategy for arithmetic operators
>>> when one of the arguments is ANY type. (was not in Calcite)
>>> - Add new method to ViewExpander interface to allow passing SchemaRoot.
>>> (was not in Calcite)
>>> - Allow a MAP literal type. (was not in Calcite)
>>> - DRILL-4047: Modify table functions to allow querying a table with
>> options
>>> in Drill (was not in Calcite)
>>> - Drill-specific change: Add back AbstractConverter in RelSet.java ...
>>> (small changes after a3bc0d8 commit)
>>> - CALCITE-628 related but not fix the problem: Ensure target traits are
>>> simple when use Frameworks or RelOptRule.convert() method. (small changes
>>> after a3bc0d8 commit)
>>> - Support select * from schema-less table in execution engine like Drill
>>> (small changes after 9bd7d75 commit)
>>>
>>> Only after adding these commits to Calcite, a part of Drill
>> functionality
>>> started to work as before. Currently, I am at Drill integration stage. I
>> am
>>> fixing Drill unit tests. Progress can be tracked in my branches.
>>>
>>> Drill: https://github.com/KulykRoman/drill/commits/CalciteForkRebase_rc1
>>> Calcite:
>>> https://github.com/KulykRoman/incubator-calcite/commits/
>>> DrillCalcite1.13.0_rc1
>>>
>>> Currently, I face ~160 errors in java-exec module unit tests. I have
>>> divided them into seven groups:
>>> 1) SYSTEM ERROR: ClassCastException
>>> 2) Unexpected column errors
>>> 3) Runtime Exceptions
>>> 4) validation errors
>>> 5) RpcExceptions (generalized group: some of them should be fixed by the
>>> previous groups)
>>> 6) IllegalState errors (generalized group: some of them should be fixed
>> by
>>> the previous group)
>>> 7) Other errors
>>>
>>> Best regards,
>>> Roman Kulyk
>>>
>>
Re: Working on upgrade Drill Calcite version
Posted by Michael Mior <mm...@uwaterloo.ca>.
Fantastic news! Thanks to all who made this happen.
--
Michael Mior
mmior@apache.org
2018-01-19 5:43 GMT-05:00 Arina Ielchiieva <ar...@apache.org>:
> I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
> information can be found in DRILL-3993 and in Google Doc [1].
> Currently changes are in master and are intended to be part of Drill 1.13.0
> release unless something critical pops up. Feel free to test them and give
> your feedback.
>
> Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
> Aman Sinha and Chunhui Shi for the code review.
> And of course Julian Hyde for helping to address / review Calcite specific
> issues.
>
> [1]
> https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pu
> i7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
>
>
> Kind regards
> Arina
>
> On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
>
> > Team, thanks for your answers!
> >
> > Julian, сontinuing the topic of Calcite tests I just want to note that I
> > have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> > ignored tests:
> >
> > TraitConversionTest.testTraitConversion
> > JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> > JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> > JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> > LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTwoLattices
> >
> > You can see detailed information in my comments. Also, I want to notice
> > that there are 2 required commits in my Calcite branch: 5ba68240 and
> > 1f5145b1. Without them, we lose a big part of Drill functionality.
> >
> > Best regards,
> > Roman Kulyk
> >
> > 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
> >
> > > +dev@calcite.
> > >
> > > Excellent work, Roman.
> > >
> > > Your changes to Calcite don’t seem to be very major, and I ran the test
> > > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > > ready, I think we can expedite getting these changes back into Calcite.
> > >
> > > Calcite will release 1.14 probably in the next two weeks. So, when
> you’re
> > > on to 1.13 let’s continue the momentum and get up to the latest
> release.
> > I
> > > would love to get to the situation that we can ask Drill to verify a
> > > Calcite release before we make it. There are possibly breaking changes
> in
> > > the area of schema and grouping-sets aggregate but it should be mostly
> > > straightforward.
> > >
> > > Julian
> > >
> > >
> > > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > > >
> > > > It's great to hear that someone is working on moving Drill on new
> > Calcite
> > > > release! As someone who did similar job in last two rebase work, I
> > could
> > > > see that there would be many regressions to fix. Please let me know
> if
> > I
> > > > can provide any help.
> > > >
> > > > I thought we have pushed almost Drill specific commits to Calcite.
> I'll
> > > > take a look at your Calcite branch. Ideally, we want to get rid of
> all
> > > > Drill specific commits, after the rebase work this time.
> > > >
> > > > Thanks,
> > > >
> > > > Jinfeng
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > > wrote:
> > > >
> > > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > > needed,
> > > >> we can do a separate hangout with a few selected folks who have
> worked
> > > on
> > > >> Drill+Calcite to provide feedback on how to resolve the test
> failures.
> > > >> Although, at this stage you probably know a lot more about the
> > > integration
> > > >> than many of us.
> > > >>
> > > >> -Aman
> > > >>
> > > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi, team!
> > > >>>
> > > >>> I want to inform that I am working on updating Drill Calcite
> version
> > to
> > > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > > operators
> > > >>> when one of the arguments is ANY type. (was not in Calcite)
> > > >>> - Add new method to ViewExpander interface to allow passing
> > SchemaRoot.
> > > >>> (was not in Calcite)
> > > >>> - Allow a MAP literal type. (was not in Calcite)
> > > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > > >> options
> > > >>> in Drill (was not in Calcite)
> > > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> > ...
> > > >>> (small changes after a3bc0d8 commit)
> > > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> > are
> > > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > > changes
> > > >>> after a3bc0d8 commit)
> > > >>> - Support select * from schema-less table in execution engine like
> > > Drill
> > > >>> (small changes after 9bd7d75 commit)
> > > >>>
> > > >>> Only after adding these commits to Calcite, a part of Drill
> > > >> functionality
> > > >>> started to work as before. Currently, I am at Drill integration
> > stage.
> > > I
> > > >> am
> > > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > > >>>
> > > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > > CalciteForkRebase_rc1
> > > >>> Calcite:
> > > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > > >>> DrillCalcite1.13.0_rc1
> > > >>>
> > > >>> Currently, I face ~160 errors in java-exec module unit tests. I
> have
> > > >>> divided them into seven groups:
> > > >>> 1) SYSTEM ERROR: ClassCastException
> > > >>> 2) Unexpected column errors
> > > >>> 3) Runtime Exceptions
> > > >>> 4) validation errors
> > > >>> 5) RpcExceptions (generalized group: some of them should be fixed
> by
> > > the
> > > >>> previous groups)
> > > >>> 6) IllegalState errors (generalized group: some of them should be
> > fixed
> > > >> by
> > > >>> the previous group)
> > > >>> 7) Other errors
> > > >>>
> > > >>> Best regards,
> > > >>> Roman Kulyk
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Charles Givre <cg...@gmail.com>.
Great work!
> On Jan 19, 2018, at 05:43, Arina Ielchiieva <ar...@apache.org> wrote:
>
> I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
> information can be found in DRILL-3993 and in Google Doc [1].
> Currently changes are in master and are intended to be part of Drill 1.13.0
> release unless something critical pops up. Feel free to test them and give
> your feedback.
>
> Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
> Aman Sinha and Chunhui Shi for the code review.
> And of course Julian Hyde for helping to address / review Calcite specific
> issues.
>
> [1]
> https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pui7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
>
>
> Kind regards
> Arina
>
> On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
>
>> Team, thanks for your answers!
>>
>> Julian, сontinuing the topic of Calcite tests I just want to note that I
>> have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
>> ignored tests:
>>
>> TraitConversionTest.testTraitConversion
>> JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
>> JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
>> JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
>> LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
>> LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
>> LatticeTest.testTwoLattices
>>
>> You can see detailed information in my comments. Also, I want to notice
>> that there are 2 required commits in my Calcite branch: 5ba68240 and
>> 1f5145b1. Without them, we lose a big part of Drill functionality.
>>
>> Best regards,
>> Roman Kulyk
>>
>> 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
>>
>>> +dev@calcite.
>>>
>>> Excellent work, Roman.
>>>
>>> Your changes to Calcite don’t seem to be very major, and I ran the test
>>> suite on your https://github.com/KulykRoman/incubator-calcite/commits/
>>> DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
>>> ready, I think we can expedite getting these changes back into Calcite.
>>>
>>> Calcite will release 1.14 probably in the next two weeks. So, when you’re
>>> on to 1.13 let’s continue the momentum and get up to the latest release.
>> I
>>> would love to get to the situation that we can ask Drill to verify a
>>> Calcite release before we make it. There are possibly breaking changes in
>>> the area of schema and grouping-sets aggregate but it should be mostly
>>> straightforward.
>>>
>>> Julian
>>>
>>>
>>>> On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>
>>>> It's great to hear that someone is working on moving Drill on new
>> Calcite
>>>> release! As someone who did similar job in last two rebase work, I
>> could
>>>> see that there would be many regressions to fix. Please let me know if
>> I
>>>> can provide any help.
>>>>
>>>> I thought we have pushed almost Drill specific commits to Calcite. I'll
>>>> take a look at your Calcite branch. Ideally, we want to get rid of all
>>>> Drill specific commits, after the rebase work this time.
>>>>
>>>> Thanks,
>>>>
>>>> Jinfeng
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
>>> wrote:
>>>>
>>>>> Yes, this is long overdue ! Thanks for working on it Roman. If
>>> needed,
>>>>> we can do a separate hangout with a few selected folks who have worked
>>> on
>>>>> Drill+Calcite to provide feedback on how to resolve the test failures.
>>>>> Although, at this stage you probably know a lot more about the
>>> integration
>>>>> than many of us.
>>>>>
>>>>> -Aman
>>>>>
>>>>> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hi, team!
>>>>>>
>>>>>> I want to inform that I am working on updating Drill Calcite version
>> to
>>>>>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
>>>>>> - DRILL-1455: Add return type-inference strategy for arithmetic
>>> operators
>>>>>> when one of the arguments is ANY type. (was not in Calcite)
>>>>>> - Add new method to ViewExpander interface to allow passing
>> SchemaRoot.
>>>>>> (was not in Calcite)
>>>>>> - Allow a MAP literal type. (was not in Calcite)
>>>>>> - DRILL-4047: Modify table functions to allow querying a table with
>>>>> options
>>>>>> in Drill (was not in Calcite)
>>>>>> - Drill-specific change: Add back AbstractConverter in RelSet.java
>> ...
>>>>>> (small changes after a3bc0d8 commit)
>>>>>> - CALCITE-628 related but not fix the problem: Ensure target traits
>> are
>>>>>> simple when use Frameworks or RelOptRule.convert() method. (small
>>> changes
>>>>>> after a3bc0d8 commit)
>>>>>> - Support select * from schema-less table in execution engine like
>>> Drill
>>>>>> (small changes after 9bd7d75 commit)
>>>>>>
>>>>>> Only after adding these commits to Calcite, a part of Drill
>>>>> functionality
>>>>>> started to work as before. Currently, I am at Drill integration
>> stage.
>>> I
>>>>> am
>>>>>> fixing Drill unit tests. Progress can be tracked in my branches.
>>>>>>
>>>>>> Drill: https://github.com/KulykRoman/drill/commits/
>>> CalciteForkRebase_rc1
>>>>>> Calcite:
>>>>>> https://github.com/KulykRoman/incubator-calcite/commits/
>>>>>> DrillCalcite1.13.0_rc1
>>>>>>
>>>>>> Currently, I face ~160 errors in java-exec module unit tests. I have
>>>>>> divided them into seven groups:
>>>>>> 1) SYSTEM ERROR: ClassCastException
>>>>>> 2) Unexpected column errors
>>>>>> 3) Runtime Exceptions
>>>>>> 4) validation errors
>>>>>> 5) RpcExceptions (generalized group: some of them should be fixed by
>>> the
>>>>>> previous groups)
>>>>>> 6) IllegalState errors (generalized group: some of them should be
>> fixed
>>>>> by
>>>>>> the previous group)
>>>>>> 7) Other errors
>>>>>>
>>>>>> Best regards,
>>>>>> Roman Kulyk
>>>>>>
>>>>>
>>>
>>>
>>
Re: Working on upgrade Drill Calcite version
Posted by Michael Mior <mm...@uwaterloo.ca>.
Fantastic news! Thanks to all who made this happen.
--
Michael Mior
mmior@apache.org
2018-01-19 5:43 GMT-05:00 Arina Ielchiieva <ar...@apache.org>:
> I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
> information can be found in DRILL-3993 and in Google Doc [1].
> Currently changes are in master and are intended to be part of Drill 1.13.0
> release unless something critical pops up. Feel free to test them and give
> your feedback.
>
> Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
> Aman Sinha and Chunhui Shi for the code review.
> And of course Julian Hyde for helping to address / review Calcite specific
> issues.
>
> [1]
> https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pu
> i7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
>
>
> Kind regards
> Arina
>
> On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
>
> > Team, thanks for your answers!
> >
> > Julian, сontinuing the topic of Calcite tests I just want to note that I
> > have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> > ignored tests:
> >
> > TraitConversionTest.testTraitConversion
> > JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> > JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> > JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> > LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTwoLattices
> >
> > You can see detailed information in my comments. Also, I want to notice
> > that there are 2 required commits in my Calcite branch: 5ba68240 and
> > 1f5145b1. Without them, we lose a big part of Drill functionality.
> >
> > Best regards,
> > Roman Kulyk
> >
> > 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
> >
> > > +dev@calcite.
> > >
> > > Excellent work, Roman.
> > >
> > > Your changes to Calcite don’t seem to be very major, and I ran the test
> > > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > > ready, I think we can expedite getting these changes back into Calcite.
> > >
> > > Calcite will release 1.14 probably in the next two weeks. So, when
> you’re
> > > on to 1.13 let’s continue the momentum and get up to the latest
> release.
> > I
> > > would love to get to the situation that we can ask Drill to verify a
> > > Calcite release before we make it. There are possibly breaking changes
> in
> > > the area of schema and grouping-sets aggregate but it should be mostly
> > > straightforward.
> > >
> > > Julian
> > >
> > >
> > > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > > >
> > > > It's great to hear that someone is working on moving Drill on new
> > Calcite
> > > > release! As someone who did similar job in last two rebase work, I
> > could
> > > > see that there would be many regressions to fix. Please let me know
> if
> > I
> > > > can provide any help.
> > > >
> > > > I thought we have pushed almost Drill specific commits to Calcite.
> I'll
> > > > take a look at your Calcite branch. Ideally, we want to get rid of
> all
> > > > Drill specific commits, after the rebase work this time.
> > > >
> > > > Thanks,
> > > >
> > > > Jinfeng
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > > wrote:
> > > >
> > > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > > needed,
> > > >> we can do a separate hangout with a few selected folks who have
> worked
> > > on
> > > >> Drill+Calcite to provide feedback on how to resolve the test
> failures.
> > > >> Although, at this stage you probably know a lot more about the
> > > integration
> > > >> than many of us.
> > > >>
> > > >> -Aman
> > > >>
> > > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi, team!
> > > >>>
> > > >>> I want to inform that I am working on updating Drill Calcite
> version
> > to
> > > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > > operators
> > > >>> when one of the arguments is ANY type. (was not in Calcite)
> > > >>> - Add new method to ViewExpander interface to allow passing
> > SchemaRoot.
> > > >>> (was not in Calcite)
> > > >>> - Allow a MAP literal type. (was not in Calcite)
> > > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > > >> options
> > > >>> in Drill (was not in Calcite)
> > > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> > ...
> > > >>> (small changes after a3bc0d8 commit)
> > > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> > are
> > > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > > changes
> > > >>> after a3bc0d8 commit)
> > > >>> - Support select * from schema-less table in execution engine like
> > > Drill
> > > >>> (small changes after 9bd7d75 commit)
> > > >>>
> > > >>> Only after adding these commits to Calcite, a part of Drill
> > > >> functionality
> > > >>> started to work as before. Currently, I am at Drill integration
> > stage.
> > > I
> > > >> am
> > > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > > >>>
> > > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > > CalciteForkRebase_rc1
> > > >>> Calcite:
> > > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > > >>> DrillCalcite1.13.0_rc1
> > > >>>
> > > >>> Currently, I face ~160 errors in java-exec module unit tests. I
> have
> > > >>> divided them into seven groups:
> > > >>> 1) SYSTEM ERROR: ClassCastException
> > > >>> 2) Unexpected column errors
> > > >>> 3) Runtime Exceptions
> > > >>> 4) validation errors
> > > >>> 5) RpcExceptions (generalized group: some of them should be fixed
> by
> > > the
> > > >>> previous groups)
> > > >>> 6) IllegalState errors (generalized group: some of them should be
> > fixed
> > > >> by
> > > >>> the previous group)
> > > >>> 7) Other errors
> > > >>>
> > > >>> Best regards,
> > > >>> Roman Kulyk
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Michael Mior <mm...@uwaterloo.ca>.
Fantastic news! Thanks to all who made this happen.
--
Michael Mior
mmior@apache.org
2018-01-19 5:43 GMT-05:00 Arina Ielchiieva <ar...@apache.org>:
> I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
> information can be found in DRILL-3993 and in Google Doc [1].
> Currently changes are in master and are intended to be part of Drill 1.13.0
> release unless something critical pops up. Feel free to test them and give
> your feedback.
>
> Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
> Aman Sinha and Chunhui Shi for the code review.
> And of course Julian Hyde for helping to address / review Calcite specific
> issues.
>
> [1]
> https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pu
> i7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
>
>
> Kind regards
> Arina
>
> On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
>
> > Team, thanks for your answers!
> >
> > Julian, сontinuing the topic of Calcite tests I just want to note that I
> > have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> > ignored tests:
> >
> > TraitConversionTest.testTraitConversion
> > JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> > JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> > JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> > LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> > LatticeTest.testTwoLattices
> >
> > You can see detailed information in my comments. Also, I want to notice
> > that there are 2 required commits in my Calcite branch: 5ba68240 and
> > 1f5145b1. Without them, we lose a big part of Drill functionality.
> >
> > Best regards,
> > Roman Kulyk
> >
> > 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
> >
> > > +dev@calcite.
> > >
> > > Excellent work, Roman.
> > >
> > > Your changes to Calcite don’t seem to be very major, and I ran the test
> > > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > > ready, I think we can expedite getting these changes back into Calcite.
> > >
> > > Calcite will release 1.14 probably in the next two weeks. So, when
> you’re
> > > on to 1.13 let’s continue the momentum and get up to the latest
> release.
> > I
> > > would love to get to the situation that we can ask Drill to verify a
> > > Calcite release before we make it. There are possibly breaking changes
> in
> > > the area of schema and grouping-sets aggregate but it should be mostly
> > > straightforward.
> > >
> > > Julian
> > >
> > >
> > > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > > >
> > > > It's great to hear that someone is working on moving Drill on new
> > Calcite
> > > > release! As someone who did similar job in last two rebase work, I
> > could
> > > > see that there would be many regressions to fix. Please let me know
> if
> > I
> > > > can provide any help.
> > > >
> > > > I thought we have pushed almost Drill specific commits to Calcite.
> I'll
> > > > take a look at your Calcite branch. Ideally, we want to get rid of
> all
> > > > Drill specific commits, after the rebase work this time.
> > > >
> > > > Thanks,
> > > >
> > > > Jinfeng
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > > wrote:
> > > >
> > > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > > needed,
> > > >> we can do a separate hangout with a few selected folks who have
> worked
> > > on
> > > >> Drill+Calcite to provide feedback on how to resolve the test
> failures.
> > > >> Although, at this stage you probably know a lot more about the
> > > integration
> > > >> than many of us.
> > > >>
> > > >> -Aman
> > > >>
> > > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi, team!
> > > >>>
> > > >>> I want to inform that I am working on updating Drill Calcite
> version
> > to
> > > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > > operators
> > > >>> when one of the arguments is ANY type. (was not in Calcite)
> > > >>> - Add new method to ViewExpander interface to allow passing
> > SchemaRoot.
> > > >>> (was not in Calcite)
> > > >>> - Allow a MAP literal type. (was not in Calcite)
> > > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > > >> options
> > > >>> in Drill (was not in Calcite)
> > > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> > ...
> > > >>> (small changes after a3bc0d8 commit)
> > > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> > are
> > > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > > changes
> > > >>> after a3bc0d8 commit)
> > > >>> - Support select * from schema-less table in execution engine like
> > > Drill
> > > >>> (small changes after 9bd7d75 commit)
> > > >>>
> > > >>> Only after adding these commits to Calcite, a part of Drill
> > > >> functionality
> > > >>> started to work as before. Currently, I am at Drill integration
> > stage.
> > > I
> > > >> am
> > > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > > >>>
> > > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > > CalciteForkRebase_rc1
> > > >>> Calcite:
> > > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > > >>> DrillCalcite1.13.0_rc1
> > > >>>
> > > >>> Currently, I face ~160 errors in java-exec module unit tests. I
> have
> > > >>> divided them into seven groups:
> > > >>> 1) SYSTEM ERROR: ClassCastException
> > > >>> 2) Unexpected column errors
> > > >>> 3) Runtime Exceptions
> > > >>> 4) validation errors
> > > >>> 5) RpcExceptions (generalized group: some of them should be fixed
> by
> > > the
> > > >>> previous groups)
> > > >>> 6) IllegalState errors (generalized group: some of them should be
> > fixed
> > > >> by
> > > >>> the previous group)
> > > >>> 7) Other errors
> > > >>>
> > > >>> Best regards,
> > > >>> Roman Kulyk
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Charles Givre <cg...@gmail.com>.
Great work!
> On Jan 19, 2018, at 05:43, Arina Ielchiieva <ar...@apache.org> wrote:
>
> I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
> information can be found in DRILL-3993 and in Google Doc [1].
> Currently changes are in master and are intended to be part of Drill 1.13.0
> release unless something critical pops up. Feel free to test them and give
> your feedback.
>
> Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
> Aman Sinha and Chunhui Shi for the code review.
> And of course Julian Hyde for helping to address / review Calcite specific
> issues.
>
> [1]
> https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pui7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
>
>
> Kind regards
> Arina
>
> On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
>
>> Team, thanks for your answers!
>>
>> Julian, сontinuing the topic of Calcite tests I just want to note that I
>> have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
>> ignored tests:
>>
>> TraitConversionTest.testTraitConversion
>> JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
>> JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
>> JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
>> LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
>> LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
>> LatticeTest.testTwoLattices
>>
>> You can see detailed information in my comments. Also, I want to notice
>> that there are 2 required commits in my Calcite branch: 5ba68240 and
>> 1f5145b1. Without them, we lose a big part of Drill functionality.
>>
>> Best regards,
>> Roman Kulyk
>>
>> 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
>>
>>> +dev@calcite.
>>>
>>> Excellent work, Roman.
>>>
>>> Your changes to Calcite don’t seem to be very major, and I ran the test
>>> suite on your https://github.com/KulykRoman/incubator-calcite/commits/
>>> DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
>>> ready, I think we can expedite getting these changes back into Calcite.
>>>
>>> Calcite will release 1.14 probably in the next two weeks. So, when you’re
>>> on to 1.13 let’s continue the momentum and get up to the latest release.
>> I
>>> would love to get to the situation that we can ask Drill to verify a
>>> Calcite release before we make it. There are possibly breaking changes in
>>> the area of schema and grouping-sets aggregate but it should be mostly
>>> straightforward.
>>>
>>> Julian
>>>
>>>
>>>> On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>
>>>> It's great to hear that someone is working on moving Drill on new
>> Calcite
>>>> release! As someone who did similar job in last two rebase work, I
>> could
>>>> see that there would be many regressions to fix. Please let me know if
>> I
>>>> can provide any help.
>>>>
>>>> I thought we have pushed almost Drill specific commits to Calcite. I'll
>>>> take a look at your Calcite branch. Ideally, we want to get rid of all
>>>> Drill specific commits, after the rebase work this time.
>>>>
>>>> Thanks,
>>>>
>>>> Jinfeng
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
>>> wrote:
>>>>
>>>>> Yes, this is long overdue ! Thanks for working on it Roman. If
>>> needed,
>>>>> we can do a separate hangout with a few selected folks who have worked
>>> on
>>>>> Drill+Calcite to provide feedback on how to resolve the test failures.
>>>>> Although, at this stage you probably know a lot more about the
>>> integration
>>>>> than many of us.
>>>>>
>>>>> -Aman
>>>>>
>>>>> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hi, team!
>>>>>>
>>>>>> I want to inform that I am working on updating Drill Calcite version
>> to
>>>>>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
>>>>>> - DRILL-1455: Add return type-inference strategy for arithmetic
>>> operators
>>>>>> when one of the arguments is ANY type. (was not in Calcite)
>>>>>> - Add new method to ViewExpander interface to allow passing
>> SchemaRoot.
>>>>>> (was not in Calcite)
>>>>>> - Allow a MAP literal type. (was not in Calcite)
>>>>>> - DRILL-4047: Modify table functions to allow querying a table with
>>>>> options
>>>>>> in Drill (was not in Calcite)
>>>>>> - Drill-specific change: Add back AbstractConverter in RelSet.java
>> ...
>>>>>> (small changes after a3bc0d8 commit)
>>>>>> - CALCITE-628 related but not fix the problem: Ensure target traits
>> are
>>>>>> simple when use Frameworks or RelOptRule.convert() method. (small
>>> changes
>>>>>> after a3bc0d8 commit)
>>>>>> - Support select * from schema-less table in execution engine like
>>> Drill
>>>>>> (small changes after 9bd7d75 commit)
>>>>>>
>>>>>> Only after adding these commits to Calcite, a part of Drill
>>>>> functionality
>>>>>> started to work as before. Currently, I am at Drill integration
>> stage.
>>> I
>>>>> am
>>>>>> fixing Drill unit tests. Progress can be tracked in my branches.
>>>>>>
>>>>>> Drill: https://github.com/KulykRoman/drill/commits/
>>> CalciteForkRebase_rc1
>>>>>> Calcite:
>>>>>> https://github.com/KulykRoman/incubator-calcite/commits/
>>>>>> DrillCalcite1.13.0_rc1
>>>>>>
>>>>>> Currently, I face ~160 errors in java-exec module unit tests. I have
>>>>>> divided them into seven groups:
>>>>>> 1) SYSTEM ERROR: ClassCastException
>>>>>> 2) Unexpected column errors
>>>>>> 3) Runtime Exceptions
>>>>>> 4) validation errors
>>>>>> 5) RpcExceptions (generalized group: some of them should be fixed by
>>> the
>>>>>> previous groups)
>>>>>> 6) IllegalState errors (generalized group: some of them should be
>> fixed
>>>>> by
>>>>>> the previous group)
>>>>>> 7) Other errors
>>>>>>
>>>>>> Best regards,
>>>>>> Roman Kulyk
>>>>>>
>>>>>
>>>
>>>
>>
Re: Working on upgrade Drill Calcite version
Posted by Arina Ielchiieva <ar...@apache.org>.
I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
information can be found in DRILL-3993 and in Google Doc [1].
Currently changes are in master and are intended to be part of Drill 1.13.0
release unless something critical pops up. Feel free to test them and give
your feedback.
Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
Aman Sinha and Chunhui Shi for the code review.
And of course Julian Hyde for helping to address / review Calcite specific
issues.
[1]
https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pui7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
Kind regards
Arina
On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
> Team, thanks for your answers!
>
> Julian, сontinuing the topic of Calcite tests I just want to note that I
> have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> ignored tests:
>
> TraitConversionTest.testTraitConversion
> JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> LatticeTest.testTwoLattices
>
> You can see detailed information in my comments. Also, I want to notice
> that there are 2 required commits in my Calcite branch: 5ba68240 and
> 1f5145b1. Without them, we lose a big part of Drill functionality.
>
> Best regards,
> Roman Kulyk
>
> 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
>
> > +dev@calcite.
> >
> > Excellent work, Roman.
> >
> > Your changes to Calcite don’t seem to be very major, and I ran the test
> > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > ready, I think we can expedite getting these changes back into Calcite.
> >
> > Calcite will release 1.14 probably in the next two weeks. So, when you’re
> > on to 1.13 let’s continue the momentum and get up to the latest release.
> I
> > would love to get to the situation that we can ask Drill to verify a
> > Calcite release before we make it. There are possibly breaking changes in
> > the area of schema and grouping-sets aggregate but it should be mostly
> > straightforward.
> >
> > Julian
> >
> >
> > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > >
> > > It's great to hear that someone is working on moving Drill on new
> Calcite
> > > release! As someone who did similar job in last two rebase work, I
> could
> > > see that there would be many regressions to fix. Please let me know if
> I
> > > can provide any help.
> > >
> > > I thought we have pushed almost Drill specific commits to Calcite. I'll
> > > take a look at your Calcite branch. Ideally, we want to get rid of all
> > > Drill specific commits, after the rebase work this time.
> > >
> > > Thanks,
> > >
> > > Jinfeng
> > >
> > >
> > >
> > >
> > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > wrote:
> > >
> > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > needed,
> > >> we can do a separate hangout with a few selected folks who have worked
> > on
> > >> Drill+Calcite to provide feedback on how to resolve the test failures.
> > >> Although, at this stage you probably know a lot more about the
> > integration
> > >> than many of us.
> > >>
> > >> -Aman
> > >>
> > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, team!
> > >>>
> > >>> I want to inform that I am working on updating Drill Calcite version
> to
> > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > operators
> > >>> when one of the arguments is ANY type. (was not in Calcite)
> > >>> - Add new method to ViewExpander interface to allow passing
> SchemaRoot.
> > >>> (was not in Calcite)
> > >>> - Allow a MAP literal type. (was not in Calcite)
> > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > >> options
> > >>> in Drill (was not in Calcite)
> > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> ...
> > >>> (small changes after a3bc0d8 commit)
> > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> are
> > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > changes
> > >>> after a3bc0d8 commit)
> > >>> - Support select * from schema-less table in execution engine like
> > Drill
> > >>> (small changes after 9bd7d75 commit)
> > >>>
> > >>> Only after adding these commits to Calcite, a part of Drill
> > >> functionality
> > >>> started to work as before. Currently, I am at Drill integration
> stage.
> > I
> > >> am
> > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > >>>
> > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > CalciteForkRebase_rc1
> > >>> Calcite:
> > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > >>> DrillCalcite1.13.0_rc1
> > >>>
> > >>> Currently, I face ~160 errors in java-exec module unit tests. I have
> > >>> divided them into seven groups:
> > >>> 1) SYSTEM ERROR: ClassCastException
> > >>> 2) Unexpected column errors
> > >>> 3) Runtime Exceptions
> > >>> 4) validation errors
> > >>> 5) RpcExceptions (generalized group: some of them should be fixed by
> > the
> > >>> previous groups)
> > >>> 6) IllegalState errors (generalized group: some of them should be
> fixed
> > >> by
> > >>> the previous group)
> > >>> 7) Other errors
> > >>>
> > >>> Best regards,
> > >>> Roman Kulyk
> > >>>
> > >>
> >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Arina Ielchiieva <ar...@apache.org>.
I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
information can be found in DRILL-3993 and in Google Doc [1].
Currently changes are in master and are intended to be part of Drill 1.13.0
release unless something critical pops up. Feel free to test them and give
your feedback.
Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
Aman Sinha and Chunhui Shi for the code review.
And of course Julian Hyde for helping to address / review Calcite specific
issues.
[1]
https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pui7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
Kind regards
Arina
On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
> Team, thanks for your answers!
>
> Julian, сontinuing the topic of Calcite tests I just want to note that I
> have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> ignored tests:
>
> TraitConversionTest.testTraitConversion
> JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> LatticeTest.testTwoLattices
>
> You can see detailed information in my comments. Also, I want to notice
> that there are 2 required commits in my Calcite branch: 5ba68240 and
> 1f5145b1. Without them, we lose a big part of Drill functionality.
>
> Best regards,
> Roman Kulyk
>
> 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
>
> > +dev@calcite.
> >
> > Excellent work, Roman.
> >
> > Your changes to Calcite don’t seem to be very major, and I ran the test
> > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > ready, I think we can expedite getting these changes back into Calcite.
> >
> > Calcite will release 1.14 probably in the next two weeks. So, when you’re
> > on to 1.13 let’s continue the momentum and get up to the latest release.
> I
> > would love to get to the situation that we can ask Drill to verify a
> > Calcite release before we make it. There are possibly breaking changes in
> > the area of schema and grouping-sets aggregate but it should be mostly
> > straightforward.
> >
> > Julian
> >
> >
> > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > >
> > > It's great to hear that someone is working on moving Drill on new
> Calcite
> > > release! As someone who did similar job in last two rebase work, I
> could
> > > see that there would be many regressions to fix. Please let me know if
> I
> > > can provide any help.
> > >
> > > I thought we have pushed almost Drill specific commits to Calcite. I'll
> > > take a look at your Calcite branch. Ideally, we want to get rid of all
> > > Drill specific commits, after the rebase work this time.
> > >
> > > Thanks,
> > >
> > > Jinfeng
> > >
> > >
> > >
> > >
> > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > wrote:
> > >
> > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > needed,
> > >> we can do a separate hangout with a few selected folks who have worked
> > on
> > >> Drill+Calcite to provide feedback on how to resolve the test failures.
> > >> Although, at this stage you probably know a lot more about the
> > integration
> > >> than many of us.
> > >>
> > >> -Aman
> > >>
> > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, team!
> > >>>
> > >>> I want to inform that I am working on updating Drill Calcite version
> to
> > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > operators
> > >>> when one of the arguments is ANY type. (was not in Calcite)
> > >>> - Add new method to ViewExpander interface to allow passing
> SchemaRoot.
> > >>> (was not in Calcite)
> > >>> - Allow a MAP literal type. (was not in Calcite)
> > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > >> options
> > >>> in Drill (was not in Calcite)
> > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> ...
> > >>> (small changes after a3bc0d8 commit)
> > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> are
> > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > changes
> > >>> after a3bc0d8 commit)
> > >>> - Support select * from schema-less table in execution engine like
> > Drill
> > >>> (small changes after 9bd7d75 commit)
> > >>>
> > >>> Only after adding these commits to Calcite, a part of Drill
> > >> functionality
> > >>> started to work as before. Currently, I am at Drill integration
> stage.
> > I
> > >> am
> > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > >>>
> > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > CalciteForkRebase_rc1
> > >>> Calcite:
> > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > >>> DrillCalcite1.13.0_rc1
> > >>>
> > >>> Currently, I face ~160 errors in java-exec module unit tests. I have
> > >>> divided them into seven groups:
> > >>> 1) SYSTEM ERROR: ClassCastException
> > >>> 2) Unexpected column errors
> > >>> 3) Runtime Exceptions
> > >>> 4) validation errors
> > >>> 5) RpcExceptions (generalized group: some of them should be fixed by
> > the
> > >>> previous groups)
> > >>> 6) IllegalState errors (generalized group: some of them should be
> fixed
> > >> by
> > >>> the previous group)
> > >>> 7) Other errors
> > >>>
> > >>> Best regards,
> > >>> Roman Kulyk
> > >>>
> > >>
> >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Arina Ielchiieva <ar...@apache.org>.
I am happy to announce that Apache Drill has rebased on Calcite 1.15. All
information can be found in DRILL-3993 and in Google Doc [1].
Currently changes are in master and are intended to be part of Drill 1.13.0
release unless something critical pops up. Feel free to test them and give
your feedback.
Thanks Roman Kulyk and Volodymyr Vysotskyi for making this happen.
Aman Sinha and Chunhui Shi for the code review.
And of course Julian Hyde for helping to address / review Calcite specific
issues.
[1]
https://docs.google.com/document/d/1Lqk9NoKQviz0YimBmov4z1pui7QjJGjDVwMa1p0emPk/edit#heading=h.i3rowg20vxv4
Kind regards
Arina
On Fri, Sep 1, 2017 at 8:01 PM, Roman Kulyk <ro...@gmail.com> wrote:
> Team, thanks for your answers!
>
> Julian, сontinuing the topic of Calcite tests I just want to note that I
> have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
> ignored tests:
>
> TraitConversionTest.testTraitConversion
> JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
> JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
> JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
> LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
> LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
> LatticeTest.testTwoLattices
>
> You can see detailed information in my comments. Also, I want to notice
> that there are 2 required commits in my Calcite branch: 5ba68240 and
> 1f5145b1. Without them, we lose a big part of Drill functionality.
>
> Best regards,
> Roman Kulyk
>
> 2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
>
> > +dev@calcite.
> >
> > Excellent work, Roman.
> >
> > Your changes to Calcite don’t seem to be very major, and I ran the test
> > suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> > DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> > ready, I think we can expedite getting these changes back into Calcite.
> >
> > Calcite will release 1.14 probably in the next two weeks. So, when you’re
> > on to 1.13 let’s continue the momentum and get up to the latest release.
> I
> > would love to get to the situation that we can ask Drill to verify a
> > Calcite release before we make it. There are possibly breaking changes in
> > the area of schema and grouping-sets aggregate but it should be mostly
> > straightforward.
> >
> > Julian
> >
> >
> > > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > >
> > > It's great to hear that someone is working on moving Drill on new
> Calcite
> > > release! As someone who did similar job in last two rebase work, I
> could
> > > see that there would be many regressions to fix. Please let me know if
> I
> > > can provide any help.
> > >
> > > I thought we have pushed almost Drill specific commits to Calcite. I'll
> > > take a look at your Calcite branch. Ideally, we want to get rid of all
> > > Drill specific commits, after the rebase work this time.
> > >
> > > Thanks,
> > >
> > > Jinfeng
> > >
> > >
> > >
> > >
> > > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> > wrote:
> > >
> > >> Yes, this is long overdue ! Thanks for working on it Roman. If
> > needed,
> > >> we can do a separate hangout with a few selected folks who have worked
> > on
> > >> Drill+Calcite to provide feedback on how to resolve the test failures.
> > >> Although, at this stage you probably know a lot more about the
> > integration
> > >> than many of us.
> > >>
> > >> -Aman
> > >>
> > >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, team!
> > >>>
> > >>> I want to inform that I am working on updating Drill Calcite version
> to
> > >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> > >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> > operators
> > >>> when one of the arguments is ANY type. (was not in Calcite)
> > >>> - Add new method to ViewExpander interface to allow passing
> SchemaRoot.
> > >>> (was not in Calcite)
> > >>> - Allow a MAP literal type. (was not in Calcite)
> > >>> - DRILL-4047: Modify table functions to allow querying a table with
> > >> options
> > >>> in Drill (was not in Calcite)
> > >>> - Drill-specific change: Add back AbstractConverter in RelSet.java
> ...
> > >>> (small changes after a3bc0d8 commit)
> > >>> - CALCITE-628 related but not fix the problem: Ensure target traits
> are
> > >>> simple when use Frameworks or RelOptRule.convert() method. (small
> > changes
> > >>> after a3bc0d8 commit)
> > >>> - Support select * from schema-less table in execution engine like
> > Drill
> > >>> (small changes after 9bd7d75 commit)
> > >>>
> > >>> Only after adding these commits to Calcite, a part of Drill
> > >> functionality
> > >>> started to work as before. Currently, I am at Drill integration
> stage.
> > I
> > >> am
> > >>> fixing Drill unit tests. Progress can be tracked in my branches.
> > >>>
> > >>> Drill: https://github.com/KulykRoman/drill/commits/
> > CalciteForkRebase_rc1
> > >>> Calcite:
> > >>> https://github.com/KulykRoman/incubator-calcite/commits/
> > >>> DrillCalcite1.13.0_rc1
> > >>>
> > >>> Currently, I face ~160 errors in java-exec module unit tests. I have
> > >>> divided them into seven groups:
> > >>> 1) SYSTEM ERROR: ClassCastException
> > >>> 2) Unexpected column errors
> > >>> 3) Runtime Exceptions
> > >>> 4) validation errors
> > >>> 5) RpcExceptions (generalized group: some of them should be fixed by
> > the
> > >>> previous groups)
> > >>> 6) IllegalState errors (generalized group: some of them should be
> fixed
> > >> by
> > >>> the previous group)
> > >>> 7) Other errors
> > >>>
> > >>> Best regards,
> > >>> Roman Kulyk
> > >>>
> > >>
> >
> >
>
Re: Working on upgrade Drill Calcite version
Posted by Roman Kulyk <ro...@gmail.com>.
Team, thanks for your answers!
Julian, сontinuing the topic of Calcite tests I just want to note that I
have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
ignored tests:
TraitConversionTest.testTraitConversion
JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
LatticeTest.testTwoLattices
You can see detailed information in my comments. Also, I want to notice
that there are 2 required commits in my Calcite branch: 5ba68240 and
1f5145b1. Without them, we lose a big part of Drill functionality.
Best regards,
Roman Kulyk
2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
> +dev@calcite.
>
> Excellent work, Roman.
>
> Your changes to Calcite don’t seem to be very major, and I ran the test
> suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> ready, I think we can expedite getting these changes back into Calcite.
>
> Calcite will release 1.14 probably in the next two weeks. So, when you’re
> on to 1.13 let’s continue the momentum and get up to the latest release. I
> would love to get to the situation that we can ask Drill to verify a
> Calcite release before we make it. There are possibly breaking changes in
> the area of schema and grouping-sets aggregate but it should be mostly
> straightforward.
>
> Julian
>
>
> > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> >
> > It's great to hear that someone is working on moving Drill on new Calcite
> > release! As someone who did similar job in last two rebase work, I could
> > see that there would be many regressions to fix. Please let me know if I
> > can provide any help.
> >
> > I thought we have pushed almost Drill specific commits to Calcite. I'll
> > take a look at your Calcite branch. Ideally, we want to get rid of all
> > Drill specific commits, after the rebase work this time.
> >
> > Thanks,
> >
> > Jinfeng
> >
> >
> >
> >
> > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> wrote:
> >
> >> Yes, this is long overdue ! Thanks for working on it Roman. If
> needed,
> >> we can do a separate hangout with a few selected folks who have worked
> on
> >> Drill+Calcite to provide feedback on how to resolve the test failures.
> >> Although, at this stage you probably know a lot more about the
> integration
> >> than many of us.
> >>
> >> -Aman
> >>
> >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> wrote:
> >>
> >>> Hi, team!
> >>>
> >>> I want to inform that I am working on updating Drill Calcite version to
> >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> operators
> >>> when one of the arguments is ANY type. (was not in Calcite)
> >>> - Add new method to ViewExpander interface to allow passing SchemaRoot.
> >>> (was not in Calcite)
> >>> - Allow a MAP literal type. (was not in Calcite)
> >>> - DRILL-4047: Modify table functions to allow querying a table with
> >> options
> >>> in Drill (was not in Calcite)
> >>> - Drill-specific change: Add back AbstractConverter in RelSet.java ...
> >>> (small changes after a3bc0d8 commit)
> >>> - CALCITE-628 related but not fix the problem: Ensure target traits are
> >>> simple when use Frameworks or RelOptRule.convert() method. (small
> changes
> >>> after a3bc0d8 commit)
> >>> - Support select * from schema-less table in execution engine like
> Drill
> >>> (small changes after 9bd7d75 commit)
> >>>
> >>> Only after adding these commits to Calcite, a part of Drill
> >> functionality
> >>> started to work as before. Currently, I am at Drill integration stage.
> I
> >> am
> >>> fixing Drill unit tests. Progress can be tracked in my branches.
> >>>
> >>> Drill: https://github.com/KulykRoman/drill/commits/
> CalciteForkRebase_rc1
> >>> Calcite:
> >>> https://github.com/KulykRoman/incubator-calcite/commits/
> >>> DrillCalcite1.13.0_rc1
> >>>
> >>> Currently, I face ~160 errors in java-exec module unit tests. I have
> >>> divided them into seven groups:
> >>> 1) SYSTEM ERROR: ClassCastException
> >>> 2) Unexpected column errors
> >>> 3) Runtime Exceptions
> >>> 4) validation errors
> >>> 5) RpcExceptions (generalized group: some of them should be fixed by
> the
> >>> previous groups)
> >>> 6) IllegalState errors (generalized group: some of them should be fixed
> >> by
> >>> the previous group)
> >>> 7) Other errors
> >>>
> >>> Best regards,
> >>> Roman Kulyk
> >>>
> >>
>
>
Re: Working on upgrade Drill Calcite version
Posted by Roman Kulyk <ro...@gmail.com>.
Team, thanks for your answers!
Julian, сontinuing the topic of Calcite tests I just want to note that I
have ignored 7 test in 5ba68240 commit in Calcite. Here is a list of
ignored tests:
TraitConversionTest.testTraitConversion
JdbcAdapterTest.testCrossJoinWithJoinKeyPlan
JdbcAdapterTest.testCrossJoinWithJoinKeyAndFilterPlan
JdbcTest.testJoinManyWay (it was also ignored by 8a87ad1e5)
LatticeTest.testTileAlgorithm (it was also ignored by 2e076e1a)
LatticeTest.testTileAlgorithm2 (it was also ignored by 8a87ad1e5)
LatticeTest.testTwoLattices
You can see detailed information in my comments. Also, I want to notice
that there are 2 required commits in my Calcite branch: 5ba68240 and
1f5145b1. Without them, we lose a big part of Drill functionality.
Best regards,
Roman Kulyk
2017-08-31 18:43 GMT+00:00 Julian Hyde <jh...@apache.org>:
> +dev@calcite.
>
> Excellent work, Roman.
>
> Your changes to Calcite don’t seem to be very major, and I ran the test
> suite on your https://github.com/KulykRoman/incubator-calcite/commits/
> DrillCalcite1.13.0_rc1 branch and all the tests pass. So when you are
> ready, I think we can expedite getting these changes back into Calcite.
>
> Calcite will release 1.14 probably in the next two weeks. So, when you’re
> on to 1.13 let’s continue the momentum and get up to the latest release. I
> would love to get to the situation that we can ask Drill to verify a
> Calcite release before we make it. There are possibly breaking changes in
> the area of schema and grouping-sets aggregate but it should be mostly
> straightforward.
>
> Julian
>
>
> > On Aug 31, 2017, at 8:12 AM, Jinfeng Ni <jn...@apache.org> wrote:
> >
> > It's great to hear that someone is working on moving Drill on new Calcite
> > release! As someone who did similar job in last two rebase work, I could
> > see that there would be many regressions to fix. Please let me know if I
> > can provide any help.
> >
> > I thought we have pushed almost Drill specific commits to Calcite. I'll
> > take a look at your Calcite branch. Ideally, we want to get rid of all
> > Drill specific commits, after the rebase work this time.
> >
> > Thanks,
> >
> > Jinfeng
> >
> >
> >
> >
> > On Thu, Aug 31, 2017 at 8:04 AM, Aman Sinha <am...@apache.org>
> wrote:
> >
> >> Yes, this is long overdue ! Thanks for working on it Roman. If
> needed,
> >> we can do a separate hangout with a few selected folks who have worked
> on
> >> Drill+Calcite to provide feedback on how to resolve the test failures.
> >> Although, at this stage you probably know a lot more about the
> integration
> >> than many of us.
> >>
> >> -Aman
> >>
> >> On Thu, Aug 31, 2017 at 7:33 AM, Roman Kulyk <ro...@gmail.com>
> wrote:
> >>
> >>> Hi, team!
> >>>
> >>> I want to inform that I am working on updating Drill Calcite version to
> >>> 1.13. I forked from Calcite "branch-1.13" and added some commits:
> >>> - DRILL-1455: Add return type-inference strategy for arithmetic
> operators
> >>> when one of the arguments is ANY type. (was not in Calcite)
> >>> - Add new method to ViewExpander interface to allow passing SchemaRoot.
> >>> (was not in Calcite)
> >>> - Allow a MAP literal type. (was not in Calcite)
> >>> - DRILL-4047: Modify table functions to allow querying a table with
> >> options
> >>> in Drill (was not in Calcite)
> >>> - Drill-specific change: Add back AbstractConverter in RelSet.java ...
> >>> (small changes after a3bc0d8 commit)
> >>> - CALCITE-628 related but not fix the problem: Ensure target traits are
> >>> simple when use Frameworks or RelOptRule.convert() method. (small
> changes
> >>> after a3bc0d8 commit)
> >>> - Support select * from schema-less table in execution engine like
> Drill
> >>> (small changes after 9bd7d75 commit)
> >>>
> >>> Only after adding these commits to Calcite, a part of Drill
> >> functionality
> >>> started to work as before. Currently, I am at Drill integration stage.
> I
> >> am
> >>> fixing Drill unit tests. Progress can be tracked in my branches.
> >>>
> >>> Drill: https://github.com/KulykRoman/drill/commits/
> CalciteForkRebase_rc1
> >>> Calcite:
> >>> https://github.com/KulykRoman/incubator-calcite/commits/
> >>> DrillCalcite1.13.0_rc1
> >>>
> >>> Currently, I face ~160 errors in java-exec module unit tests. I have
> >>> divided them into seven groups:
> >>> 1) SYSTEM ERROR: ClassCastException
> >>> 2) Unexpected column errors
> >>> 3) Runtime Exceptions
> >>> 4) validation errors
> >>> 5) RpcExceptions (generalized group: some of them should be fixed by
> the
> >>> previous groups)
> >>> 6) IllegalState errors (generalized group: some of them should be fixed
> >> by
> >>> the previous group)
> >>> 7) Other errors
> >>>
> >>> Best regards,
> >>> Roman Kulyk
> >>>
> >>
>
>