Why Drill required a special Calcite fork ?

[Muhammad Gelbana <m....@gmail.com>]

Would someone please list the reasons for which Drill required having a
custom Calcite build ? This will help integrating Calcite whenever a
release is out.

One reason I can assume is that Calcite didn't support unparsing all SQL
clauses such as FETCH and OFFSET ?

I for my self, very much need to use the latest Calcite version with Drill
and I'm willing to spend time working on that, if it's possible for my poor
Drill and Calcite knowledge.

​-
Gelbana

Re: Why Drill required a special Calcite fork ?

[Parth Chandra <pa...@apache.org>]

+1 for getting off the Calcite and Parquet forks.

On Mon, Jul 17, 2017 at 4:01 PM, Jinfeng Ni <jn...@apache.org> wrote:

> Getting Drill off the Calcite fork is clearly the best solution.  From what
> I know [1], people have spend quite lots of effort to push the patches in
> Drill's fork to Calcite master. The next step is to put Drill on top of
> Calcite master or latest release, and resolve regressions.  RomanK spent
> some time to resolve regressions, but I have not heard latest update from
> him. It would be great that the community work together to get it done
> asap. (probably same thing applies to Drill's parquet fork).
>
> Maintaining a Calcite fork is not a long term solution (We have established
> the policy that always push the changes to Calcite master, before
> cherry-picking back). But before we could eventually get off the fork, we
> probably still have to keep it somewhere.  The fork used to be hosted in
> two companies' repository, but looks like one of them stopped hosting the
> fork. The repository was forked probably before Calcite graduated from
> incubation,  and poorly named. We need correct the document/name as Julian
> suggested.
>
> Regarding to where the fork should be hosted, I'm not sure if other Apache
> projects have similar situations, and how they deal with that. Anyone have
> any idea?
>
> 1.
> https://docs.google.com/spreadsheets/d/1Z0J5aMCBfqq_9SEl-LXsi_
> B8Ahaosygqke4cH6pPwmo/edit#gid=0
>
> On Mon, Jul 17, 2017 at 11:47 AM, Julian Hyde <jh...@apache.org> wrote:
>
> > Leaving aside the fact that Drill needs a fork of Calcite (I accept the
> > arguments for that, more or less), it’s embarrassing that the fork is
> > poorly documented, poorly named (Calcite has been out of the incubator
> for
> > almost 2 years, and hasn’t been called Optiq for each longer), and is in
> a
> > particular company’s repository (thereby creating the impression that
> > contributions from non-employees would not be accepted).
> >
> > Also, has there been a release? My understanding is that the IP is on
> > shaky ground unless there is a release (not necessarily one under the
> ASF).
> >
> > What can we do to clean up this situation?
> >
> > Julian
> >
> >
> > > On Jul 17, 2017, at 9:00 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> > >
> > > Hi Muhammad,
> > >
> > > Link - https://github.com/mapr/incubator-calcite/tree/
> DrillCalcite1.4.0
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Mon, Jul 17, 2017 at 6:52 PM, Muhammad Gelbana <m.gelbana@gmail.com
> >
> > > wrote:
> > >
> > >> Could anyone at least just provide a link for that fork's repository
> > please
> > >> ?
> > >>
> > >>
> > >>
> > >> - Gelbana
> > >>
> > >> On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana <
> m.gelbana@gmail.com>
> > >> wrote:
> > >>
> > >>> Would someone please list the reasons for which Drill required
> having a
> > >>> custom Calcite build ? This will help integrating Calcite whenever a
> > >>> release is out.
> > >>>
> > >>> One reason I can assume is that Calcite didn't support unparsing all
> > SQL
> > >>> clauses such as FETCH and OFFSET ?
> > >>>
> > >>> I for my self, very much need to use the latest Calcite version with
> > >> Drill
> > >>> and I'm willing to spend time working on that, if it's possible for
> my
> > >> poor
> > >>> Drill and Calcite knowledge.
> > >>>
> > >>> ​-
> > >>> Gelbana
> > >>>
> > >>
> >
> >
>

Re: Why Drill required a special Calcite fork ?

[Jinfeng Ni <jn...@apache.org>]

Getting Drill off the Calcite fork is clearly the best solution.  From what
I know [1], people have spend quite lots of effort to push the patches in
Drill's fork to Calcite master. The next step is to put Drill on top of
Calcite master or latest release, and resolve regressions.  RomanK spent
some time to resolve regressions, but I have not heard latest update from
him. It would be great that the community work together to get it done
asap. (probably same thing applies to Drill's parquet fork).

Maintaining a Calcite fork is not a long term solution (We have established
the policy that always push the changes to Calcite master, before
cherry-picking back). But before we could eventually get off the fork, we
probably still have to keep it somewhere.  The fork used to be hosted in
two companies' repository, but looks like one of them stopped hosting the
fork. The repository was forked probably before Calcite graduated from
incubation,  and poorly named. We need correct the document/name as Julian
suggested.

Regarding to where the fork should be hosted, I'm not sure if other Apache
projects have similar situations, and how they deal with that. Anyone have
any idea?

1.
https://docs.google.com/spreadsheets/d/1Z0J5aMCBfqq_9SEl-LXsi_B8Ahaosygqke4cH6pPwmo/edit#gid=0

On Mon, Jul 17, 2017 at 11:47 AM, Julian Hyde <jh...@apache.org> wrote:

> Leaving aside the fact that Drill needs a fork of Calcite (I accept the
> arguments for that, more or less), it’s embarrassing that the fork is
> poorly documented, poorly named (Calcite has been out of the incubator for
> almost 2 years, and hasn’t been called Optiq for each longer), and is in a
> particular company’s repository (thereby creating the impression that
> contributions from non-employees would not be accepted).
>
> Also, has there been a release? My understanding is that the IP is on
> shaky ground unless there is a release (not necessarily one under the ASF).
>
> What can we do to clean up this situation?
>
> Julian
>
>
> > On Jul 17, 2017, at 9:00 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
> >
> > Hi Muhammad,
> >
> > Link - https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0
> >
> > Kind regards
> > Arina
> >
> > On Mon, Jul 17, 2017 at 6:52 PM, Muhammad Gelbana <m....@gmail.com>
> > wrote:
> >
> >> Could anyone at least just provide a link for that fork's repository
> please
> >> ?
> >>
> >>
> >>
> >> - Gelbana
> >>
> >> On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana <m....@gmail.com>
> >> wrote:
> >>
> >>> Would someone please list the reasons for which Drill required having a
> >>> custom Calcite build ? This will help integrating Calcite whenever a
> >>> release is out.
> >>>
> >>> One reason I can assume is that Calcite didn't support unparsing all
> SQL
> >>> clauses such as FETCH and OFFSET ?
> >>>
> >>> I for my self, very much need to use the latest Calcite version with
> >> Drill
> >>> and I'm willing to spend time working on that, if it's possible for my
> >> poor
> >>> Drill and Calcite knowledge.
> >>>
> >>> ​-
> >>> Gelbana
> >>>
> >>
>
>

Re: Why Drill required a special Calcite fork ?

[Julian Hyde <jh...@apache.org>]

Leaving aside the fact that Drill needs a fork of Calcite (I accept the arguments for that, more or less), it’s embarrassing that the fork is poorly documented, poorly named (Calcite has been out of the incubator for almost 2 years, and hasn’t been called Optiq for each longer), and is in a particular company’s repository (thereby creating the impression that contributions from non-employees would not be accepted).

Also, has there been a release? My understanding is that the IP is on shaky ground unless there is a release (not necessarily one under the ASF).

What can we do to clean up this situation?

Julian
 

> On Jul 17, 2017, at 9:00 AM, Arina Yelchiyeva <ar...@gmail.com> wrote:
> 
> Hi Muhammad,
> 
> Link - https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0
> 
> Kind regards
> Arina
> 
> On Mon, Jul 17, 2017 at 6:52 PM, Muhammad Gelbana <m....@gmail.com>
> wrote:
> 
>> Could anyone at least just provide a link for that fork's repository please
>> ?
>> 
>> 
>> 
>> - Gelbana
>> 
>> On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana <m....@gmail.com>
>> wrote:
>> 
>>> Would someone please list the reasons for which Drill required having a
>>> custom Calcite build ? This will help integrating Calcite whenever a
>>> release is out.
>>> 
>>> One reason I can assume is that Calcite didn't support unparsing all SQL
>>> clauses such as FETCH and OFFSET ?
>>> 
>>> I for my self, very much need to use the latest Calcite version with
>> Drill
>>> and I'm willing to spend time working on that, if it's possible for my
>> poor
>>> Drill and Calcite knowledge.
>>> 
>>> ​-
>>> Gelbana
>>> 
>> 

Re: Why Drill required a special Calcite fork ?

[Arina Yelchiyeva <ar...@gmail.com>]

Hi Muhammad,

Link - https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0

Kind regards
Arina

On Mon, Jul 17, 2017 at 6:52 PM, Muhammad Gelbana <m....@gmail.com>
wrote:

> Could anyone at least just provide a link for that fork's repository please
> ?
>
>
>
> - Gelbana
>
> On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana <m....@gmail.com>
> wrote:
>
> > Would someone please list the reasons for which Drill required having a
> > custom Calcite build ? This will help integrating Calcite whenever a
> > release is out.
> >
> > One reason I can assume is that Calcite didn't support unparsing all SQL
> > clauses such as FETCH and OFFSET ?
> >
> > I for my self, very much need to use the latest Calcite version with
> Drill
> > and I'm willing to spend time working on that, if it's possible for my
> poor
> > Drill and Calcite knowledge.
> >
> > ​-
> > Gelbana
> >
>

Re: Why Drill required a special Calcite fork ?

[Muhammad Gelbana <m....@gmail.com>]

Could anyone at least just provide a link for that fork's repository please
?



- Gelbana

On Sat, Jul 1, 2017 at 12:02 AM, Muhammad Gelbana <m....@gmail.com>
wrote:

> Would someone please list the reasons for which Drill required having a
> custom Calcite build ? This will help integrating Calcite whenever a
> release is out.
>
> One reason I can assume is that Calcite didn't support unparsing all SQL
> clauses such as FETCH and OFFSET ?
>
> I for my self, very much need to use the latest Calcite version with Drill
> and I'm willing to spend time working on that, if it's possible for my poor
> Drill and Calcite knowledge.
>
> ​-
> Gelbana
>