Fwd: [DISCUSS] fork Calcite and include the fork as a a submodule

[Julian Hyde <jh...@apache.org>]

Forwarding to dev@calcite (in bcc).

If anyone in Calcite wants to join the discussion, please reply to dev@kylin.

> Begin forwarded message:
> 
> From: hongbin ma <ma...@apache.org>
> Subject: [DISCUSS] fork Calcite and include the fork as a a submodule
> Date: April 16, 2017 at 7:39:12 AM PDT
> To: dev <de...@kylin.apache.org>, jhyde@apache.org
> 
> Recently I'm testing kylin connectivity with multiple BI tools like Tableau, Cognos, etc. During the test I find it necessary to fix several Calcite issues, like CALCITE-1754. I'm more than willing to contribute the fixes back to calcite, however there're still two potential issues:
> 
> 1. Calcite has it's own release cycles, sometimes we cannot afford to wait for calcite's next release
> 2. Some dirty hacks (yet still necessary) is not likely to be accepted by Calcite. Currently there's a weird sub-project called "AtopCalcite" in Kylin to host all the dirty hacks. 
> 
> With the above two issues, I'm wondering what is the best way to interact with Calcite releases. I'm suggesting that:
> 
> 1. We fork Apache Calcite and call it sth like calcite-for-kylin
> 2. Upon each calcite fix from our side, we double-commit to both Apache Calcite and calcite-for-kylin
> 3. For dirty hacks we only push code to calcite-for-kylin
> 4. calcite-for-kylin should be updated upon each Apache Calcite release
> 
> Any comment are welcomed!
> @Julian Looking forward to your comments as well
> 
> -- 
> Regards,
> 
> Bin Mahone | 马洪宾

Re: [DISCUSS] fork Calcite and include the fork as a a submodule

[hongbin ma <ma...@apache.org>]

Thank you Jacques for your experience, we're more than appreciated.

@Julian, the "Near Calcite Master" approach is exactly what I mean. (I have
been phrased "folk" better, as it is really a scary word).

Kylin does not have a fixed release cycle, so we cannot ask Calcite to fit
into our cycle. a "Near Calcite Master" folk could help us to solve
the "un-synced
release cycles" problem.

The "Near Calcite Master" folk will:

1. strictly minimize "dirty hacks", the goal is to maintain only one or two
additional patches for dirty hacks. The maintenance effort should equal to
the effort of maintaining "kylin/atopcalcite" today
2. temporally host commits pushed to calcite master but not yet released,
so that Kylin won't have to await calcite releases.





On Tue, Apr 18, 2017 at 3:02 AM, Julian Hyde <jh...@apache.org> wrote:

> Thanks for chiming in, Jacques. I was going to mention Drill as an example
> that the Kylin community could learn from.
>
> Also, thanks for starting this discussion, Hongbin Ma. Although “fork” is
> a scary word, it’s good to talk about development process so that we can
> find a process that works for everyone.
>
> I strongly recommend that you stay on (or very near) Calcite master. As
> Jacques says, when you get too far off master it’s really difficult to
> catch up[1].
>
> (“Very near” might mean that you make a release with one or two extra
> patches that you believe will land in Calcite master very shortly. It’s up
> to you to be disciplined and make sure that happens.)
>
> At Calcite we are committed to providing releases in a timely fashion. We
> *need* to do this because, as a framework, some of our major clients are
> Apache projects (Drill and Hive initially, but now including Kylin, Flink,
> Apex, Storm and several others). We *can* do this because we keep master
> branch stable. At a pinch, we could probably do a release from a standing
> start in a week (including a 3 day vote). But if you know you need a
> release by a particular date, we can accommodate that. Typically there is a
> discussion about timing a few weeks before a release, e.g. [2], and if you
> chime in, your voice will be heard.
>
> Remember that Calcite’s and Kylin’s interests are aligned. We want to have
> good support for Tableau just as much as you do. If you contribute a fix
> for Tableau, we are strongly motivated to accept it.
>
> That said, it does not benefit our community if we accept incomplete
> contributions (say, those without adequate test cases or documentation). We
> know from experience that once we accept crappy code, we own it. So, if you
> want to get a contribution accepted, make sure that it is complete and high
> quality, and be prepared to work with us to address our concerns. Then we
> will commit to keeping that feature (and in particular your tests) working
> in future releases.
>
> Lastly, JIRA cases and test cases are very welcome even without a fix. If
> you contribute a good test case, maybe someone else from the Calcite
> community will fix it. If you contribute a good suite of tests to Calcite’s
> suite for features that Kylin depends on, upgrading Kylin to a new release
> of Calcite will become easier.
>
> Julian
>
> [1] https://issues.apache.org/jira/browse/DRILL-3993 <
> https://issues.apache.org/jira/browse/DRILL-3993>
>
> [2] https://mail-archives.apache.org/mod_mbox/calcite-dev/
> 201612.mbox/%3C21C662E4-3C33-4633-AF34-728F6D84BA61%40apache.org%3E <
> https://mail-archives.apache.org/mod_mbox/calcite-dev/
> 201612.mbox/%3C21C662E4-3C33-4633-AF34-728F6D84BA61@apache.org%3E>
>
>
> > On Apr 17, 2017, at 10:52 AM, Jacques Nadeau <ja...@apache.org> wrote:
> >
> > I was probably the key person behind forking Calcite for Drill's purpose
> > (3+ years ago).  It was mostly driven by being unable to get key patches
> > into Calcite. It has become a mistake. I suggest you avoid making the
> same
> > one. Calcite is moving too quickly and you'll fall behind and be stuck in
> > fork-land.
> >
> > - If you can't get something merged and that is your impetus, please
> > elevate that discussion.
> > - If you need more frequent releases, you need to ask for them.
> >
> > For the second item, if you or someone else on Kylin becomes
> PMC/committer,
> > you can also get to the point where you are someone who can drive these
> > point releases.  Looking at history, no one in the Calcite community has
> > ever said "no, we can't do another release". Someone says "let's do a
> > release" and then we do one.
> >
> >
> >
> >>> From: hongbin ma <ma...@apache.org>
> >>> Subject: [DISCUSS] fork Calcite and include the fork as a a submodule
> >>> Date: April 16, 2017 at 7:39:12 AM PDT
> >>> To: dev <de...@kylin.apache.org>, jhyde@apache.org
> >>>
> >>> Recently I'm testing kylin connectivity with multiple BI tools like
> >> Tableau, Cognos, etc. During the test I find it necessary to fix several
> >> Calcite issues, like CALCITE-1754. I'm more than willing to contribute
> the
> >> fixes back to calcite, however there're still two potential issues:
> >>>
> >>> 1. Calcite has it's own release cycles, sometimes we cannot afford to
> >> wait for calcite's next release
> >>> 2. Some dirty hacks (yet still necessary) is not likely to be accepted
> >> by Calcite. Currently there's a weird sub-project called "AtopCalcite"
> in
> >> Kylin to host all the dirty hacks.
> >>>
> >>> With the above two issues, I'm wondering what is the best way to
> >> interact with Calcite releases. I'm suggesting that:
> >>>
> >>> 1. We fork Apache Calcite and call it sth like calcite-for-kylin
> >>> 2. Upon each calcite fix from our side, we double-commit to both Apache
> >> Calcite and calcite-for-kylin
> >>> 3. For dirty hacks we only push code to calcite-for-kylin
> >>> 4. calcite-for-kylin should be updated upon each Apache Calcite release
> >>>
> >>> Any comment are welcomed!
> >>> @Julian Looking forward to your comments as well
> >>>
> >>> --
> >>> Regards,
> >>>
> >>> Bin Mahone | 马洪宾
> >>
> >>
>
>


-- 
Regards,

*Bin Mahone | 马洪宾*

Re: [DISCUSS] fork Calcite and include the fork as a a submodule

[Julian Hyde <jh...@apache.org>]

Thanks for chiming in, Jacques. I was going to mention Drill as an example that the Kylin community could learn from.

Also, thanks for starting this discussion, Hongbin Ma. Although “fork” is a scary word, it’s good to talk about development process so that we can find a process that works for everyone.

I strongly recommend that you stay on (or very near) Calcite master. As Jacques says, when you get too far off master it’s really difficult to catch up[1].

(“Very near” might mean that you make a release with one or two extra patches that you believe will land in Calcite master very shortly. It’s up to you to be disciplined and make sure that happens.)

At Calcite we are committed to providing releases in a timely fashion. We *need* to do this because, as a framework, some of our major clients are Apache projects (Drill and Hive initially, but now including Kylin, Flink, Apex, Storm and several others). We *can* do this because we keep master branch stable. At a pinch, we could probably do a release from a standing start in a week (including a 3 day vote). But if you know you need a release by a particular date, we can accommodate that. Typically there is a discussion about timing a few weeks before a release, e.g. [2], and if you chime in, your voice will be heard.

Remember that Calcite’s and Kylin’s interests are aligned. We want to have good support for Tableau just as much as you do. If you contribute a fix for Tableau, we are strongly motivated to accept it.

That said, it does not benefit our community if we accept incomplete contributions (say, those without adequate test cases or documentation). We know from experience that once we accept crappy code, we own it. So, if you want to get a contribution accepted, make sure that it is complete and high quality, and be prepared to work with us to address our concerns. Then we will commit to keeping that feature (and in particular your tests) working in future releases.

Lastly, JIRA cases and test cases are very welcome even without a fix. If you contribute a good test case, maybe someone else from the Calcite community will fix it. If you contribute a good suite of tests to Calcite’s suite for features that Kylin depends on, upgrading Kylin to a new release of Calcite will become easier.

Julian

[1] https://issues.apache.org/jira/browse/DRILL-3993 <https://issues.apache.org/jira/browse/DRILL-3993> 

[2] https://mail-archives.apache.org/mod_mbox/calcite-dev/201612.mbox/%3C21C662E4-3C33-4633-AF34-728F6D84BA61%40apache.org%3E <https://mail-archives.apache.org/mod_mbox/calcite-dev/201612.mbox/%3C21C662E4-3C33-4633-AF34-728F6D84BA61@apache.org%3E> 


> On Apr 17, 2017, at 10:52 AM, Jacques Nadeau <ja...@apache.org> wrote:
> 
> I was probably the key person behind forking Calcite for Drill's purpose
> (3+ years ago).  It was mostly driven by being unable to get key patches
> into Calcite. It has become a mistake. I suggest you avoid making the same
> one. Calcite is moving too quickly and you'll fall behind and be stuck in
> fork-land.
> 
> - If you can't get something merged and that is your impetus, please
> elevate that discussion.
> - If you need more frequent releases, you need to ask for them.
> 
> For the second item, if you or someone else on Kylin becomes PMC/committer,
> you can also get to the point where you are someone who can drive these
> point releases.  Looking at history, no one in the Calcite community has
> ever said "no, we can't do another release". Someone says "let's do a
> release" and then we do one.
> 
> 
> 
>>> From: hongbin ma <ma...@apache.org>
>>> Subject: [DISCUSS] fork Calcite and include the fork as a a submodule
>>> Date: April 16, 2017 at 7:39:12 AM PDT
>>> To: dev <de...@kylin.apache.org>, jhyde@apache.org
>>> 
>>> Recently I'm testing kylin connectivity with multiple BI tools like
>> Tableau, Cognos, etc. During the test I find it necessary to fix several
>> Calcite issues, like CALCITE-1754. I'm more than willing to contribute the
>> fixes back to calcite, however there're still two potential issues:
>>> 
>>> 1. Calcite has it's own release cycles, sometimes we cannot afford to
>> wait for calcite's next release
>>> 2. Some dirty hacks (yet still necessary) is not likely to be accepted
>> by Calcite. Currently there's a weird sub-project called "AtopCalcite" in
>> Kylin to host all the dirty hacks.
>>> 
>>> With the above two issues, I'm wondering what is the best way to
>> interact with Calcite releases. I'm suggesting that:
>>> 
>>> 1. We fork Apache Calcite and call it sth like calcite-for-kylin
>>> 2. Upon each calcite fix from our side, we double-commit to both Apache
>> Calcite and calcite-for-kylin
>>> 3. For dirty hacks we only push code to calcite-for-kylin
>>> 4. calcite-for-kylin should be updated upon each Apache Calcite release
>>> 
>>> Any comment are welcomed!
>>> @Julian Looking forward to your comments as well
>>> 
>>> --
>>> Regards,
>>> 
>>> Bin Mahone | 马洪宾
>> 
>> 

Re: [DISCUSS] fork Calcite and include the fork as a a submodule

[Jacques Nadeau <ja...@apache.org>]

I was probably the key person behind forking Calcite for Drill's purpose
(3+ years ago).  It was mostly driven by being unable to get key patches
into Calcite. It has become a mistake. I suggest you avoid making the same
one. Calcite is moving too quickly and you'll fall behind and be stuck in
fork-land.

- If you can't get something merged and that is your impetus, please
elevate that discussion.
- If you need more frequent releases, you need to ask for them.

For the second item, if you or someone else on Kylin becomes PMC/committer,
you can also get to the point where you are someone who can drive these
point releases.  Looking at history, no one in the Calcite community has
ever said "no, we can't do another release". Someone says "let's do a
release" and then we do one.



> > From: hongbin ma <ma...@apache.org>
> > Subject: [DISCUSS] fork Calcite and include the fork as a a submodule
> > Date: April 16, 2017 at 7:39:12 AM PDT
> > To: dev <de...@kylin.apache.org>, jhyde@apache.org
> >
> > Recently I'm testing kylin connectivity with multiple BI tools like
> Tableau, Cognos, etc. During the test I find it necessary to fix several
> Calcite issues, like CALCITE-1754. I'm more than willing to contribute the
> fixes back to calcite, however there're still two potential issues:
> >
> > 1. Calcite has it's own release cycles, sometimes we cannot afford to
> wait for calcite's next release
> > 2. Some dirty hacks (yet still necessary) is not likely to be accepted
> by Calcite. Currently there's a weird sub-project called "AtopCalcite" in
> Kylin to host all the dirty hacks.
> >
> > With the above two issues, I'm wondering what is the best way to
> interact with Calcite releases. I'm suggesting that:
> >
> > 1. We fork Apache Calcite and call it sth like calcite-for-kylin
> > 2. Upon each calcite fix from our side, we double-commit to both Apache
> Calcite and calcite-for-kylin
> > 3. For dirty hacks we only push code to calcite-for-kylin
> > 4. calcite-for-kylin should be updated upon each Apache Calcite release
> >
> > Any comment are welcomed!
> > @Julian Looking forward to your comments as well
> >
> > --
> > Regards,
> >
> > Bin Mahone | 马洪宾
>
>