You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Gupta, Shuchi" <Sh...@ca.com> on 2016/12/19 06:24:49 UTC
Pushing order by to source
Dear Calcite Developers
I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
[cid:image002.jpg@01D259EE.AB05FAB0]
My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
Could you please suggest a way out.
Your suggestions in this regard would be highly appreciated.
Thanks and Regards
Shuchi Gupta
Software Engineer (CA)
CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
Office: +71262 | Shuchi.Gupta2@ca.com
<ma...@ca.com>[CA]<http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[Slideshare]<http://www.slideshare.net/cainc>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Google+]<https://plus.google.com/+CATechnologies>[Google+]<http://www.ca.com/us/rss.aspx?intcmp=footernav>
Re: Pushing order by to source
Posted by "Gupta, Shuchi" <Sh...@ca.com>.
Hey Jesus
Thanks for your response.
Yes, the sorting for queries having time series or group by clause are pushed downstream.
I face issues when the query only has order by(in which case, the queryType is 'select') .. somehow in the RelOptCluster I am able to get the sort order from original root, but the column on which it is to be sorted was not there.
My query goes something like this :
SELECT col1, col2 FROM table1 ORDER BY col1 LIMIT 10;
Regards.
Shuchi
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Jesus Camacho Rodriguez <jc...@hortonworks.com> on behalf of Jesus Camacho Rodriguez <jc...@apache.org>
Sent: Monday, December 19, 2016 9:37:47 PM
To: dev@calcite.apache.org
Subject: Re: Pushing order by to source
Hi Suchi,
Order by-limit is already pushed to Druid for some kind of queries (CALCITE-1357). Maybe you just need an extension to that code, as it does not cover all cases. Sharing the specific query / Calcite plan would help us to know what we are missing.
Thanks,
Jesús
On 12/19/16, 9:43 AM, "Gupta, Shuchi" <Sh...@ca.com> wrote:
>I could see that if order by is present in the query, Calcite is handling this sorting on its own. Can you tell me the class where this sorting is happening.
>
>I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept) method. But while debugging it appears this code is not being used.
>
>-----Original Message-----
>From: Julian Hyde [mailto:jhyde@apache.org]
>Sent: Monday, December 19, 2016 12:22 PM
>To: dev@calcite.apache.org
>Subject: Re: Pushing order by to source
>
>The solution is simple — someone needs to fix CALCITE-1206. Contributions welcome.
>
>> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>
>> Hi Julian
>>
>> The image is a clip from : https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
>>
>> "That plan shows that Calcite was able to push down the GROUP BY part of the query to Druid, including the COUNT(*) function, but not the ORDER BY ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
>>
>> -----Original Message-----
>> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
>> Sent: Monday, December 19, 2016 12:07 PM
>> To: dev@calcite.apache.org <ma...@calcite.apache.org>
>> Subject: Re: Pushing order by to source
>>
>> The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
>>
>>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>>
>>> Dear Calcite Developers
>>>
>>> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>>>
>>>
>>>
>>>
>>> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
>>> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>>>
>>> Could you please suggest a way out.
>>> Your suggestions in this regard would be highly appreciated.
>>>
>>> Thanks and Regards
>>> Shuchi Gupta
>>> Software Engineer (CA)
>>>
>>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
>>> Office: +71262 | Shuchi.Gupta2@ca.com
>>>
>>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.aspx?intcmp=footernav>>
>
Re: Pushing order by to source
Posted by Jesus Camacho Rodriguez <jc...@apache.org>.
Hi Suchi,
Order by-limit is already pushed to Druid for some kind of queries (CALCITE-1357). Maybe you just need an extension to that code, as it does not cover all cases. Sharing the specific query / Calcite plan would help us to know what we are missing.
Thanks,
Jesús
On 12/19/16, 9:43 AM, "Gupta, Shuchi" <Sh...@ca.com> wrote:
>I could see that if order by is present in the query, Calcite is handling this sorting on its own. Can you tell me the class where this sorting is happening.
>
>I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept) method. But while debugging it appears this code is not being used.
>
>-----Original Message-----
>From: Julian Hyde [mailto:jhyde@apache.org]
>Sent: Monday, December 19, 2016 12:22 PM
>To: dev@calcite.apache.org
>Subject: Re: Pushing order by to source
>
>The solution is simple — someone needs to fix CALCITE-1206. Contributions welcome.
>
>> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>
>> Hi Julian
>>
>> The image is a clip from : https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
>>
>> "That plan shows that Calcite was able to push down the GROUP BY part of the query to Druid, including the COUNT(*) function, but not the ORDER BY ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
>>
>> -----Original Message-----
>> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
>> Sent: Monday, December 19, 2016 12:07 PM
>> To: dev@calcite.apache.org <ma...@calcite.apache.org>
>> Subject: Re: Pushing order by to source
>>
>> The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
>>
>>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>>
>>> Dear Calcite Developers
>>>
>>> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>>>
>>>
>>>
>>>
>>> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
>>> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>>>
>>> Could you please suggest a way out.
>>> Your suggestions in this regard would be highly appreciated.
>>>
>>> Thanks and Regards
>>> Shuchi Gupta
>>> Software Engineer (CA)
>>>
>>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
>>> Office: +71262 | Shuchi.Gupta2@ca.com
>>>
>>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.aspx?intcmp=footernav>>
>
Re: Pushing order by to source
Posted by Gian Merlino <gi...@imply.io>.
I think the main difference between the two is that Druid's built-in SQL
uses Druid classes internally to model things, rather than just sticking to
the HTTP API like the Druid rules in Calcite. If you think it makes sense
for Calcite's Druid adapter to depend on Druid jars to get those classes
too, then I think it makes sense to collaborate to try to keep the rule
sets in sync. Possibly by having Calcite depend on druid-sql and get the
rules that way, or vice-versa by having Druid depend on calcite-druid. But
if not, I think close collaboration would be tough since rules would need
to be rewritten to be ported from Calcite to Druid or vice versa.
Gian
On Mon, Dec 19, 2016 at 11:40 AM, Julian Hyde <jh...@apache.org> wrote:
> Gian,
>
> It’s off-topic for this thread, but I wanted to ask whether you have any
> plans to keep the Calcite rules in Druid in sync with the Druid rules in
> Calcite (the “druid adapter”).
>
> My goal is to enable two modes of operation: (a) native SQL for Druid, (b)
> hybrid systems that combine Druid rules with rules for other stores (e.g.
> Hbase) and other processing engines (e.g. Drill, Spark, Hive). We have
> something of a fork right now, and that’s not bad per se, but it will be
> bad if one of the copies of the code starts to fall behind in functionality.
>
> Julian
>
>
> > On Dec 19, 2016, at 10:57 AM, Gian Merlino <gi...@imply.io> wrote:
> >
> > Fwiw, Druid has a built-in set of Calcite rules now, and it does support
> > pushing down order by and limit. It can even re-order the grouping
> > dimensions for better performance, and choose timeseries, topN, or
> groupBy
> > as appropriate. Depending on what you're trying to do, this may work for
> > you. The code is in master, but not yet in a release.
> >
> > Gian
> >
> > On Mon, Dec 19, 2016 at 1:43 AM, Gupta, Shuchi <Sh...@ca.com>
> wrote:
> >
> >> I could see that if order by is present in the query, Calcite is
> handling
> >> this sorting on its own. Can you tell me the class where this sorting is
> >> happening.
> >>
> >> I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept)
> >> method. But while debugging it appears this code is not being used.
> >>
> >> -----Original Message-----
> >> From: Julian Hyde [mailto:jhyde@apache.org]
> >> Sent: Monday, December 19, 2016 12:22 PM
> >> To: dev@calcite.apache.org
> >> Subject: Re: Pushing order by to source
> >>
> >> The solution is simple — someone needs to fix CALCITE-1206.
> Contributions
> >> welcome.
> >>
> >>> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com>
> >> wrote:
> >>>
> >>> Hi Julian
> >>>
> >>> The image is a clip from : https://urldefense.proofpoint.
> >> com/v2/url?u=https-3A__calcite.apache.org_docs_druid-
> >> 5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
> >> bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.
> >> proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_
> >> druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
> >> bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
> >>>
> >>> "That plan shows that Calcite was able to push down the GROUP BY part
> of
> >> the query to Druid, including the COUNT(*) function, but not the ORDER
> BY
> >> ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
> >>>
> >>> -----Original Message-----
> >>> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
> >>> Sent: Monday, December 19, 2016 12:07 PM
> >>> To: dev@calcite.apache.org <ma...@calcite.apache.org>
> >>> Subject: Re: Pushing order by to source
> >>>
> >>> The image didn’t come through in your email. Could you convert to text,
> >> or post an image hosted elsewhere?
> >>>
> >>>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com>
> >> wrote:
> >>>>
> >>>> Dear Calcite Developers
> >>>>
> >>>> I am referring to Calcite-Druid and could see an open issue to support
> >> pushdown order by.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> My custom database(based on elastic search) will return 100 records by
> >> default at a time, unless a limit is specified.
> >>>> Here, for a use case with order by and limit in place, limit
> definitely
> >> would have to be pushed to source. Of course, order by needs to be
> applied
> >> before limit, we need to pass it to source too.
> >>>>
> >>>> Could you please suggest a way out.
> >>>> Your suggestions in this regard would be highly appreciated.
> >>>>
> >>>> Thanks and Regards
> >>>> Shuchi Gupta
> >>>> Software Engineer (CA)
> >>>>
> >>>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> >>>> Office: +71262 | Shuchi.Gupta2@ca.com
> >>>>
> >>>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <
> >> http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>>
> <
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.
> com_CAInc&d=
> >> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> >> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> >> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.
> com_CAInc&d=
> >> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> >> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> >> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <
> >> https://urldefense.proofpoint.com/v2/url?u=http-
> >> 3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-
> >> bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.
> >> proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_
> >> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> >> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> >> hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_
> >> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.
> >> proofpoint.com/v2/url?u=https-3A__www.facebook.com_
> >> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.
> >> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
> >> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.
> >> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
> >> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.
> >> proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_
> >> 1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_
> >> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> >> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> >> hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <
> >> https://urldefense.proofpoint.com/v2/url?u=http-
> >> 3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-
> >> 5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_
> >> 4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.
> >> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
> >> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.
> >> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
> >> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> >> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> >> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> >> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <
> >> http://www.ca.com/us/rss.aspx?intcmp=footernav <
> http://www.ca.com/us/rss.
> >> aspx?intcmp=footernav>>
> >>
> >>
>
>
Re: Pushing order by to source
Posted by Julian Hyde <jh...@apache.org>.
Gian,
It’s off-topic for this thread, but I wanted to ask whether you have any plans to keep the Calcite rules in Druid in sync with the Druid rules in Calcite (the “druid adapter”).
My goal is to enable two modes of operation: (a) native SQL for Druid, (b) hybrid systems that combine Druid rules with rules for other stores (e.g. Hbase) and other processing engines (e.g. Drill, Spark, Hive). We have something of a fork right now, and that’s not bad per se, but it will be bad if one of the copies of the code starts to fall behind in functionality.
Julian
> On Dec 19, 2016, at 10:57 AM, Gian Merlino <gi...@imply.io> wrote:
>
> Fwiw, Druid has a built-in set of Calcite rules now, and it does support
> pushing down order by and limit. It can even re-order the grouping
> dimensions for better performance, and choose timeseries, topN, or groupBy
> as appropriate. Depending on what you're trying to do, this may work for
> you. The code is in master, but not yet in a release.
>
> Gian
>
> On Mon, Dec 19, 2016 at 1:43 AM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
>> I could see that if order by is present in the query, Calcite is handling
>> this sorting on its own. Can you tell me the class where this sorting is
>> happening.
>>
>> I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept)
>> method. But while debugging it appears this code is not being used.
>>
>> -----Original Message-----
>> From: Julian Hyde [mailto:jhyde@apache.org]
>> Sent: Monday, December 19, 2016 12:22 PM
>> To: dev@calcite.apache.org
>> Subject: Re: Pushing order by to source
>>
>> The solution is simple — someone needs to fix CALCITE-1206. Contributions
>> welcome.
>>
>>> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com>
>> wrote:
>>>
>>> Hi Julian
>>>
>>> The image is a clip from : https://urldefense.proofpoint.
>> com/v2/url?u=https-3A__calcite.apache.org_docs_druid-
>> 5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
>> bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.
>> proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_
>> druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
>> bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
>>>
>>> "That plan shows that Calcite was able to push down the GROUP BY part of
>> the query to Druid, including the COUNT(*) function, but not the ORDER BY
>> ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
>>>
>>> -----Original Message-----
>>> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
>>> Sent: Monday, December 19, 2016 12:07 PM
>>> To: dev@calcite.apache.org <ma...@calcite.apache.org>
>>> Subject: Re: Pushing order by to source
>>>
>>> The image didn’t come through in your email. Could you convert to text,
>> or post an image hosted elsewhere?
>>>
>>>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com>
>> wrote:
>>>>
>>>> Dear Calcite Developers
>>>>
>>>> I am referring to Calcite-Druid and could see an open issue to support
>> pushdown order by.
>>>>
>>>>
>>>>
>>>>
>>>> My custom database(based on elastic search) will return 100 records by
>> default at a time, unless a limit is specified.
>>>> Here, for a use case with order by and limit in place, limit definitely
>> would have to be pushed to source. Of course, order by needs to be applied
>> before limit, we need to pass it to source too.
>>>>
>>>> Could you please suggest a way out.
>>>> Your suggestions in this regard would be highly appreciated.
>>>>
>>>> Thanks and Regards
>>>> Shuchi Gupta
>>>> Software Engineer (CA)
>>>>
>>>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
>>>> Office: +71262 | Shuchi.Gupta2@ca.com
>>>>
>>>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <
>> http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=
>> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
>> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
>> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=
>> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
>> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
>> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <
>> https://urldefense.proofpoint.com/v2/url?u=http-
>> 3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-
>> bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.
>> proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_
>> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
>> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
>> hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_
>> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.
>> proofpoint.com/v2/url?u=https-3A__www.facebook.com_
>> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.
>> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
>> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.
>> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
>> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.
>> proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_
>> 1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_
>> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
>> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
>> hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <
>> https://urldefense.proofpoint.com/v2/url?u=http-
>> 3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-
>> 5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_
>> 4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.
>> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
>> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.
>> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
>> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
>> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
>> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
>> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <
>> http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.
>> aspx?intcmp=footernav>>
>>
>>
Re: Pushing order by to source
Posted by Gian Merlino <gi...@imply.io>.
Fwiw, Druid has a built-in set of Calcite rules now, and it does support
pushing down order by and limit. It can even re-order the grouping
dimensions for better performance, and choose timeseries, topN, or groupBy
as appropriate. Depending on what you're trying to do, this may work for
you. The code is in master, but not yet in a release.
Gian
On Mon, Dec 19, 2016 at 1:43 AM, Gupta, Shuchi <Sh...@ca.com> wrote:
> I could see that if order by is present in the query, Calcite is handling
> this sorting on its own. Can you tell me the class where this sorting is
> happening.
>
> I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept)
> method. But while debugging it appears this code is not being used.
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org]
> Sent: Monday, December 19, 2016 12:22 PM
> To: dev@calcite.apache.org
> Subject: Re: Pushing order by to source
>
> The solution is simple — someone needs to fix CALCITE-1206. Contributions
> welcome.
>
> > On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com>
> wrote:
> >
> > Hi Julian
> >
> > The image is a clip from : https://urldefense.proofpoint.
> com/v2/url?u=https-3A__calcite.apache.org_docs_druid-
> 5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
> bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_
> druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_
> bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
> >
> > "That plan shows that Calcite was able to push down the GROUP BY part of
> the query to Druid, including the COUNT(*) function, but not the ORDER BY
> ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
> >
> > -----Original Message-----
> > From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
> > Sent: Monday, December 19, 2016 12:07 PM
> > To: dev@calcite.apache.org <ma...@calcite.apache.org>
> > Subject: Re: Pushing order by to source
> >
> > The image didn’t come through in your email. Could you convert to text,
> or post an image hosted elsewhere?
> >
> >> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com>
> wrote:
> >>
> >> Dear Calcite Developers
> >>
> >> I am referring to Calcite-Druid and could see an open issue to support
> pushdown order by.
> >>
> >>
> >>
> >>
> >> My custom database(based on elastic search) will return 100 records by
> default at a time, unless a limit is specified.
> >> Here, for a use case with order by and limit in place, limit definitely
> would have to be pushed to source. Of course, order by needs to be applied
> before limit, we need to pass it to source too.
> >>
> >> Could you please suggest a way out.
> >> Your suggestions in this regard would be highly appreciated.
> >>
> >> Thanks and Regards
> >> Shuchi Gupta
> >> Software Engineer (CA)
> >>
> >> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> >> Office: +71262 | Shuchi.Gupta2@ca.com
> >>
> >> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <
> http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=
> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=
> DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-
> bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.
> proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_
> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_
> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__www.facebook.com_
> CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.
> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.
> proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_
> catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.
> proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_
> 1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_
> hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_
> BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-
> hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-
> 5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_
> 4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.
> proofpoint.com/v2/url?u=https-3A__plus.google.com_-
> 2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxf
> kkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=
> AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=
> Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <
> http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.
> aspx?intcmp=footernav>>
>
>
RE: Pushing order by to source
Posted by "Gupta, Shuchi" <Sh...@ca.com>.
I could see that if order by is present in the query, Calcite is handling this sorting on its own. Can you tell me the class where this sorting is happening.
I tried to debug in SqlParserImpl class , in the OrderBy(boolean accept) method. But while debugging it appears this code is not being used.
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Monday, December 19, 2016 12:22 PM
To: dev@calcite.apache.org
Subject: Re: Pushing order by to source
The solution is simple — someone needs to fix CALCITE-1206. Contributions welcome.
> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Hi Julian
>
> The image is a clip from : https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__calcite.apache.org_docs_druid-5Fadapter.html&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=nESklzzFnQyTkJm40O_uL3ZyMWQfnBNuQyU_AZBBNhs&s=D3wrMo9-XzMRmZl1OI_bGEQncVEtyaDWyD4MgJNrrXQ&e= >which discusses issue : [CALCITE-1206]
>
> "That plan shows that Calcite was able to push down the GROUP BY part of the query to Druid, including the COUNT(*) function, but not the ORDER BY ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
> Sent: Monday, December 19, 2016 12:07 PM
> To: dev@calcite.apache.org <ma...@calcite.apache.org>
> Subject: Re: Pushing order by to source
>
> The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
>
>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>
>> Dear Calcite Developers
>>
>> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>>
>>
>>
>>
>> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
>> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>>
>> Could you please suggest a way out.
>> Your suggestions in this regard would be highly appreciated.
>>
>> Thanks and Regards
>> Shuchi Gupta
>> Software Engineer (CA)
>>
>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
>> Office: +71262 | Shuchi.Gupta2@ca.com
>>
>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.aspx?intcmp=footernav>>
Re: Pushing order by to source
Posted by Julian Hyde <jh...@apache.org>.
The solution is simple — someone needs to fix CALCITE-1206. Contributions welcome.
> On Dec 18, 2016, at 10:45 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Hi Julian
>
> The image is a clip from : https://calcite.apache.org/docs/druid_adapter.html <https://calcite.apache.org/docs/druid_adapter.html>which discusses issue : [CALCITE-1206]
>
> "That plan shows that Calcite was able to push down the GROUP BY part of the query to Druid, including the COUNT(*) function, but not the ORDER BY ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org <ma...@apache.org>]
> Sent: Monday, December 19, 2016 12:07 PM
> To: dev@calcite.apache.org <ma...@calcite.apache.org>
> Subject: Re: Pushing order by to source
>
> The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
>
>> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>>
>> Dear Calcite Developers
>>
>> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>>
>>
>>
>>
>> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
>> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>>
>> Could you please suggest a way out.
>> Your suggestions in this regard would be highly appreciated.
>>
>> Thanks and Regards
>> Shuchi Gupta
>> Software Engineer (CA)
>>
>> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
>> Office: +71262 | Shuchi.Gupta2@ca.com
>>
>> <mailto:Shuchi.Gupta2@ca.com <ma...@ca.com>> <http://www.ca.com/us/default.aspx <http://www.ca.com/us/default.aspx>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e=> > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e=> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e=> > <http://www.ca.com/us/rss.aspx?intcmp=footernav <http://www.ca.com/us/rss.aspx?intcmp=footernav>>
RE: Pushing order by to source
Posted by "Gupta, Shuchi" <Sh...@ca.com>.
Hi Julian
The image is a clip from : https://calcite.apache.org/docs/druid_adapter.html which discusses issue : [CALCITE-1206]
"That plan shows that Calcite was able to push down the GROUP BY part of the query to Druid, including the COUNT(*) function, but not the ORDER BY ... LIMIT. (We plan to lift this restriction; see [CALCITE-1206].)"
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Monday, December 19, 2016 12:07 PM
To: dev@calcite.apache.org
Subject: Re: Pushing order by to source
The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Dear Calcite Developers
>
> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>
>
>
>
> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>
> Could you please suggest a way out.
> Your suggestions in this regard would be highly appreciated.
>
> Thanks and Regards
> Shuchi Gupta
> Software Engineer (CA)
>
> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> Office: +71262 | Shuchi.Gupta2@ca.com
>
> <ma...@ca.com> <http://www.ca.com/us/default.aspx> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=NKfWiWlYFTnagSsZ3eHKWOzBNoQkTpcO-8OdlF2nhfI&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=0PSRyw5KQS6JsRWn9-bIrM4uoep1OPfJNE7v5FL8-5I&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=d4IxMhqg2zJ7qb3TujVJen6cJPr7mifND9ibWnD-JmA&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=JPbD4L92sDXh3qOx83yFKMR2xMhX2x9_fMlI-aY16L4&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Y5qwUboGHA_4Em2wQlcmcsvQrCQPLPb5ubMlUqwqIBw&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=AbKoujKDKHK07fU7vUzCTVq1jh-hu70PQgQ6btO15h0&s=Aet4z5ajwvD5r7049sMVcnxREu4ET88HmWE1I7sYaRg&e= > <http://www.ca.com/us/rss.aspx?intcmp=footernav>
Re: Pushing order by to source
Posted by Julian Hyde <jh...@apache.org>.
The image didn’t come through in your email. Could you convert to text, or post an image hosted elsewhere?
> On Dec 18, 2016, at 10:24 PM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Dear Calcite Developers
>
> I am referring to Calcite-Druid and could see an open issue to support pushdown order by.
>
>
>
>
> My custom database(based on elastic search) will return 100 records by default at a time, unless a limit is specified.
> Here, for a use case with order by and limit in place, limit definitely would have to be pushed to source. Of course, order by needs to be applied before limit, we need to pass it to source too.
>
> Could you please suggest a way out.
> Your suggestions in this regard would be highly appreciated.
>
> Thanks and Regards
> Shuchi Gupta
> Software Engineer (CA)
>
> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> Office: +71262 | Shuchi.Gupta2@ca.com
>
> <ma...@ca.com> <http://www.ca.com/us/default.aspx> <http://twitter.com/CAInc> <http://www.slideshare.net/cainc> <https://www.facebook.com/CATechnologies> <http://www.youtube.com/user/catechnologies> <http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1> <https://plus.google.com/+CATechnologies> <http://www.ca.com/us/rss.aspx?intcmp=footernav>