[storage engine plugin] How to push down sort/order by to the storage engine?

[Tugdual Grall <tu...@gmail.com>]

Hello,

I would like to push down the ORDER BY to the MongoDB database itself ( see
https://issues.apache.org/jira/browse/DRILL-3165 )

I have looked into the Optimizer, Record Reader and Scan, I do not see how
I can get the "Order by" clause information from the context.

I see how the filters and projection are used.

Any clue?


PS: another side question about Projection & MongoDB:
   Any reason why the _id is by default removed from the "select * from
collection" ?



Regards
Tug
@tgrall

Re: [storage engine plugin] How to push down sort/order by to the storage engine?

[Tugdual Grall <tu...@gmail.com>]

>
> Hi Tugdual,
>

Thanks for the response


>  The only reason for removing _id field from select * is, in most of the
> cases value of _id field generated by Mongo system and users normally do
> not use.
>

It is not true, it is by default generated and use the timestamp that is
part of the ObjectID in many use case (this give me an idea on an UDF for
mongo)

Also developer can set it when needed, and since it is used as PK it is
often use for lookup in apps ( it is true in the context of analytics it is
less useful).




> However, if users select _id field (cases where users using _id as key)
> drill returns its value.
>
>
So yes I can do a projection
select _id ....
but
select _id, *  ... does not work

so I  need to specify the full list of field. (Will see where it is coming
from)

Anyway it is not a blocking issue at all since we can access it, I was just
surprised that it was removed/hidden by default.

regards
Tug


>
> On Thu, Jul 16, 2015 at 5:53 PM, Tugdual Grall <tu...@gmail.com> wrote:
>
>> Hello,
>>
>> I would like to push down the ORDER BY to the MongoDB database itself (
>> see
>> https://issues.apache.org/jira/browse/DRILL-3165 )
>>
>> I have looked into the Optimizer, Record Reader and Scan, I do not see how
>> I can get the "Order by" clause information from the context.
>>
>> I see how the filters and projection are used.
>>
>> Any clue?
>>
>>
>> PS: another side question about Projection & MongoDB:
>>    Any reason why the _id is by default removed from the "select * from
>> collection" ?
>>
>>
>>
>> Regards
>> Tug
>> @tgrall
>>
>
>
>
> --
> Kamesh.
>

Re: [storage engine plugin] How to push down sort/order by to the storage engine?

[Kamesh <ka...@gmail.com>]

Hi Tugdual,
 The only reason for removing _id field from select * is, in most of the
cases value of _id field generated by Mongo system and users normally do
not use. However, if users select _id field (cases where users using _id as
key) drill returns its value.



On Thu, Jul 16, 2015 at 5:53 PM, Tugdual Grall <tu...@gmail.com> wrote:

> Hello,
>
> I would like to push down the ORDER BY to the MongoDB database itself ( see
> https://issues.apache.org/jira/browse/DRILL-3165 )
>
> I have looked into the Optimizer, Record Reader and Scan, I do not see how
> I can get the "Order by" clause information from the context.
>
> I see how the filters and projection are used.
>
> Any clue?
>
>
> PS: another side question about Projection & MongoDB:
>    Any reason why the _id is by default removed from the "select * from
> collection" ?
>
>
>
> Regards
> Tug
> @tgrall
>



-- 
Kamesh.