[jira] [Commented] (CALCITE-1234) Annotate table functions to allow pushing down project, filter

["Michael Mior (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279248#comment-15279248 ] 

Michael Mior commented on CALCITE-1234:
---------------------------------------

Here's a few references you may want to take a look at

SOFA: An Extensible Logical Optimizer for UDF-heavy Dataflows
http://arxiv.org/abs/1311.6335
This is in the context of map-reduce code, but SOFA makes heavy use of annotations and some of what they're doing may apply in a relational setting.

ANNOTATIONS FOR PARALLELIZATION OF USER-DEFINED FUNCTIONS WITH FLEXIBLE PARTITIONING
http://www.freepatentsonline.com/y2015/0379076.html
This is a patent application and IANAL, so not sure if anything is usable here. I'm also not really sure if Calcite takes advantage of any opportunities for intra-query parallelism.

Query Optimization in the Presence of Foreign Functions 
http://www.vldb.org/conf/1993/P529.PDF
This is an old one and maybe a bit too high-level to be useful but talks about query optimization in the context of user-specified rewrite rules for UDFs.

> Annotate table functions to allow pushing down project, filter
> --------------------------------------------------------------
>
>                 Key: CALCITE-1234
>                 URL: https://issues.apache.org/jira/browse/CALCITE-1234
>             Project: Calcite
>          Issue Type: Bug
>            Reporter: Julian Hyde
>            Assignee: Julian Hyde
>
> In general it is not possible to push relational operators through table functions but many table functions have properties that will allow push down: for instance, they might preserve fields, preserve row count, return rows in the same order. If we annotate a table function with these properties, we can automatically push down a Filter, and so forth.
> Some ideas:
> * {{PreservesFieldNames}}: If an output field has the same name as an input field, it is assumed to be the same field.
> * {{PreservesRows}}: Each input row causes exactly one output row, in the same order.
> * {{FiltersRows}}: Each input row causes at most one output row, in the same order.
> * {{PreservesFieldPositions}}: The leading N columns of the output are equivalent to the leading N columns of the input.
> If {{(PreservesFieldNames or PreservesFieldPositions) and (PreservesRows or FiltersRows)}}, it is safe to push down a Filter if all of the fields of the predicate exist in the input.
> Similarly pushing down a Project; and we can push down an Aggregate if {{PreservesRows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)