[jira] [Commented] (SPARK-25190) Better operator pushdown API

["Ryan Blue (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589070#comment-16589070 ] 

Ryan Blue commented on SPARK-25190:
-----------------------------------

The main problem I have with the current pushdown API is that Spark gets information back from pushdown before all pushdown is finished. I like the idea to have an immutable ScanConfig that is the result of pushdown operations, so it is clear that pushdown calls are finished before getting a reader factory or asking for statistics.

Unlike those methods that accept a ScanConfig, using {{SupportsPushDownXYZ}} for pushdown means that Spark is getting the results of pushdown operations back before all pushdown is complete. That means that the same confusion over pushdown order still exists, although the problem is fixed for some operations. I think that all feedback from pushdown should come from the ScanConfig.

> Better operator pushdown API
> ----------------------------
>
>                 Key: SPARK-25190
>                 URL: https://issues.apache.org/jira/browse/SPARK-25190
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Wenchen Fan
>            Priority: Major
>
> The current operator pushdown API is a little hard to use. It defines several {{SupportsPushdownXYZ}} interfaces and ask the implementation to be mutable to store the pushdown result. We should design a builder like API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org