You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takeshi Yamamuro (JIRA)" <ji...@apache.org> on 2016/04/22 11:25:13 UTC
[jira] [Comment Edited] (SPARK-14820) Reduce shuffle data by
pushing filter toward storage
[ https://issues.apache.org/jira/browse/SPARK-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253608#comment-15253608 ]
Takeshi Yamamuro edited comment on SPARK-14820 at 4/22/16 9:24 AM:
-------------------------------------------------------------------
Seems `Optimizer#PushPredicateThroughJoin` handles this kind of push-down optimization.
Why cannot the current impl. apply filter push-down into the query described in your pdf?
was (Author: maropu):
Seems `Optimizer#PushPredicateThroughJoin` handles this kind of push-down optimization.
Why cannot the current impl. apply filter push-downs into the query described in your pdf?
> Reduce shuffle data by pushing filter toward storage
> ----------------------------------------------------
>
> Key: SPARK-14820
> URL: https://issues.apache.org/jira/browse/SPARK-14820
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.1
> Reporter: Ali Tootoonchian
> Priority: Trivial
> Attachments: Reduce Shuffle Data by pushing filter toward storage.pdf
>
>
> SQL query planner can have intelligence to push down filter commands towards the storage layer. If we optimize the query planner such that the IO to the storage is reduced at the cost of running multiple filters (i.e., compute), this should be desirable when the system is IO bound.
> Proven analysis and example is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org