You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ali Tootoonchian (JIRA)" <ji...@apache.org> on 2016/04/21 22:30:13 UTC

[jira] [Updated] (SPARK-14820) Reduce shuffle data by pushing filter toward storage

     [ https://issues.apache.org/jira/browse/SPARK-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ali Tootoonchian updated SPARK-14820:
-------------------------------------
    Attachment: Reduce Shuffle Data by pushing filter toward storage.pdf

> Reduce shuffle data by pushing filter toward storage
> ----------------------------------------------------
>
>                 Key: SPARK-14820
>                 URL: https://issues.apache.org/jira/browse/SPARK-14820
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Ali Tootoonchian
>            Priority: Trivial
>         Attachments: Reduce Shuffle Data by pushing filter toward storage.pdf
>
>
> SQL query planner can have intelligence to push down filter commands towards the storage layer. If we optimize the query planner such that the IO to the storage is reduced at the cost of running multiple filters (i.e., compute), this should be desirable when the system is IO bound. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org