[jira] [Commented] (SPARK-4849) Pass partitioning information (distribute by) to In-memory caching

["Eric Pederson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594193#comment-14594193 ] 

Eric Pederson commented on SPARK-4849:
--------------------------------------

Does this also apply to in-memory tables created as the result of cached partitioned hive tables?

For example, say {{hivetable}} is partitioned by {{(s string, c string)}}.

{code}
val sql = new HiveContext(sc)
val t = sql.table("hivetable")
val c1 = t.cached()

val f1 = c1.filter("s = 'FNM30' and c = '3.0'")
val s1 = f1.groupBy("g").sum("a", "b", "c")
{code}

Should it be able to prune parts {{c1}} because of the original paritioning?

> Pass partitioning information (distribute by) to In-memory caching
> ------------------------------------------------------------------
>
>                 Key: SPARK-4849
>                 URL: https://issues.apache.org/jira/browse/SPARK-4849
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.2.0
>            Reporter: Nitin Goyal
>            Priority: Minor
>
> HQL "distribute by <column_name>" partitions data based on specified column values. We can pass this information to in-memory caching for further performance improvements. e..g. in Joins, an extra partition step can be saved based on this information.
> Refer - http://apache-spark-user-list.1001560.n3.nabble.com/SchemaRDD-partition-on-specific-column-values-td20350.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org