You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Xuefu Zhang (JIRA)" <ji...@apache.org> on 2015/03/23 21:16:53 UTC
[jira] [Comment Edited] (HIVE-10058) Log the information of cached
RDD [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376552#comment-14376552 ]
Xuefu Zhang edited comment on HIVE-10058 at 3/23/15 8:16 PM:
-------------------------------------------------------------
[~chinnalalam], Sorry that I wasn't clear. I was thinking more on HIVE-8858, where I'd like to have visual representation of SparkPlan. As you can see in the class definition, A SparkPlan consists of a graph of SparkTrans. SparkTran has a few subclasses. Some subclass, such as MapInput has property such as "toCache". What is desirable is that we log a SparkPlan in a graphical way similar to what's show for work graph in explain plan, such as:
{code}
MapInput (cache off) -> Shuffle (cache on) -> Reduce
\-> Reduce
{code}
This is will give us some idea about SparkPlan that we are executing. Let me know if you have any questions.
was (Author: xuefuz):
[~chinnalalam], Sorry that I wasn't clear. I was thinking more on HIVE-8858, where I'd like to have visual representation of SparkPlan. As you can see in the class definition, A SparkPlan consists of a graph of SparkTrans. SparkTran has a few subclasses. Some subclass, such as MapInput has property such as "toCache". What is desirable is that we log a SparkPlan in a graphical way similar to what's show for work graph in explain plan, such as:
{code}
MapInput (cache off) -> Shuffle (cache on) -> Reduce
\-> Reduce
{code}
This is will give us some idea about SparkPlan that we are executing. Let me know if you have any questions.
> Log the information of cached RDD [Spark Branch]
> ------------------------------------------------
>
> Key: HIVE-10058
> URL: https://issues.apache.org/jira/browse/HIVE-10058
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chinna Rao Lalam
> Assignee: Chinna Rao Lalam
> Fix For: spark-branch
>
> Attachments: HIVE-10058.1-spark.patch, HIVE-10058.2-spark.patch
>
>
> Log the cached RDD Id's at info level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)