You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kris Mok (JIRA)" <ji...@apache.org> on 2017/05/24 20:17:04 UTC
[jira] [Updated] (SPARK-20872) ShuffleExchange.nodeName should
handle null coordinator
[ https://issues.apache.org/jira/browse/SPARK-20872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kris Mok updated SPARK-20872:
-----------------------------
Description:
A ShuffleExchange's coordinator can be null sometimes, and when we need to do a toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError there because of inexhaustive match -- the match only handles Some and None, but doesn't handle null.
An example of this issue is when trying to inspect a Catalyst physical operator on the Executor side in an IDE:
{code:none}
child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError' exception. Cannot evaluate org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
{code}
where this WholeStageCodegenExec transitively references a ShuffleExchange.
On the Executor side, this ShuffleExchange instance is deserialized from the data sent over from the Driver, and because the coordinator field is marked transient, it's not carried over to the Executor, that's why it can be null upon inspection.
was:
A ShuffleExchange's coordinator can be null sometimes, and when we need to do a toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError there because of inexhaustive match -- the match only handles Some and None, but doesn't handle null.
An example of this issue is when trying to inspect a Catalyst physical operator in an IDE:
{code:none}
child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError' exception. Cannot evaluate org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
{code}
where this WholeStageCodegenExec transitively references a ShuffleExchange.
> ShuffleExchange.nodeName should handle null coordinator
> -------------------------------------------------------
>
> Key: SPARK-20872
> URL: https://issues.apache.org/jira/browse/SPARK-20872
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Kris Mok
> Priority: Trivial
> Labels: easyfix
>
> A ShuffleExchange's coordinator can be null sometimes, and when we need to do a toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError there because of inexhaustive match -- the match only handles Some and None, but doesn't handle null.
> An example of this issue is when trying to inspect a Catalyst physical operator on the Executor side in an IDE:
> {code:none}
> child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError' exception. Cannot evaluate org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
> {code}
> where this WholeStageCodegenExec transitively references a ShuffleExchange.
> On the Executor side, this ShuffleExchange instance is deserialized from the data sent over from the Driver, and because the coordinator field is marked transient, it's not carried over to the Executor, that's why it can be null upon inspection.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org