You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "gu-chi (JIRA)" <ji...@apache.org> on 2015/04/09 14:06:12 UTC
[jira] [Commented] (SPARK-1203) spark-shell on yarn-client race in
properly getting hdfs delegation tokens
[ https://issues.apache.org/jira/browse/SPARK-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487262#comment-14487262 ]
gu-chi commented on SPARK-1203:
-------------------------------
Seems I meet with the similar issue using saveAsNewAPIHadoopDataset
I tried to reproduce with create SC and then sleep for 10mins, then do saveAsNewAPIHadoopDataset, still not able to reproduce, can u give some advice on how to reproduce, thx
> spark-shell on yarn-client race in properly getting hdfs delegation tokens
> --------------------------------------------------------------------------
>
> Key: SPARK-1203
> URL: https://issues.apache.org/jira/browse/SPARK-1203
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.9.1, 1.0.0
>
>
> There seems to be a race when using the spark-shell on yarn (yarn-client mode) as to when it gets the hdfs delegation tokens.
> It appears to be that if you do an action that causes it to get a delegation token before the workers are launched things work fine, if you want until after the workers launch you get an error about not having a delegation token.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org