You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Markus Weimer (JIRA)" <ji...@apache.org> on 2015/05/27 18:21:17 UTC
[jira] [Commented] (REEF-348) Job localization error caused by
incorrect timestamp on HDInsight
[ https://issues.apache.org/jira/browse/REEF-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561212#comment-14561212 ]
Markus Weimer commented on REEF-348:
------------------------------------
This is the main error, right?
{noformat}
ResourceLocalizationService: DEBUG: FAILED {Resource wasb://provisioning-hdi-default@eus2prodsa01150413.blob.core.windows.net/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar changed on src filesystem (expected 1432177412000, was 1432177417000
{noformat}
This is quite odd, as we upload this JAR from the driver upon launch, and refer to it over and over again. It shouldn't change at all. Does this happen immediately after you start your job or after some time?
> Job localization error caused by incorrect timestamp on HDInsight
> -----------------------------------------------------------------
>
> Key: REEF-348
> URL: https://issues.apache.org/jira/browse/REEF-348
> Project: REEF
> Issue Type: Bug
> Components: REEF-Runtime-HDInsight, REEF.NET
> Environment: REEF.NET on HDInsight
> Reporter: Andrew Chung
>
> The timestamp is set incorrectly when uploading global.jar to blob, so the resource could not be successfully localized. Below are nodemanager logs:
> {noformat}
> 2015-05-21 03:04:16,391 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from c:/apps/temp/hdfs/nm-local-dir/nmPrivate/container_1432083697923_0483_01_000024.tokens to c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483/container_1432083697923_0483_01_000024.tokens
> 2015-05-21 03:04:16,391 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483 = file:/c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483
> 2015-05-21 03:04:16,737 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource wasb://provisioning-hdi-default@eus2prodsa01150413.blob.core.windows.net/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/reef-evaluator-3911340348394906440.jar(->file:/c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483/filecache/22/reef-evaluator-3911340348394906440.jar) transitioned from DOWNLOADING to LOCALIZED
> 2015-05-21 03:04:16,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { wasb://provisioning-hdi-default@eus2prodsa01150413.blob.core.windows.net/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar, 1432177412000, ARCHIVE, null }, Resource wasb://provisioning-hdi-default@eus2prodsa01150413.blob.core.windows.net/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar changed on src filesystem (expected 1432177412000, was 1432177417000
> 2015-05-21 03:04:16,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource wasb://provisioning-hdi-default@eus2prodsa01150413.blob.core.windows.net/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar transitioned from DOWNLOADING to FAILED
> 2015-05-21 03:04:16,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432083697923_0483_01_000024 transitioned from LOCALIZING to LOCALIZATION_FAILED
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)