You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Fernando O." <fo...@gmail.com> on 2015/04/18 18:24:41 UTC

ResourceLocalizationService: Localizer failed when running pi example

Hey All,
    It's me again with another noob question: I deployed a cluster (HA
mode) everything looked good but when I tried to run the pi example:

 bin/hadoop jar
./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100

the same error occurs if I try to generate data with teragen 100000000
/test/data


2015-04-18 15:49:04,090 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)


I'm guessing it's a configuration issue but I don't know what am I missing
:S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
"SOLVED"...

but this is weird, I added hadoop.tmp.dir in core-site.xml I was ok with it
writing to the default location (/tmp/hadoop-${user.name}) but changed it
to /tmp/hadoop instead and now everything works....

Now I'm wondering why that might be an issue

On Mon, Apr 20, 2015 at 8:54 AM, Fernando O. <fo...@gmail.com> wrote:

> I did set so the files should be there for 10 minutes...
>
>   <property>
>       <name>yarn.nodemanager.delete.debug-delay-sec</name>
>       <value>600</value>
>   </property>
>
>
>
> On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> I didn't specify it so it's using the default value (in /tmp)
>>
>> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>>
>>> Hi,
>>>
>>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>>> provide that part of yarn-site.xml?
>>>
>>> Thanks.
>>>
>>> Drake 민영근 Ph.D
>>> kt NexR
>>>
>>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>>> yeah... there's not much there:
>>>>
>>>> -bash-4.1$ cd nm-local-dir/
>>>> -bash-4.1$ ll *
>>>> filecache:
>>>> total 0
>>>>
>>>> nmPrivate:
>>>> total 0
>>>>
>>>> usercache:
>>>> total 0
>>>>
>>>> I'm using Open JDK, would that be a problem?
>>>>
>>>> More log:
>>>>
>>>> STARTUP_MSG:   java = 1.7.0_75
>>>> ************************************************************/
>>>> 2015-04-19 14:38:58,168 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>>> signal handlers for [TERM, HUP, INT]
>>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 2015-04-19 14:38:59,018 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>>> 2015-04-19 14:38:59,020 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>>> 2015-04-19 14:38:59,022 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> 2015-04-19 14:38:59,023 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>>> class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>> 2015-04-19 14:38:59,109 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>> hadoop-metrics2.properties
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>> period at 10 second(s).
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>>> system started
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> per directory file limit = 8192
>>>> 2015-04-19 14:38:59,227 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>>> 2015-04-19 14:38:59,248 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>>> 'httpshuffle'. Because these are not the same tools trying to send
>>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>>> the name in the config.
>>>> 2015-04-19 14:38:59,248 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorPlugin :
>>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorProcessTree : null
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Physical memory check enabled: true
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Virtual memory check enabled: true
>>>> 2015-04-19 14:38:59,284 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> NodeManager configured with 14 G physical memory allocated to containers,
>>>> which is more than 80% of the total physical memory available (14.7 G).
>>>> Thrashing might happen.
>>>> 2015-04-19 14:38:59,287 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Initialized nodemanager for null: physical-memory=14336
>>>> virtual-memory=30106 virtual-cores=8
>>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 38230
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Blocking new container-requests as container manager rpc server is still
>>>> starting.
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 38230: starting
>>>> 2015-04-19 14:38:59,366 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 8040
>>>> 2015-04-19 14:38:59,376 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol
>>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8040: starting
>>>> 2015-04-19 14:38:59,380 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer started on port 8040
>>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>>> IndexCache created with max memory = 10485760
>>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>>> httpshuffle listening on port 13562
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>>> NMWebApp at 0.0.0.0:8042
>>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog:
>>>> Http request log for http.requests.nodemanager is not defined
>>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> global filter 'safety'
>>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context node
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context logs
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context static
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /node/*
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /ws/*
>>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>>> bound to port 8042
>>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /node started at 8042
>>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>> 2015-04-19 14:39:00,126 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>>> out 0 NM container statuses: []
>>>> 2015-04-19 14:39:00,131 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Registering with RM using containers :[]
>>>> 2015-04-19 14:39:00,176 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Rolling master-key for container-tokens, got key with id -543066608
>>>> 2015-04-19 14:39:00,178 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>>> Rolling master-key for container-tokens, got key with id -1243797706
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>>> resource of <memory:14336, vCores:8>
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>>> ContainerManager to unblock new container-requests
>>>> 2015-04-19 19:22:17,729 INFO
>>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>>> 2015-04-19 19:22:17,807 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>>> 2015-04-19 19:22:17,828 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Creating a new application reference for app application_1429450734039_0010
>>>> 2015-04-19 19:22:17,834 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>>> IP=10.100.66.251 OPERATION=Start Container Request
>>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Adding container_1429450734039_0010_01_000001 to application
>>>> application_1429450734039_0010
>>>> 2015-04-19 19:22:17,839 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from INITING to
>>>> RUNNING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>>> LOCALIZING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>>> 2015-04-19 19:22:17,876 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Created localizer for container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,880 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>> 2015-04-19 19:22:17,882 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZING to LOCALIZATION_FAILED
>>>> 2015-04-19 19:22:17,886 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>>> failed with state: LOCALIZATION_FAILED
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZATION_FAILED to DONE
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Removing container_1429450734039_0010_01_000001 from application
>>>> application_1429450734039_0010
>>>>
>>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>>> brahmareddy.battula@hotmail.com> wrote:
>>>>
>>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>>> or mapred.local.dir)..
>>>>>
>>>>> can you able provide full logs..?
>>>>>
>>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>>> details..
>>>>>
>>>>> ------------------------------
>>>>> From: wget.null@gmail.com
>>>>> Subject: Re: ResourceLocalizationService: Localizer failed when
>>>>> running pi example
>>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>>
>>>>> But without a complete stack trace, its a blind call.
>>>>>
>>>>> BR,
>>>>>  AL
>>>>>
>>>>> --
>>>>> mapredit.blogspot.com
>>>>>
>>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>>
>>>>> Hey All,
>>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>>> mode) everything looked good but when I tried to run the pi example:
>>>>>
>>>>>  bin/hadoop jar
>>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>>
>>>>> the same error occurs if I try to generate data with teragen
>>>>> 100000000 /test/data
>>>>>
>>>>>
>>>>> 2015-04-18 15:49:04,090 INFO
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>>> Localizer failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>>
>>>>>
>>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>>> missing :S
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
"SOLVED"...

but this is weird, I added hadoop.tmp.dir in core-site.xml I was ok with it
writing to the default location (/tmp/hadoop-${user.name}) but changed it
to /tmp/hadoop instead and now everything works....

Now I'm wondering why that might be an issue

On Mon, Apr 20, 2015 at 8:54 AM, Fernando O. <fo...@gmail.com> wrote:

> I did set so the files should be there for 10 minutes...
>
>   <property>
>       <name>yarn.nodemanager.delete.debug-delay-sec</name>
>       <value>600</value>
>   </property>
>
>
>
> On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> I didn't specify it so it's using the default value (in /tmp)
>>
>> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>>
>>> Hi,
>>>
>>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>>> provide that part of yarn-site.xml?
>>>
>>> Thanks.
>>>
>>> Drake 민영근 Ph.D
>>> kt NexR
>>>
>>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>>> yeah... there's not much there:
>>>>
>>>> -bash-4.1$ cd nm-local-dir/
>>>> -bash-4.1$ ll *
>>>> filecache:
>>>> total 0
>>>>
>>>> nmPrivate:
>>>> total 0
>>>>
>>>> usercache:
>>>> total 0
>>>>
>>>> I'm using Open JDK, would that be a problem?
>>>>
>>>> More log:
>>>>
>>>> STARTUP_MSG:   java = 1.7.0_75
>>>> ************************************************************/
>>>> 2015-04-19 14:38:58,168 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>>> signal handlers for [TERM, HUP, INT]
>>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 2015-04-19 14:38:59,018 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>>> 2015-04-19 14:38:59,020 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>>> 2015-04-19 14:38:59,022 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> 2015-04-19 14:38:59,023 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>>> class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>> 2015-04-19 14:38:59,109 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>> hadoop-metrics2.properties
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>> period at 10 second(s).
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>>> system started
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> per directory file limit = 8192
>>>> 2015-04-19 14:38:59,227 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>>> 2015-04-19 14:38:59,248 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>>> 'httpshuffle'. Because these are not the same tools trying to send
>>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>>> the name in the config.
>>>> 2015-04-19 14:38:59,248 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorPlugin :
>>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorProcessTree : null
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Physical memory check enabled: true
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Virtual memory check enabled: true
>>>> 2015-04-19 14:38:59,284 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> NodeManager configured with 14 G physical memory allocated to containers,
>>>> which is more than 80% of the total physical memory available (14.7 G).
>>>> Thrashing might happen.
>>>> 2015-04-19 14:38:59,287 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Initialized nodemanager for null: physical-memory=14336
>>>> virtual-memory=30106 virtual-cores=8
>>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 38230
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Blocking new container-requests as container manager rpc server is still
>>>> starting.
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 38230: starting
>>>> 2015-04-19 14:38:59,366 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 8040
>>>> 2015-04-19 14:38:59,376 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol
>>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8040: starting
>>>> 2015-04-19 14:38:59,380 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer started on port 8040
>>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>>> IndexCache created with max memory = 10485760
>>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>>> httpshuffle listening on port 13562
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>>> NMWebApp at 0.0.0.0:8042
>>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog:
>>>> Http request log for http.requests.nodemanager is not defined
>>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> global filter 'safety'
>>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context node
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context logs
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context static
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /node/*
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /ws/*
>>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>>> bound to port 8042
>>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /node started at 8042
>>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>> 2015-04-19 14:39:00,126 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>>> out 0 NM container statuses: []
>>>> 2015-04-19 14:39:00,131 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Registering with RM using containers :[]
>>>> 2015-04-19 14:39:00,176 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Rolling master-key for container-tokens, got key with id -543066608
>>>> 2015-04-19 14:39:00,178 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>>> Rolling master-key for container-tokens, got key with id -1243797706
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>>> resource of <memory:14336, vCores:8>
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>>> ContainerManager to unblock new container-requests
>>>> 2015-04-19 19:22:17,729 INFO
>>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>>> 2015-04-19 19:22:17,807 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>>> 2015-04-19 19:22:17,828 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Creating a new application reference for app application_1429450734039_0010
>>>> 2015-04-19 19:22:17,834 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>>> IP=10.100.66.251 OPERATION=Start Container Request
>>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Adding container_1429450734039_0010_01_000001 to application
>>>> application_1429450734039_0010
>>>> 2015-04-19 19:22:17,839 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from INITING to
>>>> RUNNING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>>> LOCALIZING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>>> 2015-04-19 19:22:17,876 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Created localizer for container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,880 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>> 2015-04-19 19:22:17,882 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZING to LOCALIZATION_FAILED
>>>> 2015-04-19 19:22:17,886 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>>> failed with state: LOCALIZATION_FAILED
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZATION_FAILED to DONE
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Removing container_1429450734039_0010_01_000001 from application
>>>> application_1429450734039_0010
>>>>
>>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>>> brahmareddy.battula@hotmail.com> wrote:
>>>>
>>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>>> or mapred.local.dir)..
>>>>>
>>>>> can you able provide full logs..?
>>>>>
>>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>>> details..
>>>>>
>>>>> ------------------------------
>>>>> From: wget.null@gmail.com
>>>>> Subject: Re: ResourceLocalizationService: Localizer failed when
>>>>> running pi example
>>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>>
>>>>> But without a complete stack trace, its a blind call.
>>>>>
>>>>> BR,
>>>>>  AL
>>>>>
>>>>> --
>>>>> mapredit.blogspot.com
>>>>>
>>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>>
>>>>> Hey All,
>>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>>> mode) everything looked good but when I tried to run the pi example:
>>>>>
>>>>>  bin/hadoop jar
>>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>>
>>>>> the same error occurs if I try to generate data with teragen
>>>>> 100000000 /test/data
>>>>>
>>>>>
>>>>> 2015-04-18 15:49:04,090 INFO
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>>> Localizer failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>>
>>>>>
>>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>>> missing :S
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
"SOLVED"...

but this is weird, I added hadoop.tmp.dir in core-site.xml I was ok with it
writing to the default location (/tmp/hadoop-${user.name}) but changed it
to /tmp/hadoop instead and now everything works....

Now I'm wondering why that might be an issue

On Mon, Apr 20, 2015 at 8:54 AM, Fernando O. <fo...@gmail.com> wrote:

> I did set so the files should be there for 10 minutes...
>
>   <property>
>       <name>yarn.nodemanager.delete.debug-delay-sec</name>
>       <value>600</value>
>   </property>
>
>
>
> On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> I didn't specify it so it's using the default value (in /tmp)
>>
>> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>>
>>> Hi,
>>>
>>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>>> provide that part of yarn-site.xml?
>>>
>>> Thanks.
>>>
>>> Drake 민영근 Ph.D
>>> kt NexR
>>>
>>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>>> yeah... there's not much there:
>>>>
>>>> -bash-4.1$ cd nm-local-dir/
>>>> -bash-4.1$ ll *
>>>> filecache:
>>>> total 0
>>>>
>>>> nmPrivate:
>>>> total 0
>>>>
>>>> usercache:
>>>> total 0
>>>>
>>>> I'm using Open JDK, would that be a problem?
>>>>
>>>> More log:
>>>>
>>>> STARTUP_MSG:   java = 1.7.0_75
>>>> ************************************************************/
>>>> 2015-04-19 14:38:58,168 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>>> signal handlers for [TERM, HUP, INT]
>>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 2015-04-19 14:38:59,018 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>>> 2015-04-19 14:38:59,020 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>>> 2015-04-19 14:38:59,022 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> 2015-04-19 14:38:59,023 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>>> class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>> 2015-04-19 14:38:59,109 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>> hadoop-metrics2.properties
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>> period at 10 second(s).
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>>> system started
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> per directory file limit = 8192
>>>> 2015-04-19 14:38:59,227 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>>> 2015-04-19 14:38:59,248 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>>> 'httpshuffle'. Because these are not the same tools trying to send
>>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>>> the name in the config.
>>>> 2015-04-19 14:38:59,248 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorPlugin :
>>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorProcessTree : null
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Physical memory check enabled: true
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Virtual memory check enabled: true
>>>> 2015-04-19 14:38:59,284 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> NodeManager configured with 14 G physical memory allocated to containers,
>>>> which is more than 80% of the total physical memory available (14.7 G).
>>>> Thrashing might happen.
>>>> 2015-04-19 14:38:59,287 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Initialized nodemanager for null: physical-memory=14336
>>>> virtual-memory=30106 virtual-cores=8
>>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 38230
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Blocking new container-requests as container manager rpc server is still
>>>> starting.
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 38230: starting
>>>> 2015-04-19 14:38:59,366 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 8040
>>>> 2015-04-19 14:38:59,376 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol
>>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8040: starting
>>>> 2015-04-19 14:38:59,380 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer started on port 8040
>>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>>> IndexCache created with max memory = 10485760
>>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>>> httpshuffle listening on port 13562
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>>> NMWebApp at 0.0.0.0:8042
>>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog:
>>>> Http request log for http.requests.nodemanager is not defined
>>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> global filter 'safety'
>>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context node
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context logs
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context static
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /node/*
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /ws/*
>>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>>> bound to port 8042
>>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /node started at 8042
>>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>> 2015-04-19 14:39:00,126 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>>> out 0 NM container statuses: []
>>>> 2015-04-19 14:39:00,131 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Registering with RM using containers :[]
>>>> 2015-04-19 14:39:00,176 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Rolling master-key for container-tokens, got key with id -543066608
>>>> 2015-04-19 14:39:00,178 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>>> Rolling master-key for container-tokens, got key with id -1243797706
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>>> resource of <memory:14336, vCores:8>
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>>> ContainerManager to unblock new container-requests
>>>> 2015-04-19 19:22:17,729 INFO
>>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>>> 2015-04-19 19:22:17,807 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>>> 2015-04-19 19:22:17,828 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Creating a new application reference for app application_1429450734039_0010
>>>> 2015-04-19 19:22:17,834 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>>> IP=10.100.66.251 OPERATION=Start Container Request
>>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Adding container_1429450734039_0010_01_000001 to application
>>>> application_1429450734039_0010
>>>> 2015-04-19 19:22:17,839 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from INITING to
>>>> RUNNING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>>> LOCALIZING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>>> 2015-04-19 19:22:17,876 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Created localizer for container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,880 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>> 2015-04-19 19:22:17,882 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZING to LOCALIZATION_FAILED
>>>> 2015-04-19 19:22:17,886 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>>> failed with state: LOCALIZATION_FAILED
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZATION_FAILED to DONE
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Removing container_1429450734039_0010_01_000001 from application
>>>> application_1429450734039_0010
>>>>
>>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>>> brahmareddy.battula@hotmail.com> wrote:
>>>>
>>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>>> or mapred.local.dir)..
>>>>>
>>>>> can you able provide full logs..?
>>>>>
>>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>>> details..
>>>>>
>>>>> ------------------------------
>>>>> From: wget.null@gmail.com
>>>>> Subject: Re: ResourceLocalizationService: Localizer failed when
>>>>> running pi example
>>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>>
>>>>> But without a complete stack trace, its a blind call.
>>>>>
>>>>> BR,
>>>>>  AL
>>>>>
>>>>> --
>>>>> mapredit.blogspot.com
>>>>>
>>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>>
>>>>> Hey All,
>>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>>> mode) everything looked good but when I tried to run the pi example:
>>>>>
>>>>>  bin/hadoop jar
>>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>>
>>>>> the same error occurs if I try to generate data with teragen
>>>>> 100000000 /test/data
>>>>>
>>>>>
>>>>> 2015-04-18 15:49:04,090 INFO
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>>> Localizer failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>>
>>>>>
>>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>>> missing :S
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
"SOLVED"...

but this is weird, I added hadoop.tmp.dir in core-site.xml I was ok with it
writing to the default location (/tmp/hadoop-${user.name}) but changed it
to /tmp/hadoop instead and now everything works....

Now I'm wondering why that might be an issue

On Mon, Apr 20, 2015 at 8:54 AM, Fernando O. <fo...@gmail.com> wrote:

> I did set so the files should be there for 10 minutes...
>
>   <property>
>       <name>yarn.nodemanager.delete.debug-delay-sec</name>
>       <value>600</value>
>   </property>
>
>
>
> On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> I didn't specify it so it's using the default value (in /tmp)
>>
>> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>>
>>> Hi,
>>>
>>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>>> provide that part of yarn-site.xml?
>>>
>>> Thanks.
>>>
>>> Drake 민영근 Ph.D
>>> kt NexR
>>>
>>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>>> yeah... there's not much there:
>>>>
>>>> -bash-4.1$ cd nm-local-dir/
>>>> -bash-4.1$ ll *
>>>> filecache:
>>>> total 0
>>>>
>>>> nmPrivate:
>>>> total 0
>>>>
>>>> usercache:
>>>> total 0
>>>>
>>>> I'm using Open JDK, would that be a problem?
>>>>
>>>> More log:
>>>>
>>>> STARTUP_MSG:   java = 1.7.0_75
>>>> ************************************************************/
>>>> 2015-04-19 14:38:58,168 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>>> signal handlers for [TERM, HUP, INT]
>>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 2015-04-19 14:38:59,018 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>>> 2015-04-19 14:38:59,020 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>>> 2015-04-19 14:38:59,021 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>>> 2015-04-19 14:38:59,022 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> 2015-04-19 14:38:59,023 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>>> class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>>> 2015-04-19 14:38:59,054 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>> 2015-04-19 14:38:59,109 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>> hadoop-metrics2.properties
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>> period at 10 second(s).
>>>> 2015-04-19 14:38:59,197 INFO
>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>>> system started
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>>> 2015-04-19 14:38:59,217 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> per directory file limit = 8192
>>>> 2015-04-19 14:38:59,227 INFO
>>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>>> for class
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>>> 2015-04-19 14:38:59,248 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>>> 'httpshuffle'. Because these are not the same tools trying to send
>>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>>> the name in the config.
>>>> 2015-04-19 14:38:59,248 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorPlugin :
>>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>>  Using ResourceCalculatorProcessTree : null
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Physical memory check enabled: true
>>>> 2015-04-19 14:38:59,281 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> Virtual memory check enabled: true
>>>> 2015-04-19 14:38:59,284 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>> NodeManager configured with 14 G physical memory allocated to containers,
>>>> which is more than 80% of the total physical memory available (14.7 G).
>>>> Thrashing might happen.
>>>> 2015-04-19 14:38:59,287 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Initialized nodemanager for null: physical-memory=14336
>>>> virtual-memory=30106 virtual-cores=8
>>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 38230
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,359 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Blocking new container-requests as container manager rpc server is still
>>>> starting.
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 38230: starting
>>>> 2015-04-19 14:38:59,366 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 8040
>>>> 2015-04-19 14:38:59,376 INFO
>>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>>> protocol
>>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>>> server
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> Responder: starting
>>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8040: starting
>>>> 2015-04-19 14:38:59,380 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer started on port 8040
>>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>>> IndexCache created with max memory = 10485760
>>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>>> httpshuffle listening on port 13562
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>>> 2015-04-19 14:38:59,405 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>>> NMWebApp at 0.0.0.0:8042
>>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog:
>>>> Http request log for http.requests.nodemanager is not defined
>>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> global filter 'safety'
>>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context node
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context logs
>>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>>> filter static_user_filter
>>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>>> context static
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /node/*
>>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>>> path spec: /ws/*
>>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>>> bound to port 8042
>>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /node started at 8042
>>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>> 2015-04-19 14:39:00,126 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>>> out 0 NM container statuses: []
>>>> 2015-04-19 14:39:00,131 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>>> Registering with RM using containers :[]
>>>> 2015-04-19 14:39:00,176 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>>> Rolling master-key for container-tokens, got key with id -543066608
>>>> 2015-04-19 14:39:00,178 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>>> Rolling master-key for container-tokens, got key with id -1243797706
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>>> resource of <memory:14336, vCores:8>
>>>> 2015-04-19 14:39:00,179 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>>> ContainerManager to unblock new container-requests
>>>> 2015-04-19 19:22:17,729 INFO
>>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>>> 2015-04-19 19:22:17,807 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>>> 2015-04-19 19:22:17,828 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>>> Creating a new application reference for app application_1429450734039_0010
>>>> 2015-04-19 19:22:17,834 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>>> IP=10.100.66.251 OPERATION=Start Container Request
>>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>>> 2015-04-19 19:22:17,835 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Adding container_1429450734039_0010_01_000001 to application
>>>> application_1429450734039_0010
>>>> 2015-04-19 19:22:17,839 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Application application_1429450734039_0010 transitioned from INITING to
>>>> RUNNING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>>> LOCALIZING
>>>> 2015-04-19 19:22:17,843 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>>> 2015-04-19 19:22:17,876 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>>> Resource
>>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>>> transitioned from INIT to DOWNLOADING
>>>> 2015-04-19 19:22:17,877 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Created localizer for container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,880 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>> 2015-04-19 19:22:17,882 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZING to LOCALIZATION_FAILED
>>>> 2015-04-19 19:22:17,886 WARN
>>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>>> failed with state: LOCALIZATION_FAILED
>>>> APPID=application_1429450734039_0010
>>>> CONTAINERID=container_1429450734039_0010_01_000001
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>>> Container container_1429450734039_0010_01_000001 transitioned from
>>>> LOCALIZATION_FAILED to DONE
>>>> 2015-04-19 19:22:17,889 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>>> Removing container_1429450734039_0010_01_000001 from application
>>>> application_1429450734039_0010
>>>>
>>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>>> brahmareddy.battula@hotmail.com> wrote:
>>>>
>>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>>> or mapred.local.dir)..
>>>>>
>>>>> can you able provide full logs..?
>>>>>
>>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>>> details..
>>>>>
>>>>> ------------------------------
>>>>> From: wget.null@gmail.com
>>>>> Subject: Re: ResourceLocalizationService: Localizer failed when
>>>>> running pi example
>>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>>
>>>>> But without a complete stack trace, its a blind call.
>>>>>
>>>>> BR,
>>>>>  AL
>>>>>
>>>>> --
>>>>> mapredit.blogspot.com
>>>>>
>>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>>
>>>>> Hey All,
>>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>>> mode) everything looked good but when I tried to run the pi example:
>>>>>
>>>>>  bin/hadoop jar
>>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>>
>>>>> the same error occurs if I try to generate data with teragen
>>>>> 100000000 /test/data
>>>>>
>>>>>
>>>>> 2015-04-18 15:49:04,090 INFO
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>>> Localizer failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>>> at
>>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>>
>>>>>
>>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>>> missing :S
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I did set so the files should be there for 10 minutes...

  <property>
      <name>yarn.nodemanager.delete.debug-delay-sec</name>
      <value>600</value>
  </property>



On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:

> I didn't specify it so it's using the default value (in /tmp)
>
> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>
>> Hi,
>>
>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>> provide that part of yarn-site.xml?
>>
>> Thanks.
>>
>> Drake 민영근 Ph.D
>> kt NexR
>>
>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> yeah... there's not much there:
>>>
>>> -bash-4.1$ cd nm-local-dir/
>>> -bash-4.1$ ll *
>>> filecache:
>>> total 0
>>>
>>> nmPrivate:
>>> total 0
>>>
>>> usercache:
>>> total 0
>>>
>>> I'm using Open JDK, would that be a problem?
>>>
>>> More log:
>>>
>>> STARTUP_MSG:   java = 1.7.0_75
>>> ************************************************************/
>>> 2015-04-19 14:38:58,168 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>> signal handlers for [TERM, HUP, INT]
>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2015-04-19 14:38:59,018 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>> 2015-04-19 14:38:59,020 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>> 2015-04-19 14:38:59,022 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> 2015-04-19 14:38:59,023 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>> class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>> 2015-04-19 14:38:59,109 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>> system started
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> per directory file limit = 8192
>>> 2015-04-19 14:38:59,227 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>> 2015-04-19 14:38:59,248 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>> 'httpshuffle'. Because these are not the same tools trying to send
>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>> the name in the config.
>>> 2015-04-19 14:38:59,248 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorProcessTree : null
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Physical memory check enabled: true
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Virtual memory check enabled: true
>>> 2015-04-19 14:38:59,284 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> NodeManager configured with 14 G physical memory allocated to containers,
>>> which is more than 80% of the total physical memory available (14.7 G).
>>> Thrashing might happen.
>>> 2015-04-19 14:38:59,287 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Initialized nodemanager for null: physical-memory=14336
>>> virtual-memory=30106 virtual-cores=8
>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 38230
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Blocking new container-requests as container manager rpc server is still
>>> starting.
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 38230: starting
>>> 2015-04-19 14:38:59,366 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 8040
>>> 2015-04-19 14:38:59,376 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol
>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 8040: starting
>>> 2015-04-19 14:38:59,380 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer started on port 8040
>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>> httpshuffle listening on port 13562
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>> NMWebApp at 0.0.0.0:8042
>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>>> request log for http.requests.nodemanager is not defined
>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>> global filter 'safety'
>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context node
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context logs
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context static
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /node/*
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /ws/*
>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>> bound to port 8042
>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /node started at 8042
>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>> 2015-04-19 14:39:00,126 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>> out 0 NM container statuses: []
>>> 2015-04-19 14:39:00,131 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Registering with RM using containers :[]
>>> 2015-04-19 14:39:00,176 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Rolling master-key for container-tokens, got key with id -543066608
>>> 2015-04-19 14:39:00,178 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>> Rolling master-key for container-tokens, got key with id -1243797706
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>> resource of <memory:14336, vCores:8>
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>> ContainerManager to unblock new container-requests
>>> 2015-04-19 19:22:17,729 INFO
>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>> 2015-04-19 19:22:17,807 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>> 2015-04-19 19:22:17,828 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Creating a new application reference for app application_1429450734039_0010
>>> 2015-04-19 19:22:17,834 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>> IP=10.100.66.251 OPERATION=Start Container Request
>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Adding container_1429450734039_0010_01_000001 to application
>>> application_1429450734039_0010
>>> 2015-04-19 19:22:17,839 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from INITING to
>>> RUNNING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>> LOCALIZING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>> 2015-04-19 19:22:17,876 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Created localizer for container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,880 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>> 2015-04-19 19:22:17,882 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZING to LOCALIZATION_FAILED
>>> 2015-04-19 19:22:17,886 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>> failed with state: LOCALIZATION_FAILED
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZATION_FAILED to DONE
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Removing container_1429450734039_0010_01_000001 from application
>>> application_1429450734039_0010
>>>
>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>> brahmareddy.battula@hotmail.com> wrote:
>>>
>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>> or mapred.local.dir)..
>>>>
>>>> can you able provide full logs..?
>>>>
>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>> details..
>>>>
>>>> ------------------------------
>>>> From: wget.null@gmail.com
>>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>>> pi example
>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>
>>>> But without a complete stack trace, its a blind call.
>>>>
>>>> BR,
>>>>  AL
>>>>
>>>> --
>>>> mapredit.blogspot.com
>>>>
>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>
>>>> Hey All,
>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>> mode) everything looked good but when I tried to run the pi example:
>>>>
>>>>  bin/hadoop jar
>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>
>>>> the same error occurs if I try to generate data with teragen 100000000
>>>> /test/data
>>>>
>>>>
>>>> 2015-04-18 15:49:04,090 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>
>>>>
>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>> missing :S
>>>>
>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I did set so the files should be there for 10 minutes...

  <property>
      <name>yarn.nodemanager.delete.debug-delay-sec</name>
      <value>600</value>
  </property>



On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:

> I didn't specify it so it's using the default value (in /tmp)
>
> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>
>> Hi,
>>
>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>> provide that part of yarn-site.xml?
>>
>> Thanks.
>>
>> Drake 민영근 Ph.D
>> kt NexR
>>
>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> yeah... there's not much there:
>>>
>>> -bash-4.1$ cd nm-local-dir/
>>> -bash-4.1$ ll *
>>> filecache:
>>> total 0
>>>
>>> nmPrivate:
>>> total 0
>>>
>>> usercache:
>>> total 0
>>>
>>> I'm using Open JDK, would that be a problem?
>>>
>>> More log:
>>>
>>> STARTUP_MSG:   java = 1.7.0_75
>>> ************************************************************/
>>> 2015-04-19 14:38:58,168 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>> signal handlers for [TERM, HUP, INT]
>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2015-04-19 14:38:59,018 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>> 2015-04-19 14:38:59,020 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>> 2015-04-19 14:38:59,022 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> 2015-04-19 14:38:59,023 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>> class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>> 2015-04-19 14:38:59,109 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>> system started
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> per directory file limit = 8192
>>> 2015-04-19 14:38:59,227 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>> 2015-04-19 14:38:59,248 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>> 'httpshuffle'. Because these are not the same tools trying to send
>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>> the name in the config.
>>> 2015-04-19 14:38:59,248 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorProcessTree : null
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Physical memory check enabled: true
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Virtual memory check enabled: true
>>> 2015-04-19 14:38:59,284 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> NodeManager configured with 14 G physical memory allocated to containers,
>>> which is more than 80% of the total physical memory available (14.7 G).
>>> Thrashing might happen.
>>> 2015-04-19 14:38:59,287 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Initialized nodemanager for null: physical-memory=14336
>>> virtual-memory=30106 virtual-cores=8
>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 38230
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Blocking new container-requests as container manager rpc server is still
>>> starting.
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 38230: starting
>>> 2015-04-19 14:38:59,366 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 8040
>>> 2015-04-19 14:38:59,376 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol
>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 8040: starting
>>> 2015-04-19 14:38:59,380 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer started on port 8040
>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>> httpshuffle listening on port 13562
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>> NMWebApp at 0.0.0.0:8042
>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>>> request log for http.requests.nodemanager is not defined
>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>> global filter 'safety'
>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context node
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context logs
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context static
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /node/*
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /ws/*
>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>> bound to port 8042
>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /node started at 8042
>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>> 2015-04-19 14:39:00,126 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>> out 0 NM container statuses: []
>>> 2015-04-19 14:39:00,131 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Registering with RM using containers :[]
>>> 2015-04-19 14:39:00,176 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Rolling master-key for container-tokens, got key with id -543066608
>>> 2015-04-19 14:39:00,178 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>> Rolling master-key for container-tokens, got key with id -1243797706
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>> resource of <memory:14336, vCores:8>
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>> ContainerManager to unblock new container-requests
>>> 2015-04-19 19:22:17,729 INFO
>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>> 2015-04-19 19:22:17,807 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>> 2015-04-19 19:22:17,828 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Creating a new application reference for app application_1429450734039_0010
>>> 2015-04-19 19:22:17,834 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>> IP=10.100.66.251 OPERATION=Start Container Request
>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Adding container_1429450734039_0010_01_000001 to application
>>> application_1429450734039_0010
>>> 2015-04-19 19:22:17,839 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from INITING to
>>> RUNNING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>> LOCALIZING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>> 2015-04-19 19:22:17,876 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Created localizer for container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,880 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>> 2015-04-19 19:22:17,882 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZING to LOCALIZATION_FAILED
>>> 2015-04-19 19:22:17,886 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>> failed with state: LOCALIZATION_FAILED
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZATION_FAILED to DONE
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Removing container_1429450734039_0010_01_000001 from application
>>> application_1429450734039_0010
>>>
>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>> brahmareddy.battula@hotmail.com> wrote:
>>>
>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>> or mapred.local.dir)..
>>>>
>>>> can you able provide full logs..?
>>>>
>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>> details..
>>>>
>>>> ------------------------------
>>>> From: wget.null@gmail.com
>>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>>> pi example
>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>
>>>> But without a complete stack trace, its a blind call.
>>>>
>>>> BR,
>>>>  AL
>>>>
>>>> --
>>>> mapredit.blogspot.com
>>>>
>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>
>>>> Hey All,
>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>> mode) everything looked good but when I tried to run the pi example:
>>>>
>>>>  bin/hadoop jar
>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>
>>>> the same error occurs if I try to generate data with teragen 100000000
>>>> /test/data
>>>>
>>>>
>>>> 2015-04-18 15:49:04,090 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>
>>>>
>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>> missing :S
>>>>
>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I did set so the files should be there for 10 minutes...

  <property>
      <name>yarn.nodemanager.delete.debug-delay-sec</name>
      <value>600</value>
  </property>



On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:

> I didn't specify it so it's using the default value (in /tmp)
>
> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>
>> Hi,
>>
>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>> provide that part of yarn-site.xml?
>>
>> Thanks.
>>
>> Drake 민영근 Ph.D
>> kt NexR
>>
>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> yeah... there's not much there:
>>>
>>> -bash-4.1$ cd nm-local-dir/
>>> -bash-4.1$ ll *
>>> filecache:
>>> total 0
>>>
>>> nmPrivate:
>>> total 0
>>>
>>> usercache:
>>> total 0
>>>
>>> I'm using Open JDK, would that be a problem?
>>>
>>> More log:
>>>
>>> STARTUP_MSG:   java = 1.7.0_75
>>> ************************************************************/
>>> 2015-04-19 14:38:58,168 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>> signal handlers for [TERM, HUP, INT]
>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2015-04-19 14:38:59,018 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>> 2015-04-19 14:38:59,020 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>> 2015-04-19 14:38:59,022 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> 2015-04-19 14:38:59,023 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>> class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>> 2015-04-19 14:38:59,109 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>> system started
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> per directory file limit = 8192
>>> 2015-04-19 14:38:59,227 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>> 2015-04-19 14:38:59,248 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>> 'httpshuffle'. Because these are not the same tools trying to send
>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>> the name in the config.
>>> 2015-04-19 14:38:59,248 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorProcessTree : null
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Physical memory check enabled: true
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Virtual memory check enabled: true
>>> 2015-04-19 14:38:59,284 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> NodeManager configured with 14 G physical memory allocated to containers,
>>> which is more than 80% of the total physical memory available (14.7 G).
>>> Thrashing might happen.
>>> 2015-04-19 14:38:59,287 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Initialized nodemanager for null: physical-memory=14336
>>> virtual-memory=30106 virtual-cores=8
>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 38230
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Blocking new container-requests as container manager rpc server is still
>>> starting.
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 38230: starting
>>> 2015-04-19 14:38:59,366 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 8040
>>> 2015-04-19 14:38:59,376 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol
>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 8040: starting
>>> 2015-04-19 14:38:59,380 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer started on port 8040
>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>> httpshuffle listening on port 13562
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>> NMWebApp at 0.0.0.0:8042
>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>>> request log for http.requests.nodemanager is not defined
>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>> global filter 'safety'
>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context node
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context logs
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context static
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /node/*
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /ws/*
>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>> bound to port 8042
>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /node started at 8042
>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>> 2015-04-19 14:39:00,126 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>> out 0 NM container statuses: []
>>> 2015-04-19 14:39:00,131 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Registering with RM using containers :[]
>>> 2015-04-19 14:39:00,176 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Rolling master-key for container-tokens, got key with id -543066608
>>> 2015-04-19 14:39:00,178 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>> Rolling master-key for container-tokens, got key with id -1243797706
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>> resource of <memory:14336, vCores:8>
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>> ContainerManager to unblock new container-requests
>>> 2015-04-19 19:22:17,729 INFO
>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>> 2015-04-19 19:22:17,807 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>> 2015-04-19 19:22:17,828 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Creating a new application reference for app application_1429450734039_0010
>>> 2015-04-19 19:22:17,834 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>> IP=10.100.66.251 OPERATION=Start Container Request
>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Adding container_1429450734039_0010_01_000001 to application
>>> application_1429450734039_0010
>>> 2015-04-19 19:22:17,839 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from INITING to
>>> RUNNING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>> LOCALIZING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>> 2015-04-19 19:22:17,876 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Created localizer for container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,880 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>> 2015-04-19 19:22:17,882 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZING to LOCALIZATION_FAILED
>>> 2015-04-19 19:22:17,886 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>> failed with state: LOCALIZATION_FAILED
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZATION_FAILED to DONE
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Removing container_1429450734039_0010_01_000001 from application
>>> application_1429450734039_0010
>>>
>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>> brahmareddy.battula@hotmail.com> wrote:
>>>
>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>> or mapred.local.dir)..
>>>>
>>>> can you able provide full logs..?
>>>>
>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>> details..
>>>>
>>>> ------------------------------
>>>> From: wget.null@gmail.com
>>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>>> pi example
>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>
>>>> But without a complete stack trace, its a blind call.
>>>>
>>>> BR,
>>>>  AL
>>>>
>>>> --
>>>> mapredit.blogspot.com
>>>>
>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>
>>>> Hey All,
>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>> mode) everything looked good but when I tried to run the pi example:
>>>>
>>>>  bin/hadoop jar
>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>
>>>> the same error occurs if I try to generate data with teragen 100000000
>>>> /test/data
>>>>
>>>>
>>>> 2015-04-18 15:49:04,090 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>
>>>>
>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>> missing :S
>>>>
>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I did set so the files should be there for 10 minutes...

  <property>
      <name>yarn.nodemanager.delete.debug-delay-sec</name>
      <value>600</value>
  </property>



On Mon, Apr 20, 2015 at 8:52 AM, Fernando O. <fo...@gmail.com> wrote:

> I didn't specify it so it's using the default value (in /tmp)
>
> On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:
>
>> Hi,
>>
>> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
>> provide that part of yarn-site.xml?
>>
>> Thanks.
>>
>> Drake 민영근 Ph.D
>> kt NexR
>>
>> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> yeah... there's not much there:
>>>
>>> -bash-4.1$ cd nm-local-dir/
>>> -bash-4.1$ ll *
>>> filecache:
>>> total 0
>>>
>>> nmPrivate:
>>> total 0
>>>
>>> usercache:
>>> total 0
>>>
>>> I'm using Open JDK, would that be a problem?
>>>
>>> More log:
>>>
>>> STARTUP_MSG:   java = 1.7.0_75
>>> ************************************************************/
>>> 2015-04-19 14:38:58,168 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>>> signal handlers for [TERM, HUP, INT]
>>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>>> Unable to load native-hadoop library for your platform... using
>>> builtin-java classes where applicable
>>> 2015-04-19 14:38:59,018 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>>> 2015-04-19 14:38:59,020 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>>> 2015-04-19 14:38:59,021 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>>> 2015-04-19 14:38:59,022 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> 2015-04-19 14:38:59,023 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>>> class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>>> 2015-04-19 14:38:59,054 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>> 2015-04-19 14:38:59,109 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2015-04-19 14:38:59,197 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>>> system started
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>>> 2015-04-19 14:38:59,217 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> per directory file limit = 8192
>>> 2015-04-19 14:38:59,227 INFO
>>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>>> for class
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>>> 2015-04-19 14:38:59,248 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>>> 'httpshuffle'. Because these are not the same tools trying to send
>>> ServiceData and read Service Meta Data may have issues unless the refer to
>>> the name in the config.
>>> 2015-04-19 14:38:59,248 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorPlugin :
>>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>>  Using ResourceCalculatorProcessTree : null
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Physical memory check enabled: true
>>> 2015-04-19 14:38:59,281 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> Virtual memory check enabled: true
>>> 2015-04-19 14:38:59,284 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>> NodeManager configured with 14 G physical memory allocated to containers,
>>> which is more than 80% of the total physical memory available (14.7 G).
>>> Thrashing might happen.
>>> 2015-04-19 14:38:59,287 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Initialized nodemanager for null: physical-memory=14336
>>> virtual-memory=30106 virtual-cores=8
>>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 38230
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,359 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Blocking new container-requests as container manager rpc server is still
>>> starting.
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 38230: starting
>>> 2015-04-19 14:38:59,366 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>>> Socket Reader #1 for port 8040
>>> 2015-04-19 14:38:59,376 INFO
>>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>>> protocol
>>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>>> server
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: starting
>>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> listener on 8040: starting
>>> 2015-04-19 14:38:59,380 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer started on port 8040
>>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>>> IndexCache created with max memory = 10485760
>>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>>> httpshuffle listening on port 13562
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>>> 2015-04-19 14:38:59,405 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>>> NMWebApp at 0.0.0.0:8042
>>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>>> request log for http.requests.nodemanager is not defined
>>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>>> global filter 'safety'
>>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context node
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context logs
>>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>>> filter static_user_filter
>>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>>> context static
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /node/*
>>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>>> path spec: /ws/*
>>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>>> bound to port 8042
>>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /node started at 8042
>>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>> 2015-04-19 14:39:00,126 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>>> out 0 NM container statuses: []
>>> 2015-04-19 14:39:00,131 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>>> Registering with RM using containers :[]
>>> 2015-04-19 14:39:00,176 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>>> Rolling master-key for container-tokens, got key with id -543066608
>>> 2015-04-19 14:39:00,178 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>>> Rolling master-key for container-tokens, got key with id -1243797706
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>>> resource of <memory:14336, vCores:8>
>>> 2015-04-19 14:39:00,179 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>>> ContainerManager to unblock new container-requests
>>> 2015-04-19 19:22:17,729 INFO
>>> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
>>> appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>>> 2015-04-19 19:22:17,807 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Start request for container_1429450734039_0010_01_000001 by user nobody
>>> 2015-04-19 19:22:17,828 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>>> Creating a new application reference for app application_1429450734039_0010
>>> 2015-04-19 19:22:17,834 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>>> IP=10.100.66.251 OPERATION=Start Container Request
>>> TARGET=ContainerManageImpl RESULT=SUCCESS
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from NEW to INITING
>>> 2015-04-19 19:22:17,835 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Adding container_1429450734039_0010_01_000001 to application
>>> application_1429450734039_0010
>>> 2015-04-19 19:22:17,839 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Application application_1429450734039_0010 transitioned from INITING to
>>> RUNNING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>>> LOCALIZING
>>> 2015-04-19 19:22:17,843 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>>> event CONTAINER_INIT for appId application_1429450734039_0010
>>> 2015-04-19 19:22:17,876 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> Resource
>>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>>> transitioned from INIT to DOWNLOADING
>>> 2015-04-19 19:22:17,877 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Created localizer for container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,880 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>> 2015-04-19 19:22:17,882 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZING to LOCALIZATION_FAILED
>>> 2015-04-19 19:22:17,886 WARN
>>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>>> failed with state: LOCALIZATION_FAILED
>>> APPID=application_1429450734039_0010
>>> CONTAINERID=container_1429450734039_0010_01_000001
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> Container container_1429450734039_0010_01_000001 transitioned from
>>> LOCALIZATION_FAILED to DONE
>>> 2015-04-19 19:22:17,889 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>>> Removing container_1429450734039_0010_01_000001 from application
>>> application_1429450734039_0010
>>>
>>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>>> brahmareddy.battula@hotmail.com> wrote:
>>>
>>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>>> or mapred.local.dir)..
>>>>
>>>> can you able provide full logs..?
>>>>
>>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>>> details..
>>>>
>>>> ------------------------------
>>>> From: wget.null@gmail.com
>>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>>> pi example
>>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> As you said, that looks like a config issue. I would spot on the NM's
>>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>>
>>>> But without a complete stack trace, its a blind call.
>>>>
>>>> BR,
>>>>  AL
>>>>
>>>> --
>>>> mapredit.blogspot.com
>>>>
>>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>>
>>>> Hey All,
>>>>     It's me again with another noob question: I deployed a cluster (HA
>>>> mode) everything looked good but when I tried to run the pi example:
>>>>
>>>>  bin/hadoop jar
>>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>>
>>>> the same error occurs if I try to generate data with teragen 100000000
>>>> /test/data
>>>>
>>>>
>>>> 2015-04-18 15:49:04,090 INFO
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>>> Localizer failed
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>>
>>>>
>>>> I'm guessing it's a configuration issue but I don't know what am I
>>>> missing :S
>>>>
>>>>
>>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I didn't specify it so it's using the default value (in /tmp)

On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:

> Hi,
>
> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
> provide that part of yarn-site.xml?
>
> Thanks.
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> yeah... there's not much there:
>>
>> -bash-4.1$ cd nm-local-dir/
>> -bash-4.1$ ll *
>> filecache:
>> total 0
>>
>> nmPrivate:
>> total 0
>>
>> usercache:
>> total 0
>>
>> I'm using Open JDK, would that be a problem?
>>
>> More log:
>>
>> STARTUP_MSG:   java = 1.7.0_75
>> ************************************************************/
>> 2015-04-19 14:38:58,168 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>> signal handlers for [TERM, HUP, INT]
>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2015-04-19 14:38:59,018 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>> 2015-04-19 14:38:59,020 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> 2015-04-19 14:38:59,022 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> 2015-04-19 14:38:59,023 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>> class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> 2015-04-19 14:38:59,109 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system started
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> per directory file limit = 8192
>> 2015-04-19 14:38:59,227 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> 2015-04-19 14:38:59,248 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>> 'httpshuffle'. Because these are not the same tools trying to send
>> ServiceData and read Service Meta Data may have issues unless the refer to
>> the name in the config.
>> 2015-04-19 14:38:59,248 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorProcessTree : null
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Physical memory check enabled: true
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Virtual memory check enabled: true
>> 2015-04-19 14:38:59,284 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> NodeManager configured with 14 G physical memory allocated to containers,
>> which is more than 80% of the total physical memory available (14.7 G).
>> Thrashing might happen.
>> 2015-04-19 14:38:59,287 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Initialized nodemanager for null: physical-memory=14336
>> virtual-memory=30106 virtual-cores=8
>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 38230
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>> server
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Blocking new container-requests as container manager rpc server is still
>> starting.
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 38230: starting
>> 2015-04-19 14:38:59,366 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 8040
>> 2015-04-19 14:38:59,376 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol
>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>> server
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 8040: starting
>> 2015-04-19 14:38:59,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer started on port 8040
>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>> httpshuffle listening on port 13562
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>> NMWebApp at 0.0.0.0:8042
>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.nodemanager is not defined
>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context node
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context logs
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /node/*
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /ws/*
>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 8042
>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /node started at 8042
>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>> 2015-04-19 14:39:00,126 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>> out 0 NM container statuses: []
>> 2015-04-19 14:39:00,131 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Registering with RM using containers :[]
>> 2015-04-19 14:39:00,176 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Rolling master-key for container-tokens, got key with id -543066608
>> 2015-04-19 14:39:00,178 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>> Rolling master-key for container-tokens, got key with id -1243797706
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>> resource of <memory:14336, vCores:8>
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>> ContainerManager to unblock new container-requests
>> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
>> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>> 2015-04-19 19:22:17,807 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Start request for container_1429450734039_0010_01_000001 by user nobody
>> 2015-04-19 19:22:17,828 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Creating a new application reference for app application_1429450734039_0010
>> 2015-04-19 19:22:17,834 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>> IP=10.100.66.251 OPERATION=Start Container Request
>> TARGET=ContainerManageImpl RESULT=SUCCESS
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from NEW to INITING
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Adding container_1429450734039_0010_01_000001 to application
>> application_1429450734039_0010
>> 2015-04-19 19:22:17,839 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from INITING to
>> RUNNING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>> LOCALIZING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>> event CONTAINER_INIT for appId application_1429450734039_0010
>> 2015-04-19 19:22:17,876 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Created localizer for container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,880 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>> 2015-04-19 19:22:17,882 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZING to LOCALIZATION_FAILED
>> 2015-04-19 19:22:17,886 WARN
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>> failed with state: LOCALIZATION_FAILED
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZATION_FAILED to DONE
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Removing container_1429450734039_0010_01_000001 from application
>> application_1429450734039_0010
>>
>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>> brahmareddy.battula@hotmail.com> wrote:
>>
>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>> or mapred.local.dir)..
>>>
>>> can you able provide full logs..?
>>>
>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>> details..
>>>
>>> ------------------------------
>>> From: wget.null@gmail.com
>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>> pi example
>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>> To: user@hadoop.apache.org
>>>
>>>
>>> As you said, that looks like a config issue. I would spot on the NM's
>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>
>>> But without a complete stack trace, its a blind call.
>>>
>>> BR,
>>>  AL
>>>
>>> --
>>> mapredit.blogspot.com
>>>
>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>> Hey All,
>>>     It's me again with another noob question: I deployed a cluster (HA
>>> mode) everything looked good but when I tried to run the pi example:
>>>
>>>  bin/hadoop jar
>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>
>>> the same error occurs if I try to generate data with teragen 100000000
>>> /test/data
>>>
>>>
>>> 2015-04-18 15:49:04,090 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>
>>>
>>> I'm guessing it's a configuration issue but I don't know what am I
>>> missing :S
>>>
>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I didn't specify it so it's using the default value (in /tmp)

On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:

> Hi,
>
> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
> provide that part of yarn-site.xml?
>
> Thanks.
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> yeah... there's not much there:
>>
>> -bash-4.1$ cd nm-local-dir/
>> -bash-4.1$ ll *
>> filecache:
>> total 0
>>
>> nmPrivate:
>> total 0
>>
>> usercache:
>> total 0
>>
>> I'm using Open JDK, would that be a problem?
>>
>> More log:
>>
>> STARTUP_MSG:   java = 1.7.0_75
>> ************************************************************/
>> 2015-04-19 14:38:58,168 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>> signal handlers for [TERM, HUP, INT]
>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2015-04-19 14:38:59,018 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>> 2015-04-19 14:38:59,020 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> 2015-04-19 14:38:59,022 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> 2015-04-19 14:38:59,023 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>> class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> 2015-04-19 14:38:59,109 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system started
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> per directory file limit = 8192
>> 2015-04-19 14:38:59,227 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> 2015-04-19 14:38:59,248 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>> 'httpshuffle'. Because these are not the same tools trying to send
>> ServiceData and read Service Meta Data may have issues unless the refer to
>> the name in the config.
>> 2015-04-19 14:38:59,248 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorProcessTree : null
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Physical memory check enabled: true
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Virtual memory check enabled: true
>> 2015-04-19 14:38:59,284 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> NodeManager configured with 14 G physical memory allocated to containers,
>> which is more than 80% of the total physical memory available (14.7 G).
>> Thrashing might happen.
>> 2015-04-19 14:38:59,287 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Initialized nodemanager for null: physical-memory=14336
>> virtual-memory=30106 virtual-cores=8
>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 38230
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>> server
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Blocking new container-requests as container manager rpc server is still
>> starting.
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 38230: starting
>> 2015-04-19 14:38:59,366 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 8040
>> 2015-04-19 14:38:59,376 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol
>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>> server
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 8040: starting
>> 2015-04-19 14:38:59,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer started on port 8040
>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>> httpshuffle listening on port 13562
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>> NMWebApp at 0.0.0.0:8042
>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.nodemanager is not defined
>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context node
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context logs
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /node/*
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /ws/*
>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 8042
>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /node started at 8042
>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>> 2015-04-19 14:39:00,126 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>> out 0 NM container statuses: []
>> 2015-04-19 14:39:00,131 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Registering with RM using containers :[]
>> 2015-04-19 14:39:00,176 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Rolling master-key for container-tokens, got key with id -543066608
>> 2015-04-19 14:39:00,178 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>> Rolling master-key for container-tokens, got key with id -1243797706
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>> resource of <memory:14336, vCores:8>
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>> ContainerManager to unblock new container-requests
>> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
>> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>> 2015-04-19 19:22:17,807 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Start request for container_1429450734039_0010_01_000001 by user nobody
>> 2015-04-19 19:22:17,828 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Creating a new application reference for app application_1429450734039_0010
>> 2015-04-19 19:22:17,834 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>> IP=10.100.66.251 OPERATION=Start Container Request
>> TARGET=ContainerManageImpl RESULT=SUCCESS
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from NEW to INITING
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Adding container_1429450734039_0010_01_000001 to application
>> application_1429450734039_0010
>> 2015-04-19 19:22:17,839 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from INITING to
>> RUNNING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>> LOCALIZING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>> event CONTAINER_INIT for appId application_1429450734039_0010
>> 2015-04-19 19:22:17,876 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Created localizer for container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,880 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>> 2015-04-19 19:22:17,882 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZING to LOCALIZATION_FAILED
>> 2015-04-19 19:22:17,886 WARN
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>> failed with state: LOCALIZATION_FAILED
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZATION_FAILED to DONE
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Removing container_1429450734039_0010_01_000001 from application
>> application_1429450734039_0010
>>
>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>> brahmareddy.battula@hotmail.com> wrote:
>>
>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>> or mapred.local.dir)..
>>>
>>> can you able provide full logs..?
>>>
>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>> details..
>>>
>>> ------------------------------
>>> From: wget.null@gmail.com
>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>> pi example
>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>> To: user@hadoop.apache.org
>>>
>>>
>>> As you said, that looks like a config issue. I would spot on the NM's
>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>
>>> But without a complete stack trace, its a blind call.
>>>
>>> BR,
>>>  AL
>>>
>>> --
>>> mapredit.blogspot.com
>>>
>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>> Hey All,
>>>     It's me again with another noob question: I deployed a cluster (HA
>>> mode) everything looked good but when I tried to run the pi example:
>>>
>>>  bin/hadoop jar
>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>
>>> the same error occurs if I try to generate data with teragen 100000000
>>> /test/data
>>>
>>>
>>> 2015-04-18 15:49:04,090 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>
>>>
>>> I'm guessing it's a configuration issue but I don't know what am I
>>> missing :S
>>>
>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I didn't specify it so it's using the default value (in /tmp)

On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:

> Hi,
>
> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
> provide that part of yarn-site.xml?
>
> Thanks.
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> yeah... there's not much there:
>>
>> -bash-4.1$ cd nm-local-dir/
>> -bash-4.1$ ll *
>> filecache:
>> total 0
>>
>> nmPrivate:
>> total 0
>>
>> usercache:
>> total 0
>>
>> I'm using Open JDK, would that be a problem?
>>
>> More log:
>>
>> STARTUP_MSG:   java = 1.7.0_75
>> ************************************************************/
>> 2015-04-19 14:38:58,168 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>> signal handlers for [TERM, HUP, INT]
>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2015-04-19 14:38:59,018 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>> 2015-04-19 14:38:59,020 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> 2015-04-19 14:38:59,022 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> 2015-04-19 14:38:59,023 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>> class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> 2015-04-19 14:38:59,109 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system started
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> per directory file limit = 8192
>> 2015-04-19 14:38:59,227 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> 2015-04-19 14:38:59,248 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>> 'httpshuffle'. Because these are not the same tools trying to send
>> ServiceData and read Service Meta Data may have issues unless the refer to
>> the name in the config.
>> 2015-04-19 14:38:59,248 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorProcessTree : null
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Physical memory check enabled: true
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Virtual memory check enabled: true
>> 2015-04-19 14:38:59,284 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> NodeManager configured with 14 G physical memory allocated to containers,
>> which is more than 80% of the total physical memory available (14.7 G).
>> Thrashing might happen.
>> 2015-04-19 14:38:59,287 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Initialized nodemanager for null: physical-memory=14336
>> virtual-memory=30106 virtual-cores=8
>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 38230
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>> server
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Blocking new container-requests as container manager rpc server is still
>> starting.
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 38230: starting
>> 2015-04-19 14:38:59,366 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 8040
>> 2015-04-19 14:38:59,376 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol
>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>> server
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 8040: starting
>> 2015-04-19 14:38:59,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer started on port 8040
>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>> httpshuffle listening on port 13562
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>> NMWebApp at 0.0.0.0:8042
>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.nodemanager is not defined
>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context node
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context logs
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /node/*
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /ws/*
>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 8042
>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /node started at 8042
>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>> 2015-04-19 14:39:00,126 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>> out 0 NM container statuses: []
>> 2015-04-19 14:39:00,131 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Registering with RM using containers :[]
>> 2015-04-19 14:39:00,176 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Rolling master-key for container-tokens, got key with id -543066608
>> 2015-04-19 14:39:00,178 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>> Rolling master-key for container-tokens, got key with id -1243797706
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>> resource of <memory:14336, vCores:8>
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>> ContainerManager to unblock new container-requests
>> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
>> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>> 2015-04-19 19:22:17,807 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Start request for container_1429450734039_0010_01_000001 by user nobody
>> 2015-04-19 19:22:17,828 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Creating a new application reference for app application_1429450734039_0010
>> 2015-04-19 19:22:17,834 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>> IP=10.100.66.251 OPERATION=Start Container Request
>> TARGET=ContainerManageImpl RESULT=SUCCESS
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from NEW to INITING
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Adding container_1429450734039_0010_01_000001 to application
>> application_1429450734039_0010
>> 2015-04-19 19:22:17,839 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from INITING to
>> RUNNING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>> LOCALIZING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>> event CONTAINER_INIT for appId application_1429450734039_0010
>> 2015-04-19 19:22:17,876 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Created localizer for container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,880 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>> 2015-04-19 19:22:17,882 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZING to LOCALIZATION_FAILED
>> 2015-04-19 19:22:17,886 WARN
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>> failed with state: LOCALIZATION_FAILED
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZATION_FAILED to DONE
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Removing container_1429450734039_0010_01_000001 from application
>> application_1429450734039_0010
>>
>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>> brahmareddy.battula@hotmail.com> wrote:
>>
>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>> or mapred.local.dir)..
>>>
>>> can you able provide full logs..?
>>>
>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>> details..
>>>
>>> ------------------------------
>>> From: wget.null@gmail.com
>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>> pi example
>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>> To: user@hadoop.apache.org
>>>
>>>
>>> As you said, that looks like a config issue. I would spot on the NM's
>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>
>>> But without a complete stack trace, its a blind call.
>>>
>>> BR,
>>>  AL
>>>
>>> --
>>> mapredit.blogspot.com
>>>
>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>> Hey All,
>>>     It's me again with another noob question: I deployed a cluster (HA
>>> mode) everything looked good but when I tried to run the pi example:
>>>
>>>  bin/hadoop jar
>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>
>>> the same error occurs if I try to generate data with teragen 100000000
>>> /test/data
>>>
>>>
>>> 2015-04-18 15:49:04,090 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>
>>>
>>> I'm guessing it's a configuration issue but I don't know what am I
>>> missing :S
>>>
>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
I didn't specify it so it's using the default value (in /tmp)

On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <dr...@nexr.com> wrote:

> Hi,
>
> guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
> provide that part of yarn-site.xml?
>
> Thanks.
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:
>
>> yeah... there's not much there:
>>
>> -bash-4.1$ cd nm-local-dir/
>> -bash-4.1$ ll *
>> filecache:
>> total 0
>>
>> nmPrivate:
>> total 0
>>
>> usercache:
>> total 0
>>
>> I'm using Open JDK, would that be a problem?
>>
>> More log:
>>
>> STARTUP_MSG:   java = 1.7.0_75
>> ************************************************************/
>> 2015-04-19 14:38:58,168 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
>> signal handlers for [TERM, HUP, INT]
>> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 2015-04-19 14:38:59,018 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
>> 2015-04-19 14:38:59,020 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
>> 2015-04-19 14:38:59,021 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
>> 2015-04-19 14:38:59,022 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> 2015-04-19 14:38:59,023 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
>> class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
>> 2015-04-19 14:38:59,054 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>> 2015-04-19 14:38:59,109 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2015-04-19 14:38:59,197 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
>> system started
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
>> 2015-04-19 14:38:59,217 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> per directory file limit = 8192
>> 2015-04-19 14:38:59,227 INFO
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
>> for class
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
>> 2015-04-19 14:38:59,248 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
>> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
>> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
>> 'httpshuffle'. Because these are not the same tools trying to send
>> ServiceData and read Service Meta Data may have issues unless the refer to
>> the name in the config.
>> 2015-04-19 14:38:59,248 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
>> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>>  Using ResourceCalculatorProcessTree : null
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Physical memory check enabled: true
>> 2015-04-19 14:38:59,281 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> Virtual memory check enabled: true
>> 2015-04-19 14:38:59,284 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>> NodeManager configured with 14 G physical memory allocated to containers,
>> which is more than 80% of the total physical memory available (14.7 G).
>> Thrashing might happen.
>> 2015-04-19 14:38:59,287 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Initialized nodemanager for null: physical-memory=14336
>> virtual-memory=30106 virtual-cores=8
>> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 38230
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
>> server
>> 2015-04-19 14:38:59,359 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Blocking new container-requests as container manager rpc server is still
>> starting.
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 38230: starting
>> 2015-04-19 14:38:59,366 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Updating node address : ip-10-100-70-199.ec2.internal:38230
>> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager:
>> Using callQueue class java.util.concurrent.LinkedBlockingQueue
>> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 8040
>> 2015-04-19 14:38:59,376 INFO
>> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
>> protocol
>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
>> server
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> Responder: starting
>> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
>> listener on 8040: starting
>> 2015-04-19 14:38:59,380 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer started on port 8040
>> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
>> IndexCache created with max memory = 10485760
>> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
>> httpshuffle listening on port 13562
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> ContainerManager bound to 0.0.0.0/0.0.0.0:0
>> 2015-04-19 14:38:59,405 INFO
>> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
>> NMWebApp at 0.0.0.0:8042
>> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.nodemanager is not defined
>> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context node
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context logs
>> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /node/*
>> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
>> path spec: /ws/*
>> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 8042
>> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
>> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
>> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
>> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
>> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
>> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /node started at 8042
>> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>> 2015-04-19 14:39:00,126 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
>> out 0 NM container statuses: []
>> 2015-04-19 14:39:00,131 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> Registering with RM using containers :[]
>> 2015-04-19 14:39:00,176 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>> Rolling master-key for container-tokens, got key with id -543066608
>> 2015-04-19 14:39:00,178 INFO
>> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
>> Rolling master-key for container-tokens, got key with id -1243797706
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
>> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
>> resource of <memory:14336, vCores:8>
>> 2015-04-19 14:39:00,179 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
>> ContainerManager to unblock new container-requests
>> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
>> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
>> 2015-04-19 19:22:17,807 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Start request for container_1429450734039_0010_01_000001 by user nobody
>> 2015-04-19 19:22:17,828 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Creating a new application reference for app application_1429450734039_0010
>> 2015-04-19 19:22:17,834 INFO
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
>> IP=10.100.66.251 OPERATION=Start Container Request
>> TARGET=ContainerManageImpl RESULT=SUCCESS
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from NEW to INITING
>> 2015-04-19 19:22:17,835 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Adding container_1429450734039_0010_01_000001 to application
>> application_1429450734039_0010
>> 2015-04-19 19:22:17,839 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Application application_1429450734039_0010 transitioned from INITING to
>> RUNNING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from NEW to
>> LOCALIZING
>> 2015-04-19 19:22:17,843 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
>> event CONTAINER_INIT for appId application_1429450734039_0010
>> 2015-04-19 19:22:17,876 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> Resource
>> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
>> transitioned from INIT to DOWNLOADING
>> 2015-04-19 19:22:17,877 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Created localizer for container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,880 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>> 2015-04-19 19:22:17,882 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZING to LOCALIZATION_FAILED
>> 2015-04-19 19:22:17,886 WARN
>> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
>> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
>> failed with state: LOCALIZATION_FAILED
>> APPID=application_1429450734039_0010
>> CONTAINERID=container_1429450734039_0010_01_000001
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Container container_1429450734039_0010_01_000001 transitioned from
>> LOCALIZATION_FAILED to DONE
>> 2015-04-19 19:22:17,889 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>> Removing container_1429450734039_0010_01_000001 from application
>> application_1429450734039_0010
>>
>> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
>> brahmareddy.battula@hotmail.com> wrote:
>>
>>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>>> or mapred.local.dir)..
>>>
>>> can you able provide full logs..?
>>>
>>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>>> details..
>>>
>>> ------------------------------
>>> From: wget.null@gmail.com
>>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>>> pi example
>>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>>> To: user@hadoop.apache.org
>>>
>>>
>>> As you said, that looks like a config issue. I would spot on the NM's
>>> local scratch dir (yarn.nodemanager.local-dirs).
>>>
>>> But without a complete stack trace, its a blind call.
>>>
>>> BR,
>>>  AL
>>>
>>> --
>>> mapredit.blogspot.com
>>>
>>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>>
>>> Hey All,
>>>     It's me again with another noob question: I deployed a cluster (HA
>>> mode) everything looked good but when I tried to run the pi example:
>>>
>>>  bin/hadoop jar
>>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>>
>>> the same error occurs if I try to generate data with teragen 100000000
>>> /test/data
>>>
>>>
>>> 2015-04-18 15:49:04,090 INFO
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> Localizer failed
>>> java.lang.NullPointerException
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>>> at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>>
>>>
>>> I'm guessing it's a configuration issue but I don't know what am I
>>> missing :S
>>>
>>>
>>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Drake민영근 <dr...@nexr.com>.
Hi,

guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
provide that part of yarn-site.xml?

Thanks.

Drake 민영근 Ph.D
kt NexR

On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:

> yeah... there's not much there:
>
> -bash-4.1$ cd nm-local-dir/
> -bash-4.1$ ll *
> filecache:
> total 0
>
> nmPrivate:
> total 0
>
> usercache:
> total 0
>
> I'm using Open JDK, would that be a problem?
>
> More log:
>
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-19 14:38:58,168 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
> 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
> 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
> class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> 2015-04-19 14:38:59,109 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
> system started
> 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
> 2015-04-19 14:38:59,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> per directory file limit = 8192
> 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
> 2015-04-19 14:38:59,248 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
> 'httpshuffle'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2015-04-19 14:38:59,248 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorPlugin :
> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorProcessTree : null
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Physical memory check enabled: true
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Virtual memory check enabled: true
> 2015-04-19 14:38:59,284 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> NodeManager configured with 14 G physical memory allocated to containers,
> which is more than 80% of the total physical memory available (14.7 G).
> Thrashing might happen.
> 2015-04-19 14:38:59,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Initialized nodemanager for null: physical-memory=14336
> virtual-memory=30106 virtual-cores=8
> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 38230
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
> server
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Blocking new container-requests as container manager rpc server is still
> starting.
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 38230: starting
> 2015-04-19 14:38:59,366 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Updating node address : ip-10-100-70-199.ec2.internal:38230
> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 8040
> 2015-04-19 14:38:59,376 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol
> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
> server
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8040: starting
> 2015-04-19 14:38:59,380 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer started on port 8040
> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
> httpshuffle listening on port 13562
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager bound to 0.0.0.0/0.0.0.0:0
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:8042
> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.nodemanager is not defined
> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context node
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /node/*
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /ws/*
> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 8042
> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /node started at 8042
> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2015-04-19 14:39:00,126 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
> out 0 NM container statuses: []
> 2015-04-19 14:39:00,131 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Registering with RM using containers :[]
> 2015-04-19 14:39:00,176 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Rolling master-key for container-tokens, got key with id -543066608
> 2015-04-19 14:39:00,178 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
> Rolling master-key for container-tokens, got key with id -1243797706
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
> resource of <memory:14336, vCores:8>
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
> ContainerManager to unblock new container-requests
> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
> 2015-04-19 19:22:17,807 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1429450734039_0010_01_000001 by user nobody
> 2015-04-19 19:22:17,828 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1429450734039_0010
> 2015-04-19 19:22:17,834 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
> IP=10.100.66.251 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from NEW to INITING
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Adding container_1429450734039_0010_01_000001 to application
> application_1429450734039_0010
> 2015-04-19 19:22:17,839 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from INITING to
> RUNNING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from NEW to
> LOCALIZING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1429450734039_0010
> 2015-04-19 19:22:17,876 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,880 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 2015-04-19 19:22:17,882 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2015-04-19 19:22:17,886 WARN
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
> failed with state: LOCALIZATION_FAILED
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZATION_FAILED to DONE
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_1429450734039_0010_01_000001 from application
> application_1429450734039_0010
>
> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
> brahmareddy.battula@hotmail.com> wrote:
>
>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>> or mapred.local.dir)..
>>
>> can you able provide full logs..?
>>
>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>> details..
>>
>> ------------------------------
>> From: wget.null@gmail.com
>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>> pi example
>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>> To: user@hadoop.apache.org
>>
>>
>> As you said, that looks like a config issue. I would spot on the NM's
>> local scratch dir (yarn.nodemanager.local-dirs).
>>
>> But without a complete stack trace, its a blind call.
>>
>> BR,
>>  AL
>>
>> --
>> mapredit.blogspot.com
>>
>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>> Hey All,
>>     It's me again with another noob question: I deployed a cluster (HA
>> mode) everything looked good but when I tried to run the pi example:
>>
>>  bin/hadoop jar
>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>
>> the same error occurs if I try to generate data with teragen 100000000
>> /test/data
>>
>>
>> 2015-04-18 15:49:04,090 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>
>>
>> I'm guessing it's a configuration issue but I don't know what am I
>> missing :S
>>
>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Drake민영근 <dr...@nexr.com>.
Hi,

guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
provide that part of yarn-site.xml?

Thanks.

Drake 민영근 Ph.D
kt NexR

On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:

> yeah... there's not much there:
>
> -bash-4.1$ cd nm-local-dir/
> -bash-4.1$ ll *
> filecache:
> total 0
>
> nmPrivate:
> total 0
>
> usercache:
> total 0
>
> I'm using Open JDK, would that be a problem?
>
> More log:
>
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-19 14:38:58,168 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
> 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
> 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
> class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> 2015-04-19 14:38:59,109 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
> system started
> 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
> 2015-04-19 14:38:59,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> per directory file limit = 8192
> 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
> 2015-04-19 14:38:59,248 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
> 'httpshuffle'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2015-04-19 14:38:59,248 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorPlugin :
> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorProcessTree : null
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Physical memory check enabled: true
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Virtual memory check enabled: true
> 2015-04-19 14:38:59,284 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> NodeManager configured with 14 G physical memory allocated to containers,
> which is more than 80% of the total physical memory available (14.7 G).
> Thrashing might happen.
> 2015-04-19 14:38:59,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Initialized nodemanager for null: physical-memory=14336
> virtual-memory=30106 virtual-cores=8
> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 38230
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
> server
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Blocking new container-requests as container manager rpc server is still
> starting.
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 38230: starting
> 2015-04-19 14:38:59,366 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Updating node address : ip-10-100-70-199.ec2.internal:38230
> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 8040
> 2015-04-19 14:38:59,376 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol
> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
> server
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8040: starting
> 2015-04-19 14:38:59,380 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer started on port 8040
> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
> httpshuffle listening on port 13562
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager bound to 0.0.0.0/0.0.0.0:0
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:8042
> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.nodemanager is not defined
> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context node
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /node/*
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /ws/*
> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 8042
> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /node started at 8042
> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2015-04-19 14:39:00,126 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
> out 0 NM container statuses: []
> 2015-04-19 14:39:00,131 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Registering with RM using containers :[]
> 2015-04-19 14:39:00,176 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Rolling master-key for container-tokens, got key with id -543066608
> 2015-04-19 14:39:00,178 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
> Rolling master-key for container-tokens, got key with id -1243797706
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
> resource of <memory:14336, vCores:8>
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
> ContainerManager to unblock new container-requests
> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
> 2015-04-19 19:22:17,807 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1429450734039_0010_01_000001 by user nobody
> 2015-04-19 19:22:17,828 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1429450734039_0010
> 2015-04-19 19:22:17,834 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
> IP=10.100.66.251 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from NEW to INITING
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Adding container_1429450734039_0010_01_000001 to application
> application_1429450734039_0010
> 2015-04-19 19:22:17,839 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from INITING to
> RUNNING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from NEW to
> LOCALIZING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1429450734039_0010
> 2015-04-19 19:22:17,876 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,880 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 2015-04-19 19:22:17,882 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2015-04-19 19:22:17,886 WARN
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
> failed with state: LOCALIZATION_FAILED
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZATION_FAILED to DONE
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_1429450734039_0010_01_000001 from application
> application_1429450734039_0010
>
> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
> brahmareddy.battula@hotmail.com> wrote:
>
>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>> or mapred.local.dir)..
>>
>> can you able provide full logs..?
>>
>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>> details..
>>
>> ------------------------------
>> From: wget.null@gmail.com
>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>> pi example
>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>> To: user@hadoop.apache.org
>>
>>
>> As you said, that looks like a config issue. I would spot on the NM's
>> local scratch dir (yarn.nodemanager.local-dirs).
>>
>> But without a complete stack trace, its a blind call.
>>
>> BR,
>>  AL
>>
>> --
>> mapredit.blogspot.com
>>
>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>> Hey All,
>>     It's me again with another noob question: I deployed a cluster (HA
>> mode) everything looked good but when I tried to run the pi example:
>>
>>  bin/hadoop jar
>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>
>> the same error occurs if I try to generate data with teragen 100000000
>> /test/data
>>
>>
>> 2015-04-18 15:49:04,090 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>
>>
>> I'm guessing it's a configuration issue but I don't know what am I
>> missing :S
>>
>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Drake민영근 <dr...@nexr.com>.
Hi,

guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
provide that part of yarn-site.xml?

Thanks.

Drake 민영근 Ph.D
kt NexR

On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:

> yeah... there's not much there:
>
> -bash-4.1$ cd nm-local-dir/
> -bash-4.1$ ll *
> filecache:
> total 0
>
> nmPrivate:
> total 0
>
> usercache:
> total 0
>
> I'm using Open JDK, would that be a problem?
>
> More log:
>
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-19 14:38:58,168 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
> 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
> 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
> class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> 2015-04-19 14:38:59,109 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
> system started
> 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
> 2015-04-19 14:38:59,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> per directory file limit = 8192
> 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
> 2015-04-19 14:38:59,248 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
> 'httpshuffle'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2015-04-19 14:38:59,248 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorPlugin :
> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorProcessTree : null
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Physical memory check enabled: true
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Virtual memory check enabled: true
> 2015-04-19 14:38:59,284 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> NodeManager configured with 14 G physical memory allocated to containers,
> which is more than 80% of the total physical memory available (14.7 G).
> Thrashing might happen.
> 2015-04-19 14:38:59,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Initialized nodemanager for null: physical-memory=14336
> virtual-memory=30106 virtual-cores=8
> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 38230
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
> server
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Blocking new container-requests as container manager rpc server is still
> starting.
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 38230: starting
> 2015-04-19 14:38:59,366 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Updating node address : ip-10-100-70-199.ec2.internal:38230
> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 8040
> 2015-04-19 14:38:59,376 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol
> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
> server
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8040: starting
> 2015-04-19 14:38:59,380 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer started on port 8040
> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
> httpshuffle listening on port 13562
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager bound to 0.0.0.0/0.0.0.0:0
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:8042
> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.nodemanager is not defined
> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context node
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /node/*
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /ws/*
> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 8042
> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /node started at 8042
> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2015-04-19 14:39:00,126 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
> out 0 NM container statuses: []
> 2015-04-19 14:39:00,131 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Registering with RM using containers :[]
> 2015-04-19 14:39:00,176 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Rolling master-key for container-tokens, got key with id -543066608
> 2015-04-19 14:39:00,178 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
> Rolling master-key for container-tokens, got key with id -1243797706
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
> resource of <memory:14336, vCores:8>
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
> ContainerManager to unblock new container-requests
> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
> 2015-04-19 19:22:17,807 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1429450734039_0010_01_000001 by user nobody
> 2015-04-19 19:22:17,828 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1429450734039_0010
> 2015-04-19 19:22:17,834 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
> IP=10.100.66.251 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from NEW to INITING
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Adding container_1429450734039_0010_01_000001 to application
> application_1429450734039_0010
> 2015-04-19 19:22:17,839 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from INITING to
> RUNNING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from NEW to
> LOCALIZING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1429450734039_0010
> 2015-04-19 19:22:17,876 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,880 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 2015-04-19 19:22:17,882 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2015-04-19 19:22:17,886 WARN
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
> failed with state: LOCALIZATION_FAILED
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZATION_FAILED to DONE
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_1429450734039_0010_01_000001 from application
> application_1429450734039_0010
>
> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
> brahmareddy.battula@hotmail.com> wrote:
>
>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>> or mapred.local.dir)..
>>
>> can you able provide full logs..?
>>
>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>> details..
>>
>> ------------------------------
>> From: wget.null@gmail.com
>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>> pi example
>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>> To: user@hadoop.apache.org
>>
>>
>> As you said, that looks like a config issue. I would spot on the NM's
>> local scratch dir (yarn.nodemanager.local-dirs).
>>
>> But without a complete stack trace, its a blind call.
>>
>> BR,
>>  AL
>>
>> --
>> mapredit.blogspot.com
>>
>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>> Hey All,
>>     It's me again with another noob question: I deployed a cluster (HA
>> mode) everything looked good but when I tried to run the pi example:
>>
>>  bin/hadoop jar
>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>
>> the same error occurs if I try to generate data with teragen 100000000
>> /test/data
>>
>>
>> 2015-04-18 15:49:04,090 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>
>>
>> I'm guessing it's a configuration issue but I don't know what am I
>> missing :S
>>
>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Drake민영근 <dr...@nexr.com>.
Hi,

guess the "yarn.nodemanager.local-dirs" property is the problem. Can you
provide that part of yarn-site.xml?

Thanks.

Drake 민영근 Ph.D
kt NexR

On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <fo...@gmail.com> wrote:

> yeah... there's not much there:
>
> -bash-4.1$ cd nm-local-dir/
> -bash-4.1$ ll *
> filecache:
> total 0
>
> nmPrivate:
> total 0
>
> usercache:
> total 0
>
> I'm using Open JDK, would that be a problem?
>
> More log:
>
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-19 14:38:58,168 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
> 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
> 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
> 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
> class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
> 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> 2015-04-19 14:38:59,109 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-19 14:38:59,197 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
> system started
> 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
> 2015-04-19 14:38:59,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> per directory file limit = 8192
> 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Registering class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
> for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
> 2015-04-19 14:38:59,248 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
> 'httpshuffle'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2015-04-19 14:38:59,248 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Adding auxiliary service httpshuffle, "mapreduce_shuffle"
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorPlugin :
> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Using ResourceCalculatorProcessTree : null
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Physical memory check enabled: true
> 2015-04-19 14:38:59,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Virtual memory check enabled: true
> 2015-04-19 14:38:59,284 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> NodeManager configured with 14 G physical memory allocated to containers,
> which is more than 80% of the total physical memory available (14.7 G).
> Thrashing might happen.
> 2015-04-19 14:38:59,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Initialized nodemanager for null: physical-memory=14336
> virtual-memory=30106 virtual-cores=8
> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 38230
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
> server
> 2015-04-19 14:38:59,359 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Blocking new container-requests as container manager rpc server is still
> starting.
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 38230: starting
> 2015-04-19 14:38:59,366 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Updating node address : ip-10-100-70-199.ec2.internal:38230
> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
> callQueue class java.util.concurrent.LinkedBlockingQueue
> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 8040
> 2015-04-19 14:38:59,376 INFO
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol
> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
> server
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8040: starting
> 2015-04-19 14:38:59,380 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer started on port 8040
> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
> httpshuffle listening on port 13562
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> ContainerManager bound to 0.0.0.0/0.0.0.0:0
> 2015-04-19 14:38:59,405 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:8042
> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.nodemanager is not defined
> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context node
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /node/*
> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
> path spec: /ws/*
> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 8042
> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /node started at 8042
> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2015-04-19 14:39:00,126 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
> out 0 NM container statuses: []
> 2015-04-19 14:39:00,131 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> Registering with RM using containers :[]
> 2015-04-19 14:39:00,176 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
> Rolling master-key for container-tokens, got key with id -543066608
> 2015-04-19 14:39:00,178 INFO
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
> Rolling master-key for container-tokens, got key with id -1243797706
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
> resource of <memory:14336, vCores:8>
> 2015-04-19 14:39:00,179 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
> ContainerManager to unblock new container-requests
> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
> 2015-04-19 19:22:17,807 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1429450734039_0010_01_000001 by user nobody
> 2015-04-19 19:22:17,828 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1429450734039_0010
> 2015-04-19 19:22:17,834 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
> IP=10.100.66.251 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from NEW to INITING
> 2015-04-19 19:22:17,835 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Adding container_1429450734039_0010_01_000001 to application
> application_1429450734039_0010
> 2015-04-19 19:22:17,839 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1429450734039_0010 transitioned from INITING to
> RUNNING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from NEW to
> LOCALIZING
> 2015-04-19 19:22:17,843 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1429450734039_0010
> 2015-04-19 19:22:17,876 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource
> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
> transitioned from INIT to DOWNLOADING
> 2015-04-19 19:22:17,877 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,880 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 2015-04-19 19:22:17,882 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2015-04-19 19:22:17,886 WARN
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container
> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
> failed with state: LOCALIZATION_FAILED
> APPID=application_1429450734039_0010
> CONTAINERID=container_1429450734039_0010_01_000001
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1429450734039_0010_01_000001 transitioned from
> LOCALIZATION_FAILED to DONE
> 2015-04-19 19:22:17,889 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_1429450734039_0010_01_000001 from application
> application_1429450734039_0010
>
> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
> brahmareddy.battula@hotmail.com> wrote:
>
>> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
>> or mapred.local.dir)..
>>
>> can you able provide full logs..?
>>
>> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
>> details..
>>
>> ------------------------------
>> From: wget.null@gmail.com
>> Subject: Re: ResourceLocalizationService: Localizer failed when running
>> pi example
>> Date: Sun, 19 Apr 2015 17:59:13 +0200
>> To: user@hadoop.apache.org
>>
>>
>> As you said, that looks like a config issue. I would spot on the NM's
>> local scratch dir (yarn.nodemanager.local-dirs).
>>
>> But without a complete stack trace, its a blind call.
>>
>> BR,
>>  AL
>>
>> --
>> mapredit.blogspot.com
>>
>> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>> Hey All,
>>     It's me again with another noob question: I deployed a cluster (HA
>> mode) everything looked good but when I tried to run the pi example:
>>
>>  bin/hadoop jar
>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>>
>> the same error occurs if I try to generate data with teragen 100000000
>> /test/data
>>
>>
>> 2015-04-18 15:49:04,090 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Localizer failed
>> java.lang.NullPointerException
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>>
>>
>> I'm guessing it's a configuration issue but I don't know what am I
>> missing :S
>>
>>
>>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
yeah... there's not much there:

-bash-4.1$ cd nm-local-dir/
-bash-4.1$ ll *
filecache:
total 0

nmPrivate:
total 0

usercache:
total 0

I'm using Open JDK, would that be a problem?

More log:

STARTUP_MSG:   java = 1.7.0_75
************************************************************/
2015-04-19 14:38:58,168 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
signal handlers for [TERM, HUP, INT]
2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
org.apache.hadoop.yarn.server.nodemanager.NodeManager
2015-04-19 14:38:59,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
system started
2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-04-19 14:38:59,217 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
per directory file limit = 8192
2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-04-19 14:38:59,248 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send
ServiceData and read Service Meta Data may have issues unless the refer to
the name in the config.
2015-04-19 14:38:59,248 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorPlugin :
org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorProcessTree : null
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Physical memory check enabled: true
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Virtual memory check enabled: true
2015-04-19 14:38:59,284 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
NodeManager configured with 14 G physical memory allocated to containers,
which is more than 80% of the total physical memory available (14.7 G).
Thrashing might happen.
2015-04-19 14:38:59,287 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Initialized nodemanager for null: physical-memory=14336
virtual-memory=30106 virtual-cores=8
2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 38230
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
server
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Blocking new container-requests as container manager rpc server is still
starting.
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 38230: starting
2015-04-19 14:38:59,366 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Updating node address : ip-10-100-70-199.ec2.internal:38230
2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8040
2015-04-19 14:38:59,376 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol
org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
server
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8040: starting
2015-04-19 14:38:59,380 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer started on port 8040
2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
httpshuffle listening on port 13562
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:8042
2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.nodemanager is not defined
2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context node
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /node/*
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 8042
2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 8042
2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-04-19 14:39:00,126 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
out 0 NM container statuses: []
2015-04-19 14:39:00,131 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Registering with RM using containers :[]
2015-04-19 14:39:00,176 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Rolling master-key for container-tokens, got key with id -543066608
2015-04-19 14:39:00,178 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
Rolling master-key for container-tokens, got key with id -1243797706
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
resource of <memory:14336, vCores:8>
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
ContainerManager to unblock new container-requests
2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
2015-04-19 19:22:17,807 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1429450734039_0010_01_000001 by user nobody
2015-04-19 19:22:17,828 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1429450734039_0010
2015-04-19 19:22:17,834 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
IP=10.100.66.251 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from NEW to INITING
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding container_1429450734039_0010_01_000001 to application
application_1429450734039_0010
2015-04-19 19:22:17,839 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from INITING to
RUNNING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from NEW to
LOCALIZING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1429450734039_0010
2015-04-19 19:22:17,876 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1429450734039_0010_01_000001
2015-04-19 19:22:17,880 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
2015-04-19 19:22:17,882 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2015-04-19 19:22:17,886 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
OPERATION=Container
Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
failed with state: LOCALIZATION_FAILED APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZATION_FAILED to DONE
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1429450734039_0010_01_000001 from application
application_1429450734039_0010

On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
brahmareddy.battula@hotmail.com> wrote:

> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
> or mapred.local.dir)..
>
> can you able provide full logs..?
>
> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
> details..
>
> ------------------------------
> From: wget.null@gmail.com
> Subject: Re: ResourceLocalizationService: Localizer failed when running pi
> example
> Date: Sun, 19 Apr 2015 17:59:13 +0200
> To: user@hadoop.apache.org
>
>
> As you said, that looks like a config issue. I would spot on the NM's
> local scratch dir (yarn.nodemanager.local-dirs).
>
> But without a complete stack trace, its a blind call.
>
> BR,
>  AL
>
> --
> mapredit.blogspot.com
>
> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>
>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
yeah... there's not much there:

-bash-4.1$ cd nm-local-dir/
-bash-4.1$ ll *
filecache:
total 0

nmPrivate:
total 0

usercache:
total 0

I'm using Open JDK, would that be a problem?

More log:

STARTUP_MSG:   java = 1.7.0_75
************************************************************/
2015-04-19 14:38:58,168 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
signal handlers for [TERM, HUP, INT]
2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
org.apache.hadoop.yarn.server.nodemanager.NodeManager
2015-04-19 14:38:59,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
system started
2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-04-19 14:38:59,217 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
per directory file limit = 8192
2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-04-19 14:38:59,248 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send
ServiceData and read Service Meta Data may have issues unless the refer to
the name in the config.
2015-04-19 14:38:59,248 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorPlugin :
org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorProcessTree : null
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Physical memory check enabled: true
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Virtual memory check enabled: true
2015-04-19 14:38:59,284 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
NodeManager configured with 14 G physical memory allocated to containers,
which is more than 80% of the total physical memory available (14.7 G).
Thrashing might happen.
2015-04-19 14:38:59,287 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Initialized nodemanager for null: physical-memory=14336
virtual-memory=30106 virtual-cores=8
2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 38230
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
server
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Blocking new container-requests as container manager rpc server is still
starting.
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 38230: starting
2015-04-19 14:38:59,366 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Updating node address : ip-10-100-70-199.ec2.internal:38230
2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8040
2015-04-19 14:38:59,376 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol
org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
server
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8040: starting
2015-04-19 14:38:59,380 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer started on port 8040
2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
httpshuffle listening on port 13562
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:8042
2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.nodemanager is not defined
2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context node
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /node/*
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 8042
2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 8042
2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-04-19 14:39:00,126 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
out 0 NM container statuses: []
2015-04-19 14:39:00,131 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Registering with RM using containers :[]
2015-04-19 14:39:00,176 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Rolling master-key for container-tokens, got key with id -543066608
2015-04-19 14:39:00,178 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
Rolling master-key for container-tokens, got key with id -1243797706
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
resource of <memory:14336, vCores:8>
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
ContainerManager to unblock new container-requests
2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
2015-04-19 19:22:17,807 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1429450734039_0010_01_000001 by user nobody
2015-04-19 19:22:17,828 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1429450734039_0010
2015-04-19 19:22:17,834 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
IP=10.100.66.251 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from NEW to INITING
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding container_1429450734039_0010_01_000001 to application
application_1429450734039_0010
2015-04-19 19:22:17,839 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from INITING to
RUNNING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from NEW to
LOCALIZING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1429450734039_0010
2015-04-19 19:22:17,876 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1429450734039_0010_01_000001
2015-04-19 19:22:17,880 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
2015-04-19 19:22:17,882 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2015-04-19 19:22:17,886 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
OPERATION=Container
Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
failed with state: LOCALIZATION_FAILED APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZATION_FAILED to DONE
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1429450734039_0010_01_000001 from application
application_1429450734039_0010

On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
brahmareddy.battula@hotmail.com> wrote:

> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
> or mapred.local.dir)..
>
> can you able provide full logs..?
>
> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
> details..
>
> ------------------------------
> From: wget.null@gmail.com
> Subject: Re: ResourceLocalizationService: Localizer failed when running pi
> example
> Date: Sun, 19 Apr 2015 17:59:13 +0200
> To: user@hadoop.apache.org
>
>
> As you said, that looks like a config issue. I would spot on the NM's
> local scratch dir (yarn.nodemanager.local-dirs).
>
> But without a complete stack trace, its a blind call.
>
> BR,
>  AL
>
> --
> mapredit.blogspot.com
>
> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>
>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
yeah... there's not much there:

-bash-4.1$ cd nm-local-dir/
-bash-4.1$ ll *
filecache:
total 0

nmPrivate:
total 0

usercache:
total 0

I'm using Open JDK, would that be a problem?

More log:

STARTUP_MSG:   java = 1.7.0_75
************************************************************/
2015-04-19 14:38:58,168 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
signal handlers for [TERM, HUP, INT]
2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
org.apache.hadoop.yarn.server.nodemanager.NodeManager
2015-04-19 14:38:59,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
system started
2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-04-19 14:38:59,217 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
per directory file limit = 8192
2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-04-19 14:38:59,248 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send
ServiceData and read Service Meta Data may have issues unless the refer to
the name in the config.
2015-04-19 14:38:59,248 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorPlugin :
org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorProcessTree : null
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Physical memory check enabled: true
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Virtual memory check enabled: true
2015-04-19 14:38:59,284 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
NodeManager configured with 14 G physical memory allocated to containers,
which is more than 80% of the total physical memory available (14.7 G).
Thrashing might happen.
2015-04-19 14:38:59,287 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Initialized nodemanager for null: physical-memory=14336
virtual-memory=30106 virtual-cores=8
2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 38230
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
server
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Blocking new container-requests as container manager rpc server is still
starting.
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 38230: starting
2015-04-19 14:38:59,366 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Updating node address : ip-10-100-70-199.ec2.internal:38230
2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8040
2015-04-19 14:38:59,376 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol
org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
server
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8040: starting
2015-04-19 14:38:59,380 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer started on port 8040
2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
httpshuffle listening on port 13562
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:8042
2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.nodemanager is not defined
2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context node
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /node/*
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 8042
2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 8042
2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-04-19 14:39:00,126 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
out 0 NM container statuses: []
2015-04-19 14:39:00,131 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Registering with RM using containers :[]
2015-04-19 14:39:00,176 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Rolling master-key for container-tokens, got key with id -543066608
2015-04-19 14:39:00,178 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
Rolling master-key for container-tokens, got key with id -1243797706
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
resource of <memory:14336, vCores:8>
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
ContainerManager to unblock new container-requests
2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
2015-04-19 19:22:17,807 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1429450734039_0010_01_000001 by user nobody
2015-04-19 19:22:17,828 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1429450734039_0010
2015-04-19 19:22:17,834 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
IP=10.100.66.251 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from NEW to INITING
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding container_1429450734039_0010_01_000001 to application
application_1429450734039_0010
2015-04-19 19:22:17,839 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from INITING to
RUNNING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from NEW to
LOCALIZING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1429450734039_0010
2015-04-19 19:22:17,876 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1429450734039_0010_01_000001
2015-04-19 19:22:17,880 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
2015-04-19 19:22:17,882 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2015-04-19 19:22:17,886 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
OPERATION=Container
Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
failed with state: LOCALIZATION_FAILED APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZATION_FAILED to DONE
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1429450734039_0010_01_000001 from application
application_1429450734039_0010

On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
brahmareddy.battula@hotmail.com> wrote:

> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
> or mapred.local.dir)..
>
> can you able provide full logs..?
>
> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
> details..
>
> ------------------------------
> From: wget.null@gmail.com
> Subject: Re: ResourceLocalizationService: Localizer failed when running pi
> example
> Date: Sun, 19 Apr 2015 17:59:13 +0200
> To: user@hadoop.apache.org
>
>
> As you said, that looks like a config issue. I would spot on the NM's
> local scratch dir (yarn.nodemanager.local-dirs).
>
> But without a complete stack trace, its a blind call.
>
> BR,
>  AL
>
> --
> mapredit.blogspot.com
>
> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>
>
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
yeah... there's not much there:

-bash-4.1$ cd nm-local-dir/
-bash-4.1$ ll *
filecache:
total 0

nmPrivate:
total 0

usercache:
total 0

I'm using Open JDK, would that be a problem?

More log:

STARTUP_MSG:   java = 1.7.0_75
************************************************************/
2015-04-19 14:38:58,168 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX
signal handlers for [TERM, HUP, INT]
2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for
class
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class
org.apache.hadoop.yarn.server.nodemanager.NodeManager
2015-04-19 14:38:59,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2015-04-19 14:38:59,197 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics
system started
2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-04-19 14:38:59,217 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
per directory file limit = 8192
2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
for class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2015-04-19 14:38:59,248 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for
class class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send
ServiceData and read Service Meta Data may have issues unless the refer to
the name in the config.
2015-04-19 14:38:59,248 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorPlugin :
org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Using ResourceCalculatorProcessTree : null
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Physical memory check enabled: true
2015-04-19 14:38:59,281 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Virtual memory check enabled: true
2015-04-19 14:38:59,284 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
NodeManager configured with 14 G physical memory allocated to containers,
which is more than 80% of the total physical memory available (14.7 G).
Thrashing might happen.
2015-04-19 14:38:59,287 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Initialized nodemanager for null: physical-memory=14336
virtual-memory=30106 virtual-cores=8
2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 38230
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the
server
2015-04-19 14:38:59,359 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Blocking new container-requests as container manager rpc server is still
starting.
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 38230: starting
2015-04-19 14:38:59,366 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Updating node address : ip-10-100-70-199.ec2.internal:38230
2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8040
2015-04-19 14:38:59,376 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol
org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the
server
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8040: starting
2015-04-19 14:38:59,380 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer started on port 8040
2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler:
httpshuffle listening on port 13562
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager started at datanode-03.prod.com/10.100.70.199:38230
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-04-19 14:38:59,405 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:8042
2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.nodemanager is not defined
2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context node
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /node/*
2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty
bound to port 8042
2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26
2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract
jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node
to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$
SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 8042
2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-04-19 14:39:00,126 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending
out 0 NM container statuses: []
2015-04-19 14:39:00,131 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Registering with RM using containers :[]
2015-04-19 14:39:00,176 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Rolling master-key for container-tokens, got key with id -543066608
2015-04-19 14:39:00,178 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
Rolling master-key for container-tokens, got key with id -1243797706
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total
resource of <memory:14336, vCores:8>
2015-04-19 14:39:00,179 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
ContainerManager to unblock new container-requests
2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE)
2015-04-19 19:22:17,807 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1429450734039_0010_01_000001 by user nobody
2015-04-19 19:22:17,828 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1429450734039_0010
2015-04-19 19:22:17,834 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
IP=10.100.66.251 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from NEW to INITING
2015-04-19 19:22:17,835 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding container_1429450734039_0010_01_000001 to application
application_1429450734039_0010
2015-04-19 19:22:17,839 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1429450734039_0010 transitioned from INITING to
RUNNING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from NEW to
LOCALIZING
2015-04-19 19:22:17,843 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1429450734039_0010
2015-04-19 19:22:17,876 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml
transitioned from INIT to DOWNLOADING
2015-04-19 19:22:17,877 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1429450734039_0010_01_000001
2015-04-19 19:22:17,880 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
2015-04-19 19:22:17,882 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2015-04-19 19:22:17,886 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody
OPERATION=Container
Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container
failed with state: LOCALIZATION_FAILED APPID=application_1429450734039_0010
CONTAINERID=container_1429450734039_0010_01_000001
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1429450734039_0010_01_000001 transitioned from
LOCALIZATION_FAILED to DONE
2015-04-19 19:22:17,889 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1429450734039_0010_01_000001 from application
application_1429450734039_0010

On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula <
brahmareddy.battula@hotmail.com> wrote:

> As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs
> or mapred.local.dir)..
>
> can you able provide full logs..?
>
> Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more
> details..
>
> ------------------------------
> From: wget.null@gmail.com
> Subject: Re: ResourceLocalizationService: Localizer failed when running pi
> example
> Date: Sun, 19 Apr 2015 17:59:13 +0200
> To: user@hadoop.apache.org
>
>
> As you said, that looks like a config issue. I would spot on the NM's
> local scratch dir (yarn.nodemanager.local-dirs).
>
> But without a complete stack trace, its a blind call.
>
> BR,
>  AL
>
> --
> mapredit.blogspot.com
>
> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
>
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>
>
>

RE: ResourceLocalizationService: Localizer failed when running pi example

Posted by Brahma Reddy Battula <br...@hotmail.com>.
As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs or mapred.local.dir)..
can you able provide full logs..?
Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more details..

From: wget.null@gmail.com
Subject: Re: ResourceLocalizationService: Localizer failed when running pi example
Date: Sun, 19 Apr 2015 17:59:13 +0200
To: user@hadoop.apache.org

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs).
But without a complete stack trace, its a blind call.
BR, AL
--mapredit.blogspot.com
On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:Hey All,    It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
 bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100

the same error occurs if I try to generate data with teragen 100000000 /test/data

2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failedjava.lang.NullPointerException	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)

I'm guessing it's a configuration issue but I don't know what am I missing :S

RE: ResourceLocalizationService: Localizer failed when running pi example

Posted by Brahma Reddy Battula <br...@hotmail.com>.
As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs or mapred.local.dir)..
can you able provide full logs..?
Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more details..

From: wget.null@gmail.com
Subject: Re: ResourceLocalizationService: Localizer failed when running pi example
Date: Sun, 19 Apr 2015 17:59:13 +0200
To: user@hadoop.apache.org

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs).
But without a complete stack trace, its a blind call.
BR, AL
--mapredit.blogspot.com
On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:Hey All,    It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
 bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100

the same error occurs if I try to generate data with teragen 100000000 /test/data

2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failedjava.lang.NullPointerException	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)

I'm guessing it's a configuration issue but I don't know what am I missing :S

RE: ResourceLocalizationService: Localizer failed when running pi example

Posted by Brahma Reddy Battula <br...@hotmail.com>.
As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs or mapred.local.dir)..
can you able provide full logs..?
Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more details..

From: wget.null@gmail.com
Subject: Re: ResourceLocalizationService: Localizer failed when running pi example
Date: Sun, 19 Apr 2015 17:59:13 +0200
To: user@hadoop.apache.org

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs).
But without a complete stack trace, its a blind call.
BR, AL
--mapredit.blogspot.com
On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:Hey All,    It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
 bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100

the same error occurs if I try to generate data with teragen 100000000 /test/data

2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failedjava.lang.NullPointerException	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)

I'm guessing it's a configuration issue but I don't know what am I missing :S

RE: ResourceLocalizationService: Localizer failed when running pi example

Posted by Brahma Reddy Battula <br...@hotmail.com>.
As Alexander Alten-Lorenz pointed, it mostly config issue(yarn.nodemanager.local-dirs or mapred.local.dir)..
can you able provide full logs..?
Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more details..

From: wget.null@gmail.com
Subject: Re: ResourceLocalizationService: Localizer failed when running pi example
Date: Sun, 19 Apr 2015 17:59:13 +0200
To: user@hadoop.apache.org

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs).
But without a complete stack trace, its a blind call.
BR, AL
--mapredit.blogspot.com
On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:Hey All,    It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
 bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100

the same error occurs if I try to generate data with teragen 100000000 /test/data

2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failedjava.lang.NullPointerException	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)

I'm guessing it's a configuration issue but I don't know what am I missing :S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). <>

But without a complete stack trace, its a blind call.

BR,
 AL

--
mapredit.blogspot.com

> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
> 
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
> 
>  bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
> 
> the same error occurs if I try to generate data with teragen 100000000 /test/data
> 
> 
> 2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> 	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 
> 
> I'm guessing it's a configuration issue but I don't know what am I missing :S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). <>

But without a complete stack trace, its a blind call.

BR,
 AL

--
mapredit.blogspot.com

> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
> 
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
> 
>  bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
> 
> the same error occurs if I try to generate data with teragen 100000000 /test/data
> 
> 
> 2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> 	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 
> 
> I'm guessing it's a configuration issue but I don't know what am I missing :S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). <>

But without a complete stack trace, its a blind call.

BR,
 AL

--
mapredit.blogspot.com

> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
> 
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
> 
>  bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
> 
> the same error occurs if I try to generate data with teragen 100000000 /test/data
> 
> 
> 2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> 	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 
> 
> I'm guessing it's a configuration issue but I don't know what am I missing :S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
nobody had this issue? :(

On Sat, Apr 18, 2015 at 1:24 PM, Fernando O. <fo...@gmail.com> wrote:

> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). <>

But without a complete stack trace, its a blind call.

BR,
 AL

--
mapredit.blogspot.com

> On Apr 18, 2015, at 6:24 PM, Fernando O. <fo...@gmail.com> wrote:
> 
> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA mode) everything looked good but when I tried to run the pi example:
> 
>  bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
> 
> the same error occurs if I try to generate data with teragen 100000000 /test/data
> 
> 
> 2015-04-18 15:49:04,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> 	at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
> 
> 
> I'm guessing it's a configuration issue but I don't know what am I missing :S

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
nobody had this issue? :(

On Sat, Apr 18, 2015 at 1:24 PM, Fernando O. <fo...@gmail.com> wrote:

> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
nobody had this issue? :(

On Sat, Apr 18, 2015 at 1:24 PM, Fernando O. <fo...@gmail.com> wrote:

> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>

Re: ResourceLocalizationService: Localizer failed when running pi example

Posted by "Fernando O." <fo...@gmail.com>.
nobody had this issue? :(

On Sat, Apr 18, 2015 at 1:24 PM, Fernando O. <fo...@gmail.com> wrote:

> Hey All,
>     It's me again with another noob question: I deployed a cluster (HA
> mode) everything looked good but when I tried to run the pi example:
>
>  bin/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100
>
> the same error occurs if I try to generate data with teragen 100000000
> /test/data
>
>
> 2015-04-18 15:49:04,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)
>
>
> I'm guessing it's a configuration issue but I don't know what am I missing
> :S
>