Fwd: REQUESTING ASSISTANCE IN RESOLVING CONTAINER FAILURE

[JOHN MILLER <jm...@gmail.com>]

*Greetings all*


*its been a while since we corresponded.  Hope all is well on your end.*


*I seem to be having a problem with my AM Container.   For whatever reason,
it crashes 2 to 3 seconds after being launched.   I am actually running a
cascading job when i get this error (it also happens when running mapreduce
in HIVE)  Attached below is a description of my cascading job   Attached
above are the yarn log files and configuration files*








































































*Please advise if u can assist in resolving this
issue ../../../../opt/hadoop/bin/hadoop jar build/libs/impatient.jar
data/rain.txt output/rain16/04/25 13:43:04 INFO planner.HadoopPlanner:
using application jar:
/home/jmill383/Impatient/part1/build/libs/impatient.jar16/04/25 13:43:04
INFO property.AppProps: using app.id <http://app.id>:
643884BEA82048FBAFD2A3519ADB6B2116/04/25 13:43:05 WARN
util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable16/04/25 13:43:05
INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead,
use mapreduce.job.working.dir16/04/25 13:43:05 INFO
Configuration.deprecation: mapred.input.dir is deprecated. Instead, use
mapreduce.input.fileinputformat.inputdir16/04/25 13:43:05 INFO
mapred.FileInputFormat: Total input paths to process : 116/04/25 13:43:05
INFO flow.Flow: [Copy] executed rule registry: MapReduceHadoopRuleRegistry,
completed as: SUCCESS, in: 00:00.04816/04/25 13:43:05 INFO flow.Flow:
[Copy] rule registry: MapReduceHadoopRuleRegistry, supports assembly with
steps: 1, nodes: 116/04/25 13:43:05 INFO flow.Flow: [Copy] rule registry:
MapReduceHadoopRuleRegistry, result was selected using: 'default
comparator: selects plan with fewest steps and fewest nodes'16/04/25
13:43:05 INFO Configuration.deprecation: mapred.used.genericoptionsparser
is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used16/04/25 13:43:05 INFO
Configuration.deprecation: mapred.output.dir is deprecated. Instead, use
mapreduce.output.fileoutputformat.outputdir16/04/25 13:43:05 INFO
Configuration.deprecation: mapred.output.compress is deprecated. Instead,
use mapreduce.output.fileoutputformat.compress16/04/25 13:43:05 INFO
Configuration.deprecation: mapred.output.key.class is deprecated. Instead,
use mapreduce.job.output.key.class16/04/25 13:43:05 INFO
Configuration.deprecation: mapred.output.value.class is deprecated.
Instead, use mapreduce.job.output.value.class16/04/25 13:43:05 INFO
util.Version: Concurrent, Inc - Cascading 3.0.216/04/25 13:43:05 INFO
flow.Flow: [Copy] starting16/04/25 13:43:05 INFO flow.Flow: [Copy]  source:
Hfs["TextDelimited[['doc_id', 'text']]"]["data/rain.txt"]16/04/25 13:43:05
INFO flow.Flow: [Copy]  sink: Hfs["TextDelimited[['doc_id',
'text']]"]["output/rain"]16/04/25 13:43:05 INFO flow.Flow: [Copy]  parallel
execution of steps is enabled: true16/04/25 13:43:05 INFO flow.Flow:
[Copy]  executing total steps: 116/04/25 13:43:05 INFO flow.Flow: [Copy]
allocating management threads: 116/04/25 13:43:05 INFO flow.Flow: [Copy]
starting step: (1/1) output/rain16/04/25 13:43:05 INFO client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:9002
<http://127.0.0.1:9002>16/04/25 13:43:05 INFO client.RMProxy: Connecting to
ResourceManager at localhost/127.0.0.1:9002 <http://127.0.0.1:9002>16/04/25
13:43:06 INFO mapred.FileInputFormat: Total input paths to process :
116/04/25 13:43:06 INFO mapreduce.JobSubmitter: number of splits:216/04/25
13:43:06 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1461603000811_000216/04/25 13:43:06 INFO impl.YarnClientImpl: Submitted
application application_1461603000811_000216/04/25 13:43:06 INFO
mapreduce.Job: The url to track the job:
http://starchild:8088/proxy/application_1461603000811_0002/
<http://starchild:8088/proxy/application_1461603000811_0002/>16/04/25
13:43:06 INFO flow.Flow: [Copy] submitted hadoop job:
job_1461603000811_000216/04/25 13:43:06 INFO flow.Flow: [Copy] tracking
url: http://starchild:8088/proxy/application_1461603000811_0002/
<http://starchild:8088/proxy/application_1461603000811_0002/>16/04/25
13:43:11 WARN flow.Flow: [Copy] hadoop job job_1461603000811_0002 state at
FAILED16/04/25 13:43:11 WARN flow.Flow: [Copy] failure info: Application
application_1461603000811_0002 failed 2 times due to AM Container for
appattempt_1461603000811_0002_000002 exited with  exitCode: 1For more
detailed output, check application tracking
page:http://starchild:8088/proxy/application_1461603000811_0002/Then
<http://starchild:8088/proxy/application_1461603000811_0002/Then>, click on
links to logs of each attempt.Diagnostics: Exception from
container-launch.Container id: container_1461603000811_0002_02_000001Exit
code: 1Stack trace: ExitCodeException exitCode=1:     at
org.apache.hadoop.util.Shell.runCommand(Shell.java:538)    at
org.apache.hadoop.util.Shell.run(Shell.java:455)    at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Container exited with a non-zero
exit code 1Failing this attempt. Failing the application.16/04/25 13:43:11
WARN flow.Flow: [Copy] task completion events identify failed tasks16/04/25
13:43:11 WARN flow.Flow: [Copy] task completion events count: 016/04/25
13:43:11 INFO flow.Flow: [Copy] stopping all jobs16/04/25 13:43:11 INFO
flow.Flow: [Copy] stopping: (1/1) output/rain16/04/25 13:43:11 INFO
flow.Flow: [Copy] stopped all jobs16/04/25 13:43:11 INFO flow.Flow: [Copy]
completed in: 00:05.774Exception in thread "main"
cascading.flow.FlowException: step failed: (1/1) output/rain, step id:
D8A8C1F16C654AAEA532F4BF69565AC9, job id: job_1461603000811_0002, please
see cluster logs for failure messages    at
cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:291)    at
cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:184)    at
cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:146)    at
cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:48)    at
java.util.concurrent.FutureTask.run(FutureTask.java:262)    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)*

*John M*