You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Morbious <kn...@gmail.com> on 2015/03/19 22:46:57 UTC
Yarn setup problems
Hello,
Recently I switched from MR1 to YARN on my cluster.
When I tried some tests like :
> hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
16 10
I got errors like:
15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
attempt_1426770785980_0012_m_000006_2, Status : FAILED
Exception from container-launch.
Container id: container_e61_1426770785980_0012_01_000057
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
--------------------------------------------------------------------------
In nodemanager log there is something like this:
2015-03-19 21:40:00,957 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
code from container container_e61_1426770785980_0012_01_000057 is : 1
2015-03-19 21:40:00,957 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exception from container-launch with container ID:
container_e61_1426770785980_0012_01_000057 and exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
container-launch.
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
container_e61_1426770785980_0012_01_000057
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
ExitCodeException exitCode=1:
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell.run(Shell.java:455)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-03-19 21:40:00,958 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
java.lang.Thread.run(Thread.java:745)
I was trying to find out what went wrong, but couldn't find anything. I
have similiar cluster with the same configuration which runs YARN too
without any problems...
Best regards,
Morbious
Re: Yarn setup problems
Posted by Morbious <kn...@gmail.com>.
Hi,
In container logs I have :
cat
/srv/data/*/yarn/logs/application_1426770785980_0010/container_e61_1426770785980_0010_01_0000*/stderr
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
In command line I get:
Job Finished in 22.395 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://spark-cluster-nn/user/morbious/QuasiMonteCarlo_1426835737919_2089306463/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Best regards,
Morbious
2015-03-20 0:32 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
> Please check the app logs.
>
> On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
> wrote:
>
>> Hello,
>>
>> Recently I switched from MR1 to YARN on my cluster.
>> When I tried some tests like :
>>
>> > hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
>> 16 10
>>
>> I got errors like:
>>
>> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
>> attempt_1426770785980_0012_m_000006_2, Status : FAILED
>> Exception from container-launch.
>> Container id: container_e61_1426770785980_0012_01_000057
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>> --------------------------------------------------------------------------
>>
>> In nodemanager log there is something like this:
>>
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
>> code from container container_e61_1426770785980_0012_01_000057 is : 1
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
>> Exception from container-launch with container ID:
>> container_e61_1426770785980_0012_01_000057 and exit code: 1
>> ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
>> container-launch.
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
>> container_e61_1426770785980_0012_01_000057
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
>> ExitCodeException exitCode=1:
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.run(Shell.java:455)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.lang.Thread.run(Thread.java:745)
>>
>>
>> I was trying to find out what went wrong, but couldn't find anything. I
>> have similiar cluster with the same configuration which runs YARN too
>> without any problems...
>>
>> Best regards,
>>
>> Morbious
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Yarn setup problems
Posted by Morbious <kn...@gmail.com>.
Hi,
In container logs I have :
cat
/srv/data/*/yarn/logs/application_1426770785980_0010/container_e61_1426770785980_0010_01_0000*/stderr
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
In command line I get:
Job Finished in 22.395 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://spark-cluster-nn/user/morbious/QuasiMonteCarlo_1426835737919_2089306463/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Best regards,
Morbious
2015-03-20 0:32 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
> Please check the app logs.
>
> On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
> wrote:
>
>> Hello,
>>
>> Recently I switched from MR1 to YARN on my cluster.
>> When I tried some tests like :
>>
>> > hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
>> 16 10
>>
>> I got errors like:
>>
>> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
>> attempt_1426770785980_0012_m_000006_2, Status : FAILED
>> Exception from container-launch.
>> Container id: container_e61_1426770785980_0012_01_000057
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>> --------------------------------------------------------------------------
>>
>> In nodemanager log there is something like this:
>>
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
>> code from container container_e61_1426770785980_0012_01_000057 is : 1
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
>> Exception from container-launch with container ID:
>> container_e61_1426770785980_0012_01_000057 and exit code: 1
>> ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
>> container-launch.
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
>> container_e61_1426770785980_0012_01_000057
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
>> ExitCodeException exitCode=1:
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.run(Shell.java:455)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.lang.Thread.run(Thread.java:745)
>>
>>
>> I was trying to find out what went wrong, but couldn't find anything. I
>> have similiar cluster with the same configuration which runs YARN too
>> without any problems...
>>
>> Best regards,
>>
>> Morbious
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Yarn setup problems
Posted by Morbious <kn...@gmail.com>.
Hi,
In container logs I have :
cat
/srv/data/*/yarn/logs/application_1426770785980_0010/container_e61_1426770785980_0010_01_0000*/stderr
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
In command line I get:
Job Finished in 22.395 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://spark-cluster-nn/user/morbious/QuasiMonteCarlo_1426835737919_2089306463/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Best regards,
Morbious
2015-03-20 0:32 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
> Please check the app logs.
>
> On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
> wrote:
>
>> Hello,
>>
>> Recently I switched from MR1 to YARN on my cluster.
>> When I tried some tests like :
>>
>> > hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
>> 16 10
>>
>> I got errors like:
>>
>> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
>> attempt_1426770785980_0012_m_000006_2, Status : FAILED
>> Exception from container-launch.
>> Container id: container_e61_1426770785980_0012_01_000057
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>> --------------------------------------------------------------------------
>>
>> In nodemanager log there is something like this:
>>
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
>> code from container container_e61_1426770785980_0012_01_000057 is : 1
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
>> Exception from container-launch with container ID:
>> container_e61_1426770785980_0012_01_000057 and exit code: 1
>> ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
>> container-launch.
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
>> container_e61_1426770785980_0012_01_000057
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
>> ExitCodeException exitCode=1:
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.run(Shell.java:455)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.lang.Thread.run(Thread.java:745)
>>
>>
>> I was trying to find out what went wrong, but couldn't find anything. I
>> have similiar cluster with the same configuration which runs YARN too
>> without any problems...
>>
>> Best regards,
>>
>> Morbious
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Yarn setup problems
Posted by Morbious <kn...@gmail.com>.
Hi,
In container logs I have :
cat
/srv/data/*/yarn/logs/application_1426770785980_0010/container_e61_1426770785980_0010_01_0000*/stderr
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
Error: Could not find or load main class 8345
In command line I get:
Job Finished in 22.395 seconds
java.io.FileNotFoundException: File does not exist:
hdfs://spark-cluster-nn/user/morbious/QuasiMonteCarlo_1426835737919_2089306463/out/reduce-out
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Best regards,
Morbious
2015-03-20 0:32 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
> Please check the app logs.
>
> On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
> wrote:
>
>> Hello,
>>
>> Recently I switched from MR1 to YARN on my cluster.
>> When I tried some tests like :
>>
>> > hadoop jar
>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
>> 16 10
>>
>> I got errors like:
>>
>> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
>> attempt_1426770785980_0012_m_000006_2, Status : FAILED
>> Exception from container-launch.
>> Container id: container_e61_1426770785980_0012_01_000057
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>> --------------------------------------------------------------------------
>>
>> In nodemanager log there is something like this:
>>
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
>> code from container container_e61_1426770785980_0012_01_000057 is : 1
>> 2015-03-19 21:40:00,957 WARN
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
>> Exception from container-launch with container ID:
>> container_e61_1426770785980_0012_01_000057 and exit code: 1
>> ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
>> container-launch.
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
>> container_e61_1426770785980_0012_01_000057
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
>> ExitCodeException exitCode=1:
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell.run(Shell.java:455)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 2015-03-19 21:40:00,958 INFO
>> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
>> java.lang.Thread.run(Thread.java:745)
>>
>>
>> I was trying to find out what went wrong, but couldn't find anything. I
>> have similiar cluster with the same configuration which runs YARN too
>> without any problems...
>>
>> Best regards,
>>
>> Morbious
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Yarn setup problems
Posted by Jeff Zhang <zj...@gmail.com>.
Please check the app logs.
On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
wrote:
> Hello,
>
> Recently I switched from MR1 to YARN on my cluster.
> When I tried some tests like :
>
> > hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
> 16 10
>
> I got errors like:
>
> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
> attempt_1426770785980_0012_m_000006_2, Status : FAILED
> Exception from container-launch.
> Container id: container_e61_1426770785980_0012_01_000057
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> --------------------------------------------------------------------------
>
> In nodemanager log there is something like this:
>
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
> code from container container_e61_1426770785980_0012_01_000057 is : 1
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exception from container-launch with container ID:
> container_e61_1426770785980_0012_01_000057 and exit code: 1
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
> container-launch.
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
> container_e61_1426770785980_0012_01_000057
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
> ExitCodeException exitCode=1:
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.run(Shell.java:455)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.lang.Thread.run(Thread.java:745)
>
>
> I was trying to find out what went wrong, but couldn't find anything. I
> have similiar cluster with the same configuration which runs YARN too
> without any problems...
>
> Best regards,
>
> Morbious
>
--
Best Regards
Jeff Zhang
Re: Yarn setup problems
Posted by Jeff Zhang <zj...@gmail.com>.
Please check the app logs.
On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
wrote:
> Hello,
>
> Recently I switched from MR1 to YARN on my cluster.
> When I tried some tests like :
>
> > hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
> 16 10
>
> I got errors like:
>
> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
> attempt_1426770785980_0012_m_000006_2, Status : FAILED
> Exception from container-launch.
> Container id: container_e61_1426770785980_0012_01_000057
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> --------------------------------------------------------------------------
>
> In nodemanager log there is something like this:
>
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
> code from container container_e61_1426770785980_0012_01_000057 is : 1
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exception from container-launch with container ID:
> container_e61_1426770785980_0012_01_000057 and exit code: 1
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
> container-launch.
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
> container_e61_1426770785980_0012_01_000057
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
> ExitCodeException exitCode=1:
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.run(Shell.java:455)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.lang.Thread.run(Thread.java:745)
>
>
> I was trying to find out what went wrong, but couldn't find anything. I
> have similiar cluster with the same configuration which runs YARN too
> without any problems...
>
> Best regards,
>
> Morbious
>
--
Best Regards
Jeff Zhang
Re: Yarn setup problems
Posted by Jeff Zhang <zj...@gmail.com>.
Please check the app logs.
On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
wrote:
> Hello,
>
> Recently I switched from MR1 to YARN on my cluster.
> When I tried some tests like :
>
> > hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
> 16 10
>
> I got errors like:
>
> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
> attempt_1426770785980_0012_m_000006_2, Status : FAILED
> Exception from container-launch.
> Container id: container_e61_1426770785980_0012_01_000057
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> --------------------------------------------------------------------------
>
> In nodemanager log there is something like this:
>
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
> code from container container_e61_1426770785980_0012_01_000057 is : 1
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exception from container-launch with container ID:
> container_e61_1426770785980_0012_01_000057 and exit code: 1
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
> container-launch.
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
> container_e61_1426770785980_0012_01_000057
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
> ExitCodeException exitCode=1:
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.run(Shell.java:455)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.lang.Thread.run(Thread.java:745)
>
>
> I was trying to find out what went wrong, but couldn't find anything. I
> have similiar cluster with the same configuration which runs YARN too
> without any problems...
>
> Best regards,
>
> Morbious
>
--
Best Regards
Jeff Zhang
Re: Yarn setup problems
Posted by Jeff Zhang <zj...@gmail.com>.
Please check the app logs.
On Fri, Mar 20, 2015 at 5:46 AM, Morbious <kn...@gmail.com>
wrote:
> Hello,
>
> Recently I switched from MR1 to YARN on my cluster.
> When I tried some tests like :
>
> > hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar pi
> 16 10
>
> I got errors like:
>
> 15/03/19 21:40:03 INFO mapreduce.Job: Task Id :
> attempt_1426770785980_0012_m_000006_2, Status : FAILED
> Exception from container-launch.
> Container id: container_e61_1426770785980_0012_01_000057
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> --------------------------------------------------------------------------
>
> In nodemanager log there is something like this:
>
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
> code from container container_e61_1426770785980_0012_01_000057 is : 1
> 2015-03-19 21:40:00,957 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exception from container-launch with container ID:
> container_e61_1426770785980_0012_01_000057 and exit code: 1
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
> container-launch.
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
> container_e61_1426770785980_0012_01_000057
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
> ExitCodeException exitCode=1:
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell.run(Shell.java:455)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2015-03-19 21:40:00,958 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at
> java.lang.Thread.run(Thread.java:745)
>
>
> I was trying to find out what went wrong, but couldn't find anything. I
> have similiar cluster with the same configuration which runs YARN too
> without any problems...
>
> Best regards,
>
> Morbious
>
--
Best Regards
Jeff Zhang