You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by xuwenyang2005 <xu...@126.com> on 2010/07/02 08:52:29 UTC
hadoop testing on hs
Hello,
I am doing hadoop testing on the harmony select(r946981) and when I try to run some application, I encountered some problems.Maybe some defect exists inside the harmony select caused these problems and I will be very thankful if anybody could give me some advise to solve it.
I have configured 4 machine hadoop cluster(1 namenode, 1 jobtracker, 2 slaves), on which hadoop-0.20.2 could be started up successfully.
All of the 4 machines are linux(ubuntu) platform, JAVA_HOME is set to be the path of harmony select, other configuration values are just set to be default.
After the hadoop cluster is started up, I ran the terasort, a sorting application, with 500M input data.
During the testing, the job failed.
console message：
10/07/02 13:23:44 INFO terasort.TeraSort: starting
10/07/02 13:23:45 INFO mapred.FileInputFormat: Total input paths to process : 2
10/07/02 13:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
10/07/02 13:23:47 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 100000 records
Step size is 100000.0
10/07/02 13:23:50 INFO mapred.JobClient: Running job: job_201007021322_0001
10/07/02 13:23:51 INFO mapred.JobClient: map 0% reduce 0%
10/07/02 13:24:43 INFO mapred.JobClient: map 7% reduce 0%
10/07/02 13:24:46 INFO mapred.JobClient: map 12% reduce 0%
10/07/02 13:25:06 INFO mapred.JobClient: map 13% reduce 0%
10/07/02 13:25:09 INFO mapred.JobClient: map 15% reduce 0%
10/07/02 13:25:12 INFO mapred.JobClient: map 16% reduce 0%
10/07/02 13:25:15 INFO mapred.JobClient: map 18% reduce 0%
10/07/02 13:25:18 INFO mapred.JobClient: map 19% reduce 0%
10/07/02 13:25:21 INFO mapred.JobClient: map 20% reduce 0%
10/07/02 13:25:24 INFO mapred.JobClient: map 22% reduce 0%
10/07/02 13:25:27 INFO mapred.JobClient: map 23% reduce 0%
10/07/02 13:25:30 INFO mapred.JobClient: map 24% reduce 0%
10/07/02 13:34:55 INFO mapred.JobClient: map 12% reduce 0%
10/07/02 13:35:03 INFO mapred.JobClient: Task Id : attempt_201007021322_0001_m_000002_0, Status : FAILED
Task attempt_201007021322_0001_m_000002_0 failed to report status for 601 seconds. Killing!
10/07/02 13:35:03 INFO mapred.JobClient: Task Id : attempt_201007021322_0001_m_000003_0, Status : FAILED
Task attempt_201007021322_0001_m_000003_0 failed to report status for 602 seconds. Killing!
10/07/02 13:35:16 INFO mapred.JobClient: map 17% reduce 0%
10/07/02 13:35:19 INFO mapred.JobClient: map 23% reduce 0%
10/07/02 13:35:22 INFO mapred.JobClient: map 24% reduce 0%
10/07/02 13:35:46 INFO mapred.JobClient: map 12% reduce 0%
10/07/02 13:36:06 INFO mapred.JobClient: Task Id : attempt_201007021322_0001_m_000000_0, Status : FAILED
Task attempt_201007021322_0001_m_000000_0 failed to report status for 602 seconds. Killing!
10/07/02 13:36:07 INFO mapred.JobClient: Task Id : attempt_201007021322_0001_m_000001_0, Status : FAILED
Task attempt_201007021322_0001_m_000001_0 failed to report status for 600 seconds. Killing!
10/07/02 13:36:30 INFO mapred.JobClient: map 13% reduce 0%
10/07/02 13:36:33 INFO mapred.JobClient: map 16% reduce 0%
10/07/02 13:36:36 INFO mapred.JobClient: map 18% reduce 0%
....
log :
2010-07-02 13:35:09,538 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2010-07-02 13:35:10,433 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2010-07-02 13:35:10,459 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2010-07-02 13:35:11,073 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2010-07-02 13:35:11,074 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2010-07-02 13:35:11,308 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2010-07-02 13:35:11,311 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2010-07-02 13:35:16,153 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 26214400; bufvoid = 99614720
2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
Look forward to any reply.
Best Regards,
Cindy
Re:Re: hadoop testing on hs
Posted by xuwenyang2005 <xu...@126.com>.
Thanks Jim,
In fact, no error message for the JRE could be observed during the test.
That's just what troubles me.
For the hdk, I will try it :)
在2010-07-05 13:23:39，"Jim Yu" <ju...@gmail.com> 写道：
>Hi Cindy,
>
>In the console and log message you provided, there is only trace info of
>Hadoop. Did you see any error message for the JRE? As Harmony Select is a
>subset of HDK, any luck to try with the HDK? :-)
>
>在 2010年7月2日 下午2:52，xuwenyang2005 <xu...@126.com>写道：
>
>> Hello,
>> I am doing hadoop testing on the harmony select(r946981) and when I try to
>> run some application, I encountered some problems.Maybe some defect exists
>> inside the harmony select caused these problems and I will be very thankful
>> if anybody could give me some advise to solve it.
>> I have configured 4 machine hadoop cluster(1 namenode, 1 jobtracker, 2
>> slaves), on which hadoop-0.20.2 could be started up successfully.
>> All of the 4 machines are linux(ubuntu) platform, JAVA_HOME is set to be
>> the path of harmony select, other configuration values are just set to be
>> default.
>> After the hadoop cluster is started up, I ran the terasort, a sorting
>> application, with 500M input data.
>> During the testing, the job failed.
>> console message：
>> 10/07/02 13:23:44 INFO terasort.TeraSort: starting
>> 10/07/02 13:23:45 INFO mapred.FileInputFormat: Total input paths to process
>> : 2
>> 10/07/02 13:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 10/07/02 13:23:47 INFO compress.CodecPool: Got brand-new compressor
>> Making 1 from 100000 records
>> Step size is 100000.0
>> 10/07/02 13:23:50 INFO mapred.JobClient: Running job: job_201007021322_0001
>> 10/07/02 13:23:51 INFO mapred.JobClient: map 0% reduce 0%
>> 10/07/02 13:24:43 INFO mapred.JobClient: map 7% reduce 0%
>> 10/07/02 13:24:46 INFO mapred.JobClient: map 12% reduce 0%
>> 10/07/02 13:25:06 INFO mapred.JobClient: map 13% reduce 0%
>> 10/07/02 13:25:09 INFO mapred.JobClient: map 15% reduce 0%
>> 10/07/02 13:25:12 INFO mapred.JobClient: map 16% reduce 0%
>> 10/07/02 13:25:15 INFO mapred.JobClient: map 18% reduce 0%
>> 10/07/02 13:25:18 INFO mapred.JobClient: map 19% reduce 0%
>> 10/07/02 13:25:21 INFO mapred.JobClient: map 20% reduce 0%
>> 10/07/02 13:25:24 INFO mapred.JobClient: map 22% reduce 0%
>> 10/07/02 13:25:27 INFO mapred.JobClient: map 23% reduce 0%
>> 10/07/02 13:25:30 INFO mapred.JobClient: map 24% reduce 0%
>> 10/07/02 13:34:55 INFO mapred.JobClient: map 12% reduce 0%
>> 10/07/02 13:35:03 INFO mapred.JobClient: Task Id :
>> attempt_201007021322_0001_m_000002_0, Status : FAILED
>> Task attempt_201007021322_0001_m_000002_0 failed to report status for 601
>> seconds. Killing!
>> 10/07/02 13:35:03 INFO mapred.JobClient: Task Id :
>> attempt_201007021322_0001_m_000003_0, Status : FAILED
>> Task attempt_201007021322_0001_m_000003_0 failed to report status for 602
>> seconds. Killing!
>> 10/07/02 13:35:16 INFO mapred.JobClient: map 17% reduce 0%
>> 10/07/02 13:35:19 INFO mapred.JobClient: map 23% reduce 0%
>> 10/07/02 13:35:22 INFO mapred.JobClient: map 24% reduce 0%
>> 10/07/02 13:35:46 INFO mapred.JobClient: map 12% reduce 0%
>> 10/07/02 13:36:06 INFO mapred.JobClient: Task Id :
>> attempt_201007021322_0001_m_000000_0, Status : FAILED
>> Task attempt_201007021322_0001_m_000000_0 failed to report status for 602
>> seconds. Killing!
>> 10/07/02 13:36:07 INFO mapred.JobClient: Task Id :
>> attempt_201007021322_0001_m_000001_0, Status : FAILED
>> Task attempt_201007021322_0001_m_000001_0 failed to report status for 600
>> seconds. Killing!
>> 10/07/02 13:36:30 INFO mapred.JobClient: map 13% reduce 0%
>> 10/07/02 13:36:33 INFO mapred.JobClient: map 16% reduce 0%
>> 10/07/02 13:36:36 INFO mapred.JobClient: map 18% reduce 0%
>> ....
>>
>> log :
>>
>> 2010-07-02 13:35:09,538 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=MAP, sessionId=
>> 2010-07-02 13:35:10,433 INFO org.apache.hadoop.mapred.MapTask:
>> numReduceTasks: 1
>> 2010-07-02 13:35:10,459 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
>> 100
>> 2010-07-02 13:35:11,073 INFO org.apache.hadoop.mapred.MapTask: data buffer
>> = 79691776/99614720
>> 2010-07-02 13:35:11,074 INFO org.apache.hadoop.mapred.MapTask: record
>> buffer = 262144/327680
>> 2010-07-02 13:35:11,308 WARN org.apache.hadoop.util.NativeCodeLoader:
>> Unable to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2010-07-02 13:35:11,311 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> 2010-07-02 13:35:16,153 INFO org.apache.hadoop.mapred.MapTask: Spilling map
>> output: record full = true
>> 2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>> 0; bufend = 26214400; bufvoid = 99614720
>> 2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0;
>> kvend = 262144; length = 327680
>>
>>
>> Look forward to any reply.
>>
>> Best Regards,
>> Cindy
>>
>>
>
>
>--
>Best Regards,
>Jim, Jun Jie Yu
Re: hadoop testing on hs
Posted by Jim Yu <ju...@gmail.com>.
Hi Cindy,
In the console and log message you provided, there is only trace info of
Hadoop. Did you see any error message for the JRE? As Harmony Select is a
subset of HDK, any luck to try with the HDK? :-)
在 2010年7月2日 下午2:52，xuwenyang2005 <xu...@126.com>写道：
> Hello,
> I am doing hadoop testing on the harmony select(r946981) and when I try to
> run some application, I encountered some problems.Maybe some defect exists
> inside the harmony select caused these problems and I will be very thankful
> if anybody could give me some advise to solve it.
> I have configured 4 machine hadoop cluster(1 namenode, 1 jobtracker, 2
> slaves), on which hadoop-0.20.2 could be started up successfully.
> All of the 4 machines are linux(ubuntu) platform, JAVA_HOME is set to be
> the path of harmony select, other configuration values are just set to be
> default.
> After the hadoop cluster is started up, I ran the terasort, a sorting
> application, with 500M input data.
> During the testing, the job failed.
> console message：
> 10/07/02 13:23:44 INFO terasort.TeraSort: starting
> 10/07/02 13:23:45 INFO mapred.FileInputFormat: Total input paths to process
> : 2
> 10/07/02 13:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 10/07/02 13:23:47 INFO compress.CodecPool: Got brand-new compressor
> Making 1 from 100000 records
> Step size is 100000.0
> 10/07/02 13:23:50 INFO mapred.JobClient: Running job: job_201007021322_0001
> 10/07/02 13:23:51 INFO mapred.JobClient: map 0% reduce 0%
> 10/07/02 13:24:43 INFO mapred.JobClient: map 7% reduce 0%
> 10/07/02 13:24:46 INFO mapred.JobClient: map 12% reduce 0%
> 10/07/02 13:25:06 INFO mapred.JobClient: map 13% reduce 0%
> 10/07/02 13:25:09 INFO mapred.JobClient: map 15% reduce 0%
> 10/07/02 13:25:12 INFO mapred.JobClient: map 16% reduce 0%
> 10/07/02 13:25:15 INFO mapred.JobClient: map 18% reduce 0%
> 10/07/02 13:25:18 INFO mapred.JobClient: map 19% reduce 0%
> 10/07/02 13:25:21 INFO mapred.JobClient: map 20% reduce 0%
> 10/07/02 13:25:24 INFO mapred.JobClient: map 22% reduce 0%
> 10/07/02 13:25:27 INFO mapred.JobClient: map 23% reduce 0%
> 10/07/02 13:25:30 INFO mapred.JobClient: map 24% reduce 0%
> 10/07/02 13:34:55 INFO mapred.JobClient: map 12% reduce 0%
> 10/07/02 13:35:03 INFO mapred.JobClient: Task Id :
> attempt_201007021322_0001_m_000002_0, Status : FAILED
> Task attempt_201007021322_0001_m_000002_0 failed to report status for 601
> seconds. Killing!
> 10/07/02 13:35:03 INFO mapred.JobClient: Task Id :
> attempt_201007021322_0001_m_000003_0, Status : FAILED
> Task attempt_201007021322_0001_m_000003_0 failed to report status for 602
> seconds. Killing!
> 10/07/02 13:35:16 INFO mapred.JobClient: map 17% reduce 0%
> 10/07/02 13:35:19 INFO mapred.JobClient: map 23% reduce 0%
> 10/07/02 13:35:22 INFO mapred.JobClient: map 24% reduce 0%
> 10/07/02 13:35:46 INFO mapred.JobClient: map 12% reduce 0%
> 10/07/02 13:36:06 INFO mapred.JobClient: Task Id :
> attempt_201007021322_0001_m_000000_0, Status : FAILED
> Task attempt_201007021322_0001_m_000000_0 failed to report status for 602
> seconds. Killing!
> 10/07/02 13:36:07 INFO mapred.JobClient: Task Id :
> attempt_201007021322_0001_m_000001_0, Status : FAILED
> Task attempt_201007021322_0001_m_000001_0 failed to report status for 600
> seconds. Killing!
> 10/07/02 13:36:30 INFO mapred.JobClient: map 13% reduce 0%
> 10/07/02 13:36:33 INFO mapred.JobClient: map 16% reduce 0%
> 10/07/02 13:36:36 INFO mapred.JobClient: map 18% reduce 0%
> ....
>
> log :
>
> 2010-07-02 13:35:09,538 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2010-07-02 13:35:10,433 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 1
> 2010-07-02 13:35:10,459 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
> 100
> 2010-07-02 13:35:11,073 INFO org.apache.hadoop.mapred.MapTask: data buffer
> = 79691776/99614720
> 2010-07-02 13:35:11,074 INFO org.apache.hadoop.mapred.MapTask: record
> buffer = 262144/327680
> 2010-07-02 13:35:11,308 WARN org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2010-07-02 13:35:11,311 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> 2010-07-02 13:35:16,153 INFO org.apache.hadoop.mapred.MapTask: Spilling map
> output: record full = true
> 2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: bufstart =
> 0; bufend = 26214400; bufvoid = 99614720
> 2010-07-02 13:35:16,154 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0;
> kvend = 262144; length = 327680
>
>
> Look forward to any reply.
>
> Best Regards,
> Cindy
>
>
--
Best Regards,
Jim, Jun Jie Yu