You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Basu,Indrashish" <in...@ufl.edu> on 2013/11/01 01:00:10 UTC
Error while running Hadoop Source Code
Hi,
I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
on an ARM processor and getting the below error. Can anyone please throw
some light on this ?
rmr: cannot remove output: No such file or directory.
13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
process : 1
13/10/31 13:43:13 INFO mapred.JobClient: Running job:
job_201310311320_0001
13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
... 3 more
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Volunteer
Posted by Mirko Kämpf <mi...@gmail.com>.
What is the field you want to working in, core hadoop development,
scripting and testing, documentation, tool development, app development,
benchmarking?
What is your level of experience?
What programming languages do you use?
I think you can just start with building hadoop and its related projects
you are interested in
from scratch and than if the cluster works well, explore the Jiras.
Good luck!
Mirko
2013/11/7 Mike <mi...@unitedrmr.com>
> Hi guys
> I would like to volunteer and help with hadoop. Could you point me in the
> right direction?
> Best regards
> Mike
Re: Volunteer
Posted by Mirko Kämpf <mi...@gmail.com>.
What is the field you want to working in, core hadoop development,
scripting and testing, documentation, tool development, app development,
benchmarking?
What is your level of experience?
What programming languages do you use?
I think you can just start with building hadoop and its related projects
you are interested in
from scratch and than if the cluster works well, explore the Jiras.
Good luck!
Mirko
2013/11/7 Mike <mi...@unitedrmr.com>
> Hi guys
> I would like to volunteer and help with hadoop. Could you point me in the
> right direction?
> Best regards
> Mike
Re: Volunteer
Posted by Mirko Kämpf <mi...@gmail.com>.
What is the field you want to working in, core hadoop development,
scripting and testing, documentation, tool development, app development,
benchmarking?
What is your level of experience?
What programming languages do you use?
I think you can just start with building hadoop and its related projects
you are interested in
from scratch and than if the cluster works well, explore the Jiras.
Good luck!
Mirko
2013/11/7 Mike <mi...@unitedrmr.com>
> Hi guys
> I would like to volunteer and help with hadoop. Could you point me in the
> right direction?
> Best regards
> Mike
Re: Volunteer
Posted by Mirko Kämpf <mi...@gmail.com>.
What is the field you want to working in, core hadoop development,
scripting and testing, documentation, tool development, app development,
benchmarking?
What is your level of experience?
What programming languages do you use?
I think you can just start with building hadoop and its related projects
you are interested in
from scratch and than if the cluster works well, explore the Jiras.
Good luck!
Mirko
2013/11/7 Mike <mi...@unitedrmr.com>
> Hi guys
> I would like to volunteer and help with hadoop. Could you point me in the
> right direction?
> Best regards
> Mike
Volunteer
Posted by Mike <mi...@unitedrmr.com>.
Hi guys
I would like to volunteer and help with hadoop. Could you point me in the right direction?
Best regards
Mike
Volunteer
Posted by Mike <mi...@unitedrmr.com>.
Hi guys
I would like to volunteer and help with hadoop. Could you point me in the right direction?
Best regards
Mike
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone please assist regarding this ?
Regards,
Indrashish
On Sat, 09 Nov 2013 10:58:03 -0500, Basu,Indrashish
wrote:
> Any updates on this ?
>
> Regards,
>
> Indra
>
> On
Thu, 07 Nov 2013 11:28:57 -0500, Basu,Indrashish wrote:
>
>> Hi Vinod,
>>
>> Please find the below details requested:
>>
>> A) ALL OF THE
TASKTRACKER LOG :
>>
>> Same what I have provided in the earlier
email.
>>
>> B) THE TASK-LOGS :
>>
>> SYSLOG :
>>
>> 2013-11-06
16:37:21,331 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing
JVM Metrics with processName=MAP, sessionId=
>> 2013-11-06 16:37:22,245
INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
>> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
>> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
>>
at java.io.DataInputStream.readByte(DataInputStream.java:267)
>> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>>
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>>
>> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
>> java.io.IOException: pipe child exception
>> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by:
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
... 3 more
>>
>> STDERR :
>>
>> Command:0received before
authentication. Exiting..
>>
>> STDOUT :
>>
>> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
>> deviceID:
>>
>> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>>
>>
attempt_201311061630_0001_m_000000_0
>>
>>
attempt_201311061630_0001_m_000001_0
>>
>>
attempt_201311061630_0001_m_000002_0
>>
>>
attempt_201311061630_0001_m_000000_1
>>
>>
attempt_201311061630_0001_m_000001_1
>>
>>
attempt_201311061630_0001_m_000003_0
>>
>>
attempt_201311061630_0001_m_000000_2
>>
>>
attempt_201311061630_0001_m_000001_2
>>
>>
attempt_201311061630_0001_m_000000_3
>>
>>
attempt_201311061630_0001_m_000001_3
>>
>> Kindly let me know in case
of any more details required.
>>
>> Thanks again for your time and
patience regarding this.
>>
>> Regards,
>>
>> Indrashish.
>>
>> On
Wed, 6 Nov 2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>>
>>>
Don't see anything in the logs that you pasted.
>>> Can you paste the
following in say pastebin?
>>> - All of the TaskTracker log
>>> - The
task-logs. These are syslog, stderr, stdout files for a specific
TaskAttempt.
>>> - And specific TaskAttempt's TaskAttemtpID that is
failing.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Nov 6, 2013, at 2:50
PM, Basu,Indrashish wrote:
>>>
>>>> Can anyone please assist regarding
this ?
>>>>
>>>> Thanks in advance
>>>>
>>>> Regards,
>>>>
>>>>
Indra
>>>>
>>>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for your help
regarding this. I checked the task logs, this is what it is giving as
output.
>>>>>
>>>>> 2013-11-06 06:40:05,541 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_1100862588
>>>>>
>>>>> 2013-11-06 06:40:05,553
INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
>>>>>
>>>>> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>>
>>>>>
2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>>
>>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>>
>>>>> 636_0001_m_000001_3
>>>>>
>>>>>
2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_-1960039766 given task:
attempt_20131106
>>>>>
>>>>> 0636_0001_m_000000_3
>>>>>
>>>>>
2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>>>>>
>>>>> 2013-11-06
06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>>
>>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>>
>>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>>
>>>>> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>>>>>
>>>>>
state:UNASSIGNED
>>>>>
>>>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>>
>>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>>
>>>>> 11060636_0001_m_000002_0
>>>>>
>>>>> 2013-11-06 06:40:14,909
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000000_3
>>>>>
>>>>> 2013-11-06
06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>>>>>
>>>>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>>
>>>>> 2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
>>>>>
>>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>>
>>>>> 36_0001_m_000002_0
>>>>>
>>>>> Regards,
>>>>>
>>>>>
Indrashish
>>>>>
>>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar
Vavilapalli wrote:
>>>>>
>>>>>> It seems like your pipes mapper is
exiting before consuming all the input. Did you check the task-logs on
the web UI?
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>>>>>
>>>>>>>
Hi,
>>>>>>>
>>>>>>> Can anyone kindly assist on this ?
>>>>>>>
>>>>>>>
Regards,
>>>>>>> Indrashish
>>>>>>>
>>>>>>> On Mon, 04 Nov 2013
10:23:23 -0500, Basu,Indrashish wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>
>>>>>>>> Any update on the below post ?
>>>>>>> solution as " The
*nopipe* example needs more documentation. It
src/test/org/apache/*hadoop*/mapred/*pipes*/ idth:
100%;">*WordCountInputFormat*.java, which
>>>>>>> ecific input split
format.
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid;
>>>>>>>
5px; width: 100%;">as the input split and won't work right. The *nopipe*
example should
>>>>>>>
>>>>>>>> probably be recoded *to* use libhdfs
*too*, but that is more
>>>>>>> ckquote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">complicated
>>>>>>>
>>>>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100
>>>>>>> le is using local file reads, it
will only work on a cluster could elaborate a bit about the
>>>>>>>
needs to be done exactly. To m
>>>>>>>
>>>>>>>> px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
>>>>>>> Hadoop.
Regard
>>>>>>> e> Indr
>>>>>>>
>>>>>>>> t: 5px; width: 100%;">On Thu,
31 Oct 2013 20:00:10 -0400,
>>>>>>> sh wrote: Hi,
>>>>>>> dding-left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> to
run a sample Hadoop GPU source code (kmeans algorithm)
>>>>>>> te>
2px
>>>>>>> argin-left: 5px; width: 100%;">13/10/31 13:43:12 WARN
mapred.JobClient: No job jar file set. User eft: 5px; w
>>>>>>>
cl
>>>>>>>
>>>>>>>> x; border-left: #1010ff 2px solid; margin-left:
5p
>>>>>>> %;">
>>>>>>> gin-left: 5px; width: 100%;">13/10/31 13:43:12
INFO mapred.FileInputFormat: Total input paths to : 100%;">
>>>>>>>>
>>>>>>>>> process : 1
>>>>>>> ft: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/10
>>>>>>>
>>>>>>>> e style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
>>>>>>>
00%;">13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>>> ; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Appli
>>>>>>>
>>>>>>>>
px solid; margin-left: 5px; width: 100%;">
>>>>>>>>
>>>>>>>>>
at
>>>>>>> ckquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>> at
org.a
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> Caused
>>>>>>> .SocketException: Broken pipe at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>>> ockquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> &
>>>>>>>
>>>>>>>> quote>
>>>>>>> width: 100%;"> at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>
at java.io.BufferedOutputStream.write(Buffere
>>>>>>>
>>>>>>>> f 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
.io.DataOutputStream.write(DataOutputStream.java:107) at
>>>>>>> solid;
margin-left: 5px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333
>>>>>>>
>>>>>>>> argin-left: 5px; width: 100%;"> at
>>>>>>> lockquote> at c,
exec
>>>>>>> x solid; margin-left: 5px; width:
100%;">'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>>
>>>>>>> ote style="padding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>>>
margin-left: 5px; width: 100%;"> Regards,
>>>>>>> --
>>>>>>>
Indrashish Basu
>>>>>>> Graduate Student
>>>>>>> Departmen
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the
use of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>>
>>>>> --
>>>>>
>>>>> Indrashish
Basu
>>>>> Graduate Student
>>>>> Department of Electrical and
Computer Engineering
>>>>> University of Florida
>>>>
>>>> --
>>>>
>>>> Indrashish Basu
>>>> Graduate Student
>>>> Department of
Electrical and Computer Engineering
>>>> University of Florida
>>>
>>>
CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>
>> --
>>
>> Indrashish Basu
>>
Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>
> --
>
> Indrashish Basu
> Graduate
Student
> Department of Electrical and Computer Engineering
>
University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of
Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone please assist regarding this ?
Regards,
Indrashish
On Sat, 09 Nov 2013 10:58:03 -0500, Basu,Indrashish
wrote:
> Any updates on this ?
>
> Regards,
>
> Indra
>
> On
Thu, 07 Nov 2013 11:28:57 -0500, Basu,Indrashish wrote:
>
>> Hi Vinod,
>>
>> Please find the below details requested:
>>
>> A) ALL OF THE
TASKTRACKER LOG :
>>
>> Same what I have provided in the earlier
email.
>>
>> B) THE TASK-LOGS :
>>
>> SYSLOG :
>>
>> 2013-11-06
16:37:21,331 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing
JVM Metrics with processName=MAP, sessionId=
>> 2013-11-06 16:37:22,245
INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
>> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
>> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
>>
at java.io.DataInputStream.readByte(DataInputStream.java:267)
>> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>>
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>>
>> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
>> java.io.IOException: pipe child exception
>> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by:
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
... 3 more
>>
>> STDERR :
>>
>> Command:0received before
authentication. Exiting..
>>
>> STDOUT :
>>
>> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
>> deviceID:
>>
>> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>>
>>
attempt_201311061630_0001_m_000000_0
>>
>>
attempt_201311061630_0001_m_000001_0
>>
>>
attempt_201311061630_0001_m_000002_0
>>
>>
attempt_201311061630_0001_m_000000_1
>>
>>
attempt_201311061630_0001_m_000001_1
>>
>>
attempt_201311061630_0001_m_000003_0
>>
>>
attempt_201311061630_0001_m_000000_2
>>
>>
attempt_201311061630_0001_m_000001_2
>>
>>
attempt_201311061630_0001_m_000000_3
>>
>>
attempt_201311061630_0001_m_000001_3
>>
>> Kindly let me know in case
of any more details required.
>>
>> Thanks again for your time and
patience regarding this.
>>
>> Regards,
>>
>> Indrashish.
>>
>> On
Wed, 6 Nov 2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>>
>>>
Don't see anything in the logs that you pasted.
>>> Can you paste the
following in say pastebin?
>>> - All of the TaskTracker log
>>> - The
task-logs. These are syslog, stderr, stdout files for a specific
TaskAttempt.
>>> - And specific TaskAttempt's TaskAttemtpID that is
failing.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Nov 6, 2013, at 2:50
PM, Basu,Indrashish wrote:
>>>
>>>> Can anyone please assist regarding
this ?
>>>>
>>>> Thanks in advance
>>>>
>>>> Regards,
>>>>
>>>>
Indra
>>>>
>>>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for your help
regarding this. I checked the task logs, this is what it is giving as
output.
>>>>>
>>>>> 2013-11-06 06:40:05,541 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_1100862588
>>>>>
>>>>> 2013-11-06 06:40:05,553
INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
>>>>>
>>>>> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>>
>>>>>
2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>>
>>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>>
>>>>> 636_0001_m_000001_3
>>>>>
>>>>>
2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_-1960039766 given task:
attempt_20131106
>>>>>
>>>>> 0636_0001_m_000000_3
>>>>>
>>>>>
2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>>>>>
>>>>> 2013-11-06
06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>>
>>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>>
>>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>>
>>>>> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>>>>>
>>>>>
state:UNASSIGNED
>>>>>
>>>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>>
>>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>>
>>>>> 11060636_0001_m_000002_0
>>>>>
>>>>> 2013-11-06 06:40:14,909
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000000_3
>>>>>
>>>>> 2013-11-06
06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>>>>>
>>>>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>>
>>>>> 2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
>>>>>
>>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>>
>>>>> 36_0001_m_000002_0
>>>>>
>>>>> Regards,
>>>>>
>>>>>
Indrashish
>>>>>
>>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar
Vavilapalli wrote:
>>>>>
>>>>>> It seems like your pipes mapper is
exiting before consuming all the input. Did you check the task-logs on
the web UI?
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>>>>>
>>>>>>>
Hi,
>>>>>>>
>>>>>>> Can anyone kindly assist on this ?
>>>>>>>
>>>>>>>
Regards,
>>>>>>> Indrashish
>>>>>>>
>>>>>>> On Mon, 04 Nov 2013
10:23:23 -0500, Basu,Indrashish wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>
>>>>>>>> Any update on the below post ?
>>>>>>> solution as " The
*nopipe* example needs more documentation. It
src/test/org/apache/*hadoop*/mapred/*pipes*/ idth:
100%;">*WordCountInputFormat*.java, which
>>>>>>> ecific input split
format.
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid;
>>>>>>>
5px; width: 100%;">as the input split and won't work right. The *nopipe*
example should
>>>>>>>
>>>>>>>> probably be recoded *to* use libhdfs
*too*, but that is more
>>>>>>> ckquote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">complicated
>>>>>>>
>>>>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100
>>>>>>> le is using local file reads, it
will only work on a cluster could elaborate a bit about the
>>>>>>>
needs to be done exactly. To m
>>>>>>>
>>>>>>>> px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
>>>>>>> Hadoop.
Regard
>>>>>>> e> Indr
>>>>>>>
>>>>>>>> t: 5px; width: 100%;">On Thu,
31 Oct 2013 20:00:10 -0400,
>>>>>>> sh wrote: Hi,
>>>>>>> dding-left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> to
run a sample Hadoop GPU source code (kmeans algorithm)
>>>>>>> te>
2px
>>>>>>> argin-left: 5px; width: 100%;">13/10/31 13:43:12 WARN
mapred.JobClient: No job jar file set. User eft: 5px; w
>>>>>>>
cl
>>>>>>>
>>>>>>>> x; border-left: #1010ff 2px solid; margin-left:
5p
>>>>>>> %;">
>>>>>>> gin-left: 5px; width: 100%;">13/10/31 13:43:12
INFO mapred.FileInputFormat: Total input paths to : 100%;">
>>>>>>>>
>>>>>>>>> process : 1
>>>>>>> ft: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/10
>>>>>>>
>>>>>>>> e style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
>>>>>>>
00%;">13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>>> ; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Appli
>>>>>>>
>>>>>>>>
px solid; margin-left: 5px; width: 100%;">
>>>>>>>>
>>>>>>>>>
at
>>>>>>> ckquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>> at
org.a
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> Caused
>>>>>>> .SocketException: Broken pipe at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>>> ockquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> &
>>>>>>>
>>>>>>>> quote>
>>>>>>> width: 100%;"> at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>
at java.io.BufferedOutputStream.write(Buffere
>>>>>>>
>>>>>>>> f 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
.io.DataOutputStream.write(DataOutputStream.java:107) at
>>>>>>> solid;
margin-left: 5px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333
>>>>>>>
>>>>>>>> argin-left: 5px; width: 100%;"> at
>>>>>>> lockquote> at c,
exec
>>>>>>> x solid; margin-left: 5px; width:
100%;">'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>>
>>>>>>> ote style="padding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>>>
margin-left: 5px; width: 100%;"> Regards,
>>>>>>> --
>>>>>>>
Indrashish Basu
>>>>>>> Graduate Student
>>>>>>> Departmen
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the
use of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>>
>>>>> --
>>>>>
>>>>> Indrashish
Basu
>>>>> Graduate Student
>>>>> Department of Electrical and
Computer Engineering
>>>>> University of Florida
>>>>
>>>> --
>>>>
>>>> Indrashish Basu
>>>> Graduate Student
>>>> Department of
Electrical and Computer Engineering
>>>> University of Florida
>>>
>>>
CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>
>> --
>>
>> Indrashish Basu
>>
Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>
> --
>
> Indrashish Basu
> Graduate
Student
> Department of Electrical and Computer Engineering
>
University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of
Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone please assist regarding this ?
Regards,
Indrashish
On Sat, 09 Nov 2013 10:58:03 -0500, Basu,Indrashish
wrote:
> Any updates on this ?
>
> Regards,
>
> Indra
>
> On
Thu, 07 Nov 2013 11:28:57 -0500, Basu,Indrashish wrote:
>
>> Hi Vinod,
>>
>> Please find the below details requested:
>>
>> A) ALL OF THE
TASKTRACKER LOG :
>>
>> Same what I have provided in the earlier
email.
>>
>> B) THE TASK-LOGS :
>>
>> SYSLOG :
>>
>> 2013-11-06
16:37:21,331 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing
JVM Metrics with processName=MAP, sessionId=
>> 2013-11-06 16:37:22,245
INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
>> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
>> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
>>
at java.io.DataInputStream.readByte(DataInputStream.java:267)
>> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>>
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>>
>> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
>> java.io.IOException: pipe child exception
>> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by:
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
... 3 more
>>
>> STDERR :
>>
>> Command:0received before
authentication. Exiting..
>>
>> STDOUT :
>>
>> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
>> deviceID:
>>
>> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>>
>>
attempt_201311061630_0001_m_000000_0
>>
>>
attempt_201311061630_0001_m_000001_0
>>
>>
attempt_201311061630_0001_m_000002_0
>>
>>
attempt_201311061630_0001_m_000000_1
>>
>>
attempt_201311061630_0001_m_000001_1
>>
>>
attempt_201311061630_0001_m_000003_0
>>
>>
attempt_201311061630_0001_m_000000_2
>>
>>
attempt_201311061630_0001_m_000001_2
>>
>>
attempt_201311061630_0001_m_000000_3
>>
>>
attempt_201311061630_0001_m_000001_3
>>
>> Kindly let me know in case
of any more details required.
>>
>> Thanks again for your time and
patience regarding this.
>>
>> Regards,
>>
>> Indrashish.
>>
>> On
Wed, 6 Nov 2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>>
>>>
Don't see anything in the logs that you pasted.
>>> Can you paste the
following in say pastebin?
>>> - All of the TaskTracker log
>>> - The
task-logs. These are syslog, stderr, stdout files for a specific
TaskAttempt.
>>> - And specific TaskAttempt's TaskAttemtpID that is
failing.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Nov 6, 2013, at 2:50
PM, Basu,Indrashish wrote:
>>>
>>>> Can anyone please assist regarding
this ?
>>>>
>>>> Thanks in advance
>>>>
>>>> Regards,
>>>>
>>>>
Indra
>>>>
>>>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for your help
regarding this. I checked the task logs, this is what it is giving as
output.
>>>>>
>>>>> 2013-11-06 06:40:05,541 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_1100862588
>>>>>
>>>>> 2013-11-06 06:40:05,553
INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
>>>>>
>>>>> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>>
>>>>>
2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>>
>>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>>
>>>>> 636_0001_m_000001_3
>>>>>
>>>>>
2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_-1960039766 given task:
attempt_20131106
>>>>>
>>>>> 0636_0001_m_000000_3
>>>>>
>>>>>
2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>>>>>
>>>>> 2013-11-06
06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>>
>>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>>
>>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>>
>>>>> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>>>>>
>>>>>
state:UNASSIGNED
>>>>>
>>>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>>
>>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>>
>>>>> 11060636_0001_m_000002_0
>>>>>
>>>>> 2013-11-06 06:40:14,909
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000000_3
>>>>>
>>>>> 2013-11-06
06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>>>>>
>>>>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>>
>>>>> 2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
>>>>>
>>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>>
>>>>> 36_0001_m_000002_0
>>>>>
>>>>> Regards,
>>>>>
>>>>>
Indrashish
>>>>>
>>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar
Vavilapalli wrote:
>>>>>
>>>>>> It seems like your pipes mapper is
exiting before consuming all the input. Did you check the task-logs on
the web UI?
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>>>>>
>>>>>>>
Hi,
>>>>>>>
>>>>>>> Can anyone kindly assist on this ?
>>>>>>>
>>>>>>>
Regards,
>>>>>>> Indrashish
>>>>>>>
>>>>>>> On Mon, 04 Nov 2013
10:23:23 -0500, Basu,Indrashish wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>
>>>>>>>> Any update on the below post ?
>>>>>>> solution as " The
*nopipe* example needs more documentation. It
src/test/org/apache/*hadoop*/mapred/*pipes*/ idth:
100%;">*WordCountInputFormat*.java, which
>>>>>>> ecific input split
format.
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid;
>>>>>>>
5px; width: 100%;">as the input split and won't work right. The *nopipe*
example should
>>>>>>>
>>>>>>>> probably be recoded *to* use libhdfs
*too*, but that is more
>>>>>>> ckquote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">complicated
>>>>>>>
>>>>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100
>>>>>>> le is using local file reads, it
will only work on a cluster could elaborate a bit about the
>>>>>>>
needs to be done exactly. To m
>>>>>>>
>>>>>>>> px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
>>>>>>> Hadoop.
Regard
>>>>>>> e> Indr
>>>>>>>
>>>>>>>> t: 5px; width: 100%;">On Thu,
31 Oct 2013 20:00:10 -0400,
>>>>>>> sh wrote: Hi,
>>>>>>> dding-left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> to
run a sample Hadoop GPU source code (kmeans algorithm)
>>>>>>> te>
2px
>>>>>>> argin-left: 5px; width: 100%;">13/10/31 13:43:12 WARN
mapred.JobClient: No job jar file set. User eft: 5px; w
>>>>>>>
cl
>>>>>>>
>>>>>>>> x; border-left: #1010ff 2px solid; margin-left:
5p
>>>>>>> %;">
>>>>>>> gin-left: 5px; width: 100%;">13/10/31 13:43:12
INFO mapred.FileInputFormat: Total input paths to : 100%;">
>>>>>>>>
>>>>>>>>> process : 1
>>>>>>> ft: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/10
>>>>>>>
>>>>>>>> e style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
>>>>>>>
00%;">13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>>> ; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Appli
>>>>>>>
>>>>>>>>
px solid; margin-left: 5px; width: 100%;">
>>>>>>>>
>>>>>>>>>
at
>>>>>>> ckquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>> at
org.a
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> Caused
>>>>>>> .SocketException: Broken pipe at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>>> ockquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> &
>>>>>>>
>>>>>>>> quote>
>>>>>>> width: 100%;"> at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>
at java.io.BufferedOutputStream.write(Buffere
>>>>>>>
>>>>>>>> f 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
.io.DataOutputStream.write(DataOutputStream.java:107) at
>>>>>>> solid;
margin-left: 5px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333
>>>>>>>
>>>>>>>> argin-left: 5px; width: 100%;"> at
>>>>>>> lockquote> at c,
exec
>>>>>>> x solid; margin-left: 5px; width:
100%;">'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>>
>>>>>>> ote style="padding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>>>
margin-left: 5px; width: 100%;"> Regards,
>>>>>>> --
>>>>>>>
Indrashish Basu
>>>>>>> Graduate Student
>>>>>>> Departmen
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the
use of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>>
>>>>> --
>>>>>
>>>>> Indrashish
Basu
>>>>> Graduate Student
>>>>> Department of Electrical and
Computer Engineering
>>>>> University of Florida
>>>>
>>>> --
>>>>
>>>> Indrashish Basu
>>>> Graduate Student
>>>> Department of
Electrical and Computer Engineering
>>>> University of Florida
>>>
>>>
CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>
>> --
>>
>> Indrashish Basu
>>
Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>
> --
>
> Indrashish Basu
> Graduate
Student
> Department of Electrical and Computer Engineering
>
University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of
Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone please assist regarding this ?
Regards,
Indrashish
On Sat, 09 Nov 2013 10:58:03 -0500, Basu,Indrashish
wrote:
> Any updates on this ?
>
> Regards,
>
> Indra
>
> On
Thu, 07 Nov 2013 11:28:57 -0500, Basu,Indrashish wrote:
>
>> Hi Vinod,
>>
>> Please find the below details requested:
>>
>> A) ALL OF THE
TASKTRACKER LOG :
>>
>> Same what I have provided in the earlier
email.
>>
>> B) THE TASK-LOGS :
>>
>> SYSLOG :
>>
>> 2013-11-06
16:37:21,331 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing
JVM Metrics with processName=MAP, sessionId=
>> 2013-11-06 16:37:22,245
INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
>> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
>> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
>>
at java.io.DataInputStream.readByte(DataInputStream.java:267)
>> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>>
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>>
>> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
>> java.io.IOException: pipe child exception
>> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by:
java.net.SocketException: Broken pipe
>> at
java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>
... 3 more
>>
>> STDERR :
>>
>> Command:0received before
authentication. Exiting..
>>
>> STDOUT :
>>
>> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
>> deviceID:
>>
>> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>>
>>
attempt_201311061630_0001_m_000000_0
>>
>>
attempt_201311061630_0001_m_000001_0
>>
>>
attempt_201311061630_0001_m_000002_0
>>
>>
attempt_201311061630_0001_m_000000_1
>>
>>
attempt_201311061630_0001_m_000001_1
>>
>>
attempt_201311061630_0001_m_000003_0
>>
>>
attempt_201311061630_0001_m_000000_2
>>
>>
attempt_201311061630_0001_m_000001_2
>>
>>
attempt_201311061630_0001_m_000000_3
>>
>>
attempt_201311061630_0001_m_000001_3
>>
>> Kindly let me know in case
of any more details required.
>>
>> Thanks again for your time and
patience regarding this.
>>
>> Regards,
>>
>> Indrashish.
>>
>> On
Wed, 6 Nov 2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>>
>>>
Don't see anything in the logs that you pasted.
>>> Can you paste the
following in say pastebin?
>>> - All of the TaskTracker log
>>> - The
task-logs. These are syslog, stderr, stdout files for a specific
TaskAttempt.
>>> - And specific TaskAttempt's TaskAttemtpID that is
failing.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Nov 6, 2013, at 2:50
PM, Basu,Indrashish wrote:
>>>
>>>> Can anyone please assist regarding
this ?
>>>>
>>>> Thanks in advance
>>>>
>>>> Regards,
>>>>
>>>>
Indra
>>>>
>>>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for your help
regarding this. I checked the task logs, this is what it is giving as
output.
>>>>>
>>>>> 2013-11-06 06:40:05,541 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_1100862588
>>>>>
>>>>> 2013-11-06 06:40:05,553
INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
>>>>>
>>>>> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>>
>>>>>
2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>>
>>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>>
>>>>> 636_0001_m_000001_3
>>>>>
>>>>>
2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_-1960039766 given task:
attempt_20131106
>>>>>
>>>>> 0636_0001_m_000000_3
>>>>>
>>>>>
2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>>>>>
>>>>> 2013-11-06
06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>>
>>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>>
>>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>>
>>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>>
>>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>>
>>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>>
>>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>>
>>>>> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>>>>>
>>>>>
state:UNASSIGNED
>>>>>
>>>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>>
>>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>>
>>>>> 11060636_0001_m_000002_0
>>>>>
>>>>> 2013-11-06 06:40:14,909
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000000_3
>>>>>
>>>>> 2013-11-06
06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>>>>>
>>>>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>>
>>>>> 2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
>>>>>
>>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>>
>>>>> 36_0001_m_000002_0
>>>>>
>>>>> Regards,
>>>>>
>>>>>
Indrashish
>>>>>
>>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar
Vavilapalli wrote:
>>>>>
>>>>>> It seems like your pipes mapper is
exiting before consuming all the input. Did you check the task-logs on
the web UI?
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>>>>>
>>>>>>>
Hi,
>>>>>>>
>>>>>>> Can anyone kindly assist on this ?
>>>>>>>
>>>>>>>
Regards,
>>>>>>> Indrashish
>>>>>>>
>>>>>>> On Mon, 04 Nov 2013
10:23:23 -0500, Basu,Indrashish wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>
>>>>>>>> Any update on the below post ?
>>>>>>> solution as " The
*nopipe* example needs more documentation. It
src/test/org/apache/*hadoop*/mapred/*pipes*/ idth:
100%;">*WordCountInputFormat*.java, which
>>>>>>> ecific input split
format.
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid;
>>>>>>>
5px; width: 100%;">as the input split and won't work right. The *nopipe*
example should
>>>>>>>
>>>>>>>> probably be recoded *to* use libhdfs
*too*, but that is more
>>>>>>> ckquote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">complicated
>>>>>>>
>>>>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100
>>>>>>> le is using local file reads, it
will only work on a cluster could elaborate a bit about the
>>>>>>>
needs to be done exactly. To m
>>>>>>>
>>>>>>>> px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
>>>>>>> Hadoop.
Regard
>>>>>>> e> Indr
>>>>>>>
>>>>>>>> t: 5px; width: 100%;">On Thu,
31 Oct 2013 20:00:10 -0400,
>>>>>>> sh wrote: Hi,
>>>>>>> dding-left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> to
run a sample Hadoop GPU source code (kmeans algorithm)
>>>>>>> te>
2px
>>>>>>> argin-left: 5px; width: 100%;">13/10/31 13:43:12 WARN
mapred.JobClient: No job jar file set. User eft: 5px; w
>>>>>>>
cl
>>>>>>>
>>>>>>>> x; border-left: #1010ff 2px solid; margin-left:
5p
>>>>>>> %;">
>>>>>>> gin-left: 5px; width: 100%;">13/10/31 13:43:12
INFO mapred.FileInputFormat: Total input paths to : 100%;">
>>>>>>>>
>>>>>>>>> process : 1
>>>>>>> ft: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/10
>>>>>>>
>>>>>>>> e style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
>>>>>>>
00%;">13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>>> ; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Appli
>>>>>>>
>>>>>>>>
px solid; margin-left: 5px; width: 100%;">
>>>>>>>>
>>>>>>>>>
at
>>>>>>> ckquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>> at
org.a
>>>>>>>
>>>>>>>> px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> Caused
>>>>>>> .SocketException: Broken pipe at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>>> ockquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> &
>>>>>>>
>>>>>>>> quote>
>>>>>>> width: 100%;"> at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>
at java.io.BufferedOutputStream.write(Buffere
>>>>>>>
>>>>>>>> f 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
.io.DataOutputStream.write(DataOutputStream.java:107) at
>>>>>>> solid;
margin-left: 5px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333
>>>>>>>
>>>>>>>> argin-left: 5px; width: 100%;"> at
>>>>>>> lockquote> at c,
exec
>>>>>>> x solid; margin-left: 5px; width:
100%;">'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>>
>>>>>>> ote style="padding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>>>
margin-left: 5px; width: 100%;"> Regards,
>>>>>>> --
>>>>>>>
Indrashish Basu
>>>>>>> Graduate Student
>>>>>>> Departmen
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the
use of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>>
>>>>> --
>>>>>
>>>>> Indrashish
Basu
>>>>> Graduate Student
>>>>> Department of Electrical and
Computer Engineering
>>>>> University of Florida
>>>>
>>>> --
>>>>
>>>> Indrashish Basu
>>>> Graduate Student
>>>> Department of
Electrical and Computer Engineering
>>>> University of Florida
>>>
>>>
CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>
>> --
>>
>> Indrashish Basu
>>
Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>
> --
>
> Indrashish Basu
> Graduate
Student
> Department of Electrical and Computer Engineering
>
University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of
Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Any updates on this ?
Regards,
Indra
On Thu, 07 Nov 2013
11:28:57 -0500, Basu,Indrashish wrote:
> Hi Vinod,
>
> Please find
the below details requested:
>
> A) ALL OF THE TASKTRACKER LOG :
>
>
Same what I have provided in the earlier email.
>
> B) THE TASK-LOGS :
>
> SYSLOG :
>
> 2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
> 2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
> at
java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>
> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
> java.io.IOException: pipe child exception
> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by:
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
... 3 more
>
> STDERR :
>
> Command:0received before authentication.
Exiting..
>
> STDOUT :
>
> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
> deviceID:
>
> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>
>
attempt_201311061630_0001_m_000000_0
>
>
attempt_201311061630_0001_m_000001_0
>
>
attempt_201311061630_0001_m_000002_0
>
>
attempt_201311061630_0001_m_000000_1
>
>
attempt_201311061630_0001_m_000001_1
>
>
attempt_201311061630_0001_m_000003_0
>
>
attempt_201311061630_0001_m_000000_2
>
>
attempt_201311061630_0001_m_000001_2
>
>
attempt_201311061630_0001_m_000000_3
>
>
attempt_201311061630_0001_m_000001_3
>
> Kindly let me know in case of
any more details required.
>
> Thanks again for your time and patience
regarding this.
>
> Regards,
>
> Indrashish.
>
> On Wed, 6 Nov
2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>
>> Don't see
anything in the logs that you pasted.
>> Can you paste the following in
say pastebin?
>> - All of the TaskTracker log
>> - The task-logs.
These are syslog, stderr, stdout files for a specific TaskAttempt.
>> -
And specific TaskAttempt's TaskAttemtpID that is failing.
>>
>>
Thanks,
>> +Vinod
>>
>> On Nov 6, 2013, at 2:50 PM, Basu,Indrashish
wrote:
>>
>>> Can anyone please assist regarding this ?
>>>
>>>
Thanks in advance
>>>
>>> Regards,
>>>
>>> Indra
>>>
>>> On Wed,
06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi Vinod,
>>>>
>>>> Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
>>>>
>>>> 2013-11-06
06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>>
>>>>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>>
>>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>
>>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>
>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>
>>>> 636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>>
>>>> 0636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>
>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>
>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>
>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>
>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>
>>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>>
>>>> state:UNASSIGNED
>>>>
>>>> 2013-11-06 06:40:14,902
INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>
>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>
>>>>
11060636_0001_m_000002_0
>>>>
>>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>
>>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>>
>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>
>>>> 36_0001_m_000002_0
>>>>
>>>> Regards,
>>>>
>>>> Indrashish
>>>>
>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli
wrote:
>>>>
>>>>> It seems like your pipes mapper is exiting before
consuming all the input. Did you check the task-logs on the web UI?
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Nov 5, 2013, at
7:25 AM, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can
anyone kindly assist on this ?
>>>>>>
>>>>>> Regards,
>>>>>>
Indrashish
>>>>>>
>>>>>> On Mon, 04 Nov 2013 10:23:23 -0500,
Basu,Indrashish wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>
>>>>>>> Any
update on the below post ?
>>>>>> solution as " The *nopipe* example
needs more documentation. It run with the InputFormat from
>>>>>>
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">src/test/org/apache/*hadoop*/mapred/*pipes
>>>>>>
>>>>>>> x;
width: 100%;">*WordCountInputFormat*.java, wh
>>>>>> y specific input
split fo
>>>>>>
>>>>>>> left: 5px; border-left: #1010ff 2px so
>>>>>>
eft: 5px; width: 100%;">as the input split and won't work right. The
*nopipe* example should 100%;">probably be recoded *to* use libhdfs
*too*, but that is mo
>>>>>> e> c
>>>>>>
>>>>>>> g-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px;
>>>>>> ">C++ example
is using local file reads, it will only work on a cluster s or something
working across the cluster. "
>>>>>> ding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;">I would need some
more ligh
>>>>>>
>>>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">coul
>>>>>> bit about the same as what needs to be done exactly.
using Hadoop. Regards, Indrashi
>>>>>>
>>>>>>> px; width: 100%;">On
Thu, 31 Oct 2013 20:00:10 -0400, Basu
>>>>>> rote: ote>
>>>>>> 2px
solid; margin-left: 5px; width: 100%;"> rmr: cannot remove out
>>>>>>
>>>>>>> ft: #1010ff 2px
>>>>>> rgin-left: 5px; width: 100%;">
%;">classes
>>>>>> ound. See JobConf(Class) or 1010ff 2px solid;
margin-left: 5px; width: 100%;">13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to
>>>>>> lockquote> process :
1
>>>>>> ft: #1010ff 2px solid; margin-left: 5px; width: 100%;">13/10/31
13:43:13 INFO mapred.JobClient: Running
>>>>>>
>>>>>>> 1010ff 2px
solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:14 INFO ma
>>>>>>
t: map 0% reduce 0%
>>>>>>
>>>>>>> eft: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">
>>>>>> -left: 5px; width:
100%;">attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>> kquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(App
>>>>>>
>>>>>>> 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
>>>>>>>> at
>>>>>>
lockquote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apache.hado
>>>>>>
>>>>>>>
padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;">
>>>>>> width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
>>>>>>
>>>>>>> left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
C
>>>>>> a.net.SocketException: Broken pipe p; at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>> 010ff 2px
solid; margin-left: 5px; width: 100%;"> at
java.net.SocketOutputStream.
>>>>>>
>>>>>>> border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
>>>>>>
ketOutputStream.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>
id; margin-left: 5px; width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126
>>>>>>
>>>>>>> argin-left: 5px; width: 100%;"> at java.io.DataO
>>>>>>
rite(DataOutputStream.java:107)
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
>>>>>> px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>>>>> ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
'/app/hadoop/tmp/mapred/local/taskTracker/archive/
>>>>>>
bin/cpu-kmeans2D/cpu-kmeans2D' '0' < /dev/null 1>>
>>>>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
width: 100%;"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>> quote>
Regards, gineering
>>>>>>> University of Florida
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>
>>>>> CONFIDENTIALITY
NOTICE
>>>>> NOTICE: This message is intended for the use of the
individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>
>>>> --
>>>>
>>>> Indrashish
Basu
>>>> Graduate Student
>>>> Department of Electrical and Computer
Engineering
>>>> University of Florida
>>>
>>> --
>>>
>>> Indrashish
Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of
Electrical and Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and
Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Any updates on this ?
Regards,
Indra
On Thu, 07 Nov 2013
11:28:57 -0500, Basu,Indrashish wrote:
> Hi Vinod,
>
> Please find
the below details requested:
>
> A) ALL OF THE TASKTRACKER LOG :
>
>
Same what I have provided in the earlier email.
>
> B) THE TASK-LOGS :
>
> SYSLOG :
>
> 2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
> 2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
> at
java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>
> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
> java.io.IOException: pipe child exception
> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by:
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
... 3 more
>
> STDERR :
>
> Command:0received before authentication.
Exiting..
>
> STDOUT :
>
> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
> deviceID:
>
> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>
>
attempt_201311061630_0001_m_000000_0
>
>
attempt_201311061630_0001_m_000001_0
>
>
attempt_201311061630_0001_m_000002_0
>
>
attempt_201311061630_0001_m_000000_1
>
>
attempt_201311061630_0001_m_000001_1
>
>
attempt_201311061630_0001_m_000003_0
>
>
attempt_201311061630_0001_m_000000_2
>
>
attempt_201311061630_0001_m_000001_2
>
>
attempt_201311061630_0001_m_000000_3
>
>
attempt_201311061630_0001_m_000001_3
>
> Kindly let me know in case of
any more details required.
>
> Thanks again for your time and patience
regarding this.
>
> Regards,
>
> Indrashish.
>
> On Wed, 6 Nov
2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>
>> Don't see
anything in the logs that you pasted.
>> Can you paste the following in
say pastebin?
>> - All of the TaskTracker log
>> - The task-logs.
These are syslog, stderr, stdout files for a specific TaskAttempt.
>> -
And specific TaskAttempt's TaskAttemtpID that is failing.
>>
>>
Thanks,
>> +Vinod
>>
>> On Nov 6, 2013, at 2:50 PM, Basu,Indrashish
wrote:
>>
>>> Can anyone please assist regarding this ?
>>>
>>>
Thanks in advance
>>>
>>> Regards,
>>>
>>> Indra
>>>
>>> On Wed,
06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi Vinod,
>>>>
>>>> Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
>>>>
>>>> 2013-11-06
06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>>
>>>>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>>
>>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>
>>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>
>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>
>>>> 636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>>
>>>> 0636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>
>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>
>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>
>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>
>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>
>>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>>
>>>> state:UNASSIGNED
>>>>
>>>> 2013-11-06 06:40:14,902
INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>
>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>
>>>>
11060636_0001_m_000002_0
>>>>
>>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>
>>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>>
>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>
>>>> 36_0001_m_000002_0
>>>>
>>>> Regards,
>>>>
>>>> Indrashish
>>>>
>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli
wrote:
>>>>
>>>>> It seems like your pipes mapper is exiting before
consuming all the input. Did you check the task-logs on the web UI?
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Nov 5, 2013, at
7:25 AM, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can
anyone kindly assist on this ?
>>>>>>
>>>>>> Regards,
>>>>>>
Indrashish
>>>>>>
>>>>>> On Mon, 04 Nov 2013 10:23:23 -0500,
Basu,Indrashish wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>
>>>>>>> Any
update on the below post ?
>>>>>> solution as " The *nopipe* example
needs more documentation. It run with the InputFormat from
>>>>>>
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">src/test/org/apache/*hadoop*/mapred/*pipes
>>>>>>
>>>>>>> x;
width: 100%;">*WordCountInputFormat*.java, wh
>>>>>> y specific input
split fo
>>>>>>
>>>>>>> left: 5px; border-left: #1010ff 2px so
>>>>>>
eft: 5px; width: 100%;">as the input split and won't work right. The
*nopipe* example should 100%;">probably be recoded *to* use libhdfs
*too*, but that is mo
>>>>>> e> c
>>>>>>
>>>>>>> g-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px;
>>>>>> ">C++ example
is using local file reads, it will only work on a cluster s or something
working across the cluster. "
>>>>>> ding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;">I would need some
more ligh
>>>>>>
>>>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">coul
>>>>>> bit about the same as what needs to be done exactly.
using Hadoop. Regards, Indrashi
>>>>>>
>>>>>>> px; width: 100%;">On
Thu, 31 Oct 2013 20:00:10 -0400, Basu
>>>>>> rote: ote>
>>>>>> 2px
solid; margin-left: 5px; width: 100%;"> rmr: cannot remove out
>>>>>>
>>>>>>> ft: #1010ff 2px
>>>>>> rgin-left: 5px; width: 100%;">
%;">classes
>>>>>> ound. See JobConf(Class) or 1010ff 2px solid;
margin-left: 5px; width: 100%;">13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to
>>>>>> lockquote> process :
1
>>>>>> ft: #1010ff 2px solid; margin-left: 5px; width: 100%;">13/10/31
13:43:13 INFO mapred.JobClient: Running
>>>>>>
>>>>>>> 1010ff 2px
solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:14 INFO ma
>>>>>>
t: map 0% reduce 0%
>>>>>>
>>>>>>> eft: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">
>>>>>> -left: 5px; width:
100%;">attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>> kquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(App
>>>>>>
>>>>>>> 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
>>>>>>>> at
>>>>>>
lockquote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apache.hado
>>>>>>
>>>>>>>
padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;">
>>>>>> width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
>>>>>>
>>>>>>> left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
C
>>>>>> a.net.SocketException: Broken pipe p; at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>> 010ff 2px
solid; margin-left: 5px; width: 100%;"> at
java.net.SocketOutputStream.
>>>>>>
>>>>>>> border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
>>>>>>
ketOutputStream.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>
id; margin-left: 5px; width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126
>>>>>>
>>>>>>> argin-left: 5px; width: 100%;"> at java.io.DataO
>>>>>>
rite(DataOutputStream.java:107)
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
>>>>>> px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>>>>> ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
'/app/hadoop/tmp/mapred/local/taskTracker/archive/
>>>>>>
bin/cpu-kmeans2D/cpu-kmeans2D' '0' < /dev/null 1>>
>>>>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
width: 100%;"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>> quote>
Regards, gineering
>>>>>>> University of Florida
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>
>>>>> CONFIDENTIALITY
NOTICE
>>>>> NOTICE: This message is intended for the use of the
individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>
>>>> --
>>>>
>>>> Indrashish
Basu
>>>> Graduate Student
>>>> Department of Electrical and Computer
Engineering
>>>> University of Florida
>>>
>>> --
>>>
>>> Indrashish
Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of
Electrical and Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and
Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Any updates on this ?
Regards,
Indra
On Thu, 07 Nov 2013
11:28:57 -0500, Basu,Indrashish wrote:
> Hi Vinod,
>
> Please find
the below details requested:
>
> A) ALL OF THE TASKTRACKER LOG :
>
>
Same what I have provided in the earlier email.
>
> B) THE TASK-LOGS :
>
> SYSLOG :
>
> 2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
> 2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
> at
java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>
> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
> java.io.IOException: pipe child exception
> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by:
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
... 3 more
>
> STDERR :
>
> Command:0received before authentication.
Exiting..
>
> STDOUT :
>
> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
> deviceID:
>
> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>
>
attempt_201311061630_0001_m_000000_0
>
>
attempt_201311061630_0001_m_000001_0
>
>
attempt_201311061630_0001_m_000002_0
>
>
attempt_201311061630_0001_m_000000_1
>
>
attempt_201311061630_0001_m_000001_1
>
>
attempt_201311061630_0001_m_000003_0
>
>
attempt_201311061630_0001_m_000000_2
>
>
attempt_201311061630_0001_m_000001_2
>
>
attempt_201311061630_0001_m_000000_3
>
>
attempt_201311061630_0001_m_000001_3
>
> Kindly let me know in case of
any more details required.
>
> Thanks again for your time and patience
regarding this.
>
> Regards,
>
> Indrashish.
>
> On Wed, 6 Nov
2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>
>> Don't see
anything in the logs that you pasted.
>> Can you paste the following in
say pastebin?
>> - All of the TaskTracker log
>> - The task-logs.
These are syslog, stderr, stdout files for a specific TaskAttempt.
>> -
And specific TaskAttempt's TaskAttemtpID that is failing.
>>
>>
Thanks,
>> +Vinod
>>
>> On Nov 6, 2013, at 2:50 PM, Basu,Indrashish
wrote:
>>
>>> Can anyone please assist regarding this ?
>>>
>>>
Thanks in advance
>>>
>>> Regards,
>>>
>>> Indra
>>>
>>> On Wed,
06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi Vinod,
>>>>
>>>> Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
>>>>
>>>> 2013-11-06
06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>>
>>>>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>>
>>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>
>>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>
>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>
>>>> 636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>>
>>>> 0636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>
>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>
>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>
>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>
>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>
>>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>>
>>>> state:UNASSIGNED
>>>>
>>>> 2013-11-06 06:40:14,902
INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>
>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>
>>>>
11060636_0001_m_000002_0
>>>>
>>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>
>>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>>
>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>
>>>> 36_0001_m_000002_0
>>>>
>>>> Regards,
>>>>
>>>> Indrashish
>>>>
>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli
wrote:
>>>>
>>>>> It seems like your pipes mapper is exiting before
consuming all the input. Did you check the task-logs on the web UI?
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Nov 5, 2013, at
7:25 AM, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can
anyone kindly assist on this ?
>>>>>>
>>>>>> Regards,
>>>>>>
Indrashish
>>>>>>
>>>>>> On Mon, 04 Nov 2013 10:23:23 -0500,
Basu,Indrashish wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>
>>>>>>> Any
update on the below post ?
>>>>>> solution as " The *nopipe* example
needs more documentation. It run with the InputFormat from
>>>>>>
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">src/test/org/apache/*hadoop*/mapred/*pipes
>>>>>>
>>>>>>> x;
width: 100%;">*WordCountInputFormat*.java, wh
>>>>>> y specific input
split fo
>>>>>>
>>>>>>> left: 5px; border-left: #1010ff 2px so
>>>>>>
eft: 5px; width: 100%;">as the input split and won't work right. The
*nopipe* example should 100%;">probably be recoded *to* use libhdfs
*too*, but that is mo
>>>>>> e> c
>>>>>>
>>>>>>> g-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px;
>>>>>> ">C++ example
is using local file reads, it will only work on a cluster s or something
working across the cluster. "
>>>>>> ding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;">I would need some
more ligh
>>>>>>
>>>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">coul
>>>>>> bit about the same as what needs to be done exactly.
using Hadoop. Regards, Indrashi
>>>>>>
>>>>>>> px; width: 100%;">On
Thu, 31 Oct 2013 20:00:10 -0400, Basu
>>>>>> rote: ote>
>>>>>> 2px
solid; margin-left: 5px; width: 100%;"> rmr: cannot remove out
>>>>>>
>>>>>>> ft: #1010ff 2px
>>>>>> rgin-left: 5px; width: 100%;">
%;">classes
>>>>>> ound. See JobConf(Class) or 1010ff 2px solid;
margin-left: 5px; width: 100%;">13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to
>>>>>> lockquote> process :
1
>>>>>> ft: #1010ff 2px solid; margin-left: 5px; width: 100%;">13/10/31
13:43:13 INFO mapred.JobClient: Running
>>>>>>
>>>>>>> 1010ff 2px
solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:14 INFO ma
>>>>>>
t: map 0% reduce 0%
>>>>>>
>>>>>>> eft: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">
>>>>>> -left: 5px; width:
100%;">attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>> kquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(App
>>>>>>
>>>>>>> 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
>>>>>>>> at
>>>>>>
lockquote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apache.hado
>>>>>>
>>>>>>>
padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;">
>>>>>> width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
>>>>>>
>>>>>>> left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
C
>>>>>> a.net.SocketException: Broken pipe p; at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>> 010ff 2px
solid; margin-left: 5px; width: 100%;"> at
java.net.SocketOutputStream.
>>>>>>
>>>>>>> border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
>>>>>>
ketOutputStream.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>
id; margin-left: 5px; width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126
>>>>>>
>>>>>>> argin-left: 5px; width: 100%;"> at java.io.DataO
>>>>>>
rite(DataOutputStream.java:107)
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
>>>>>> px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>>>>> ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
'/app/hadoop/tmp/mapred/local/taskTracker/archive/
>>>>>>
bin/cpu-kmeans2D/cpu-kmeans2D' '0' < /dev/null 1>>
>>>>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
width: 100%;"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>> quote>
Regards, gineering
>>>>>>> University of Florida
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>
>>>>> CONFIDENTIALITY
NOTICE
>>>>> NOTICE: This message is intended for the use of the
individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>
>>>> --
>>>>
>>>> Indrashish
Basu
>>>> Graduate Student
>>>> Department of Electrical and Computer
Engineering
>>>> University of Florida
>>>
>>> --
>>>
>>> Indrashish
Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of
Electrical and Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and
Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Any updates on this ?
Regards,
Indra
On Thu, 07 Nov 2013
11:28:57 -0500, Basu,Indrashish wrote:
> Hi Vinod,
>
> Please find
the below details requested:
>
> A) ALL OF THE TASKTRACKER LOG :
>
>
Same what I have provided in the earlier email.
>
> B) THE TASK-LOGS :
>
> SYSLOG :
>
> 2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
> 2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
> 2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
>
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
> 2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
>
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
> at
java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
>
> 2013-11-06 16:37:23,157 INFO
org.apache.hadoop.mapred.pipes.Application: Aborting because of
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> 2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
> java.io.IOException: pipe child exception
> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by:
java.net.SocketException: Broken pipe
> at
java.net.SocketOutputStream.socketWrite0(Native Method)
> at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>
at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>
... 3 more
>
> STDERR :
>
> Command:0received before authentication.
Exiting..
>
> STDOUT :
>
> cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
>
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
>
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
> deviceID:
>
> C) SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
>
>
attempt_201311061630_0001_m_000000_0
>
>
attempt_201311061630_0001_m_000001_0
>
>
attempt_201311061630_0001_m_000002_0
>
>
attempt_201311061630_0001_m_000000_1
>
>
attempt_201311061630_0001_m_000001_1
>
>
attempt_201311061630_0001_m_000003_0
>
>
attempt_201311061630_0001_m_000000_2
>
>
attempt_201311061630_0001_m_000001_2
>
>
attempt_201311061630_0001_m_000000_3
>
>
attempt_201311061630_0001_m_000001_3
>
> Kindly let me know in case of
any more details required.
>
> Thanks again for your time and patience
regarding this.
>
> Regards,
>
> Indrashish.
>
> On Wed, 6 Nov
2013 16:58:53 -0800, Vinod Kumar Vavilapalli wrote:
>
>> Don't see
anything in the logs that you pasted.
>> Can you paste the following in
say pastebin?
>> - All of the TaskTracker log
>> - The task-logs.
These are syslog, stderr, stdout files for a specific TaskAttempt.
>> -
And specific TaskAttempt's TaskAttemtpID that is failing.
>>
>>
Thanks,
>> +Vinod
>>
>> On Nov 6, 2013, at 2:50 PM, Basu,Indrashish
wrote:
>>
>>> Can anyone please assist regarding this ?
>>>
>>>
Thanks in advance
>>>
>>> Regards,
>>>
>>> Indra
>>>
>>> On Wed,
06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi Vinod,
>>>>
>>>> Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
>>>>
>>>> 2013-11-06
06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>>
>>>>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>>
>>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>>
>>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>>
>>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>>
>>>> 636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>>
>>>> 0636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>>
>>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>>
>>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>>
>>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>>
>>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>>
>>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>>
>>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>>
>>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>>
>>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>>
>>>> state:UNASSIGNED
>>>>
>>>> 2013-11-06 06:40:14,902
INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>>
>>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>>
>>>>
11060636_0001_m_000002_0
>>>>
>>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>>
>>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>>
>>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>>
>>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>>
>>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>>
>>>> 36_0001_m_000002_0
>>>>
>>>> Regards,
>>>>
>>>> Indrashish
>>>>
>>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli
wrote:
>>>>
>>>>> It seems like your pipes mapper is exiting before
consuming all the input. Did you check the task-logs on the web UI?
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Nov 5, 2013, at
7:25 AM, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can
anyone kindly assist on this ?
>>>>>>
>>>>>> Regards,
>>>>>>
Indrashish
>>>>>>
>>>>>> On Mon, 04 Nov 2013 10:23:23 -0500,
Basu,Indrashish wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>
>>>>>>> Any
update on the below post ?
>>>>>> solution as " The *nopipe* example
needs more documentation. It run with the InputFormat from
>>>>>>
border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">src/test/org/apache/*hadoop*/mapred/*pipes
>>>>>>
>>>>>>> x;
width: 100%;">*WordCountInputFormat*.java, wh
>>>>>> y specific input
split fo
>>>>>>
>>>>>>> left: 5px; border-left: #1010ff 2px so
>>>>>>
eft: 5px; width: 100%;">as the input split and won't work right. The
*nopipe* example should 100%;">probably be recoded *to* use libhdfs
*too*, but that is mo
>>>>>> e> c
>>>>>>
>>>>>>> g-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px;
>>>>>> ">C++ example
is using local file reads, it will only work on a cluster s or something
working across the cluster. "
>>>>>> ding-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;">I would need some
more ligh
>>>>>>
>>>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">coul
>>>>>> bit about the same as what needs to be done exactly.
using Hadoop. Regards, Indrashi
>>>>>>
>>>>>>> px; width: 100%;">On
Thu, 31 Oct 2013 20:00:10 -0400, Basu
>>>>>> rote: ote>
>>>>>> 2px
solid; margin-left: 5px; width: 100%;"> rmr: cannot remove out
>>>>>>
>>>>>>> ft: #1010ff 2px
>>>>>> rgin-left: 5px; width: 100%;">
%;">classes
>>>>>> ound. See JobConf(Class) or 1010ff 2px solid;
margin-left: 5px; width: 100%;">13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to
>>>>>> lockquote> process :
1
>>>>>> ft: #1010ff 2px solid; margin-left: 5px; width: 100%;">13/10/31
13:43:13 INFO mapred.JobClient: Running
>>>>>>
>>>>>>> 1010ff 2px
solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:14 INFO ma
>>>>>>
t: map 0% reduce 0%
>>>>>>
>>>>>>> eft: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">
>>>>>> -left: 5px; width:
100%;">attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>>> kquote
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(App
>>>>>>
>>>>>>> 2px
solid; margin-left: 5px; width: 100%;">
>>>>>>>
>>>>>>>> at
>>>>>>
lockquote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apache.hado
>>>>>>
>>>>>>>
padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;">
>>>>>> width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> #1010ff
2px solid; margin-left: 5px; width: 100%;"> at
>>>>>>
>>>>>>> left:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
C
>>>>>> a.net.SocketException: Broken pipe p; at
java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>>> 010ff 2px
solid; margin-left: 5px; width: 100%;"> at
java.net.SocketOutputStream.
>>>>>>
>>>>>>> border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at
>>>>>>
ketOutputStream.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>
id; margin-left: 5px; width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126
>>>>>>
>>>>>>> argin-left: 5px; width: 100%;"> at java.io.DataO
>>>>>>
rite(DataOutputStream.java:107)
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
>>>>>> px; width: 100%;">
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>>>>> ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
'/app/hadoop/tmp/mapred/local/taskTracker/archive/
>>>>>>
bin/cpu-kmeans2D/cpu-kmeans2D' '0' < /dev/null 1>>
>>>>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
width: 100%;"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>>> quote>
Regards, gineering
>>>>>>> University of Florida
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>
>>>>> CONFIDENTIALITY
NOTICE
>>>>> NOTICE: This message is intended for the use of the
individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>>
>>>> --
>>>>
>>>> Indrashish
Basu
>>>> Graduate Student
>>>> Department of Electrical and Computer
Engineering
>>>> University of Florida
>>>
>>> --
>>>
>>> Indrashish
Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of
Electrical and Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and
Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Please find the below details requested:
A) ALL OF
THE TASKTRACKER LOG :
Same what I have provided in the earlier email.
B) THE TASK-LOGS :
SYSLOG :
2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer =
262144/327680
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
at
java.io.DataInputStream.readByte(DataInputStream.java:267)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
2013-11-06
16:37:23,157 INFO org.apache.hadoop.mapred.pipes.Application: Aborting
because of java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by:
java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
... 3 more
STDERR :
Command:0received before authentication.
Exiting..
STDOUT :
cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
deviceID:
C)
SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
attempt_201311061630_0001_m_000000_0
attempt_201311061630_0001_m_000001_0
attempt_201311061630_0001_m_000002_0
attempt_201311061630_0001_m_000000_1
attempt_201311061630_0001_m_000001_1
attempt_201311061630_0001_m_000003_0
attempt_201311061630_0001_m_000000_2
attempt_201311061630_0001_m_000001_2
attempt_201311061630_0001_m_000000_3
attempt_201311061630_0001_m_000001_3
Kindly let me know in case of any
more details required.
Thanks again for your time and patience
regarding this.
Regards,
Indrashish.
On Wed, 6 Nov 2013 16:58:53
-0800, Vinod Kumar Vavilapalli wrote:
> Don't see anything in the logs
that you pasted.
> Can you paste the following in say pastebin?
> -
All of the TaskTracker log
> - The task-logs. These are syslog, stderr,
stdout files for a specific TaskAttempt.
> - And specific TaskAttempt's
TaskAttemtpID that is failing.
>
> Thanks,
> +Vinod
>
> On Nov 6,
2013, at 2:50 PM, Basu,Indrashish wrote:
>
>> Can anyone please assist
regarding this ?
>>
>> Thanks in advance
>>
>> Regards,
>>
>>
Indra
>>
>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>
>>> Hi Vinod,
>>>
>>> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>>>
>>>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>
>>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>
>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>
>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>
>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>
>>> 636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>
>>> 0636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>
>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>
>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>
>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>
>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>
>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>
>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>
>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>
>>> state:UNASSIGNED
>>>
>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>
>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>
>>>
11060636_0001_m_000002_0
>>>
>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>
>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>
>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>
>>> 36_0001_m_000002_0
>>>
>>> Regards,
>>>
>>> Indrashish
>>>
>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>>
>>>> It seems like your pipes mapper is exiting before consuming
all the input. Did you check the task-logs on the web UI?
>>>>
>>>>
Thanks,
>>>> +Vinod
>>>>
>>>> On Nov 5, 2013, at 7:25 AM,
Basu,Indrashish wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can anyone kindly
assist on this ?
>>>>>
>>>>> Regards,
>>>>> Indrashish
>>>>>
>>>>> On
Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi
All,
>>>>>
>>>>>> Any update on the below post ?
>>>>> solution as "
The *nopipe* example needs more documentation. It is run with the
InputFormat from
>>>>> x; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">src/test/org/apache/*hadoop*/mapred/*p
>>>>>
>>>>>>
: 5px; width: 100%;">*WordCountInputFormat*.java
>>>>> very specific
input split probably be recoded *to* use libhdfs *too*, but t
>>>>>
blockquote> if you have nfs or something working across the cluster.
"
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"
>>>>>
>>>>>> ding-left: 5px;
border-left: #1010ff 2px solid; margin-l
>>>>> th: 100%;">could
elaborate a bit about the same as what needs to be done exactly.
>>>>>
>>>>>>> To mention, I am trying to run a sample KMeans algorithm on a
GPU
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">usin
>>>>>
>>>>>> eft: 5px;
width: 100%;">Thanks in advance.
>>>>> g-left: 5px; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;">Regards, ndrashish.
>>>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;">On Thu, 31 Oct
2013 20:00:10 -0400, Basu,Indrashish wrote: : 100%;">
>>>>> id;
margin-left: 5px; width: 100%;">Hi, I am trying to run a sample Hadoop
GPU source
>>>>> algorithm) M processor and getting the below error. Can
anyone please
>>>>> ckquote> 2px solid; marg
>>>>> width: 100%;">rmr:
cannot remove output: No such file or directory. uote sty
>>>>> eft:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">13/10/31 13:43:12 WARN mapred.
>>>>>
>>>>>> px; border-
>>>>>
2px solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to eft: 5px; width: 100%;">
>>>>>>
>>>>>>> process : 1
>>>>> 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:13 INFO
mapred.JobClient:
>>>>>
>>>>>> er-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:14
>>>>> JobClient: map
0% reduce 0% attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>> olid; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
idth: 100%;">
>>>>>>
>>>>>>> at
>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apach
>>>>>
>>>>>>
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">
>>>>> 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) ;at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> d;
margin-left: 5px; width: 100%;"> at org.apac
>>>>>
>>>>>> 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> Caused
by
>>>>> cketException: Broken pipe at
java.net.SocketOutputStream.socketWrite(Soc
>>>>>
>>>>>> #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at java.net.Socke
>>>>>
.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
100%;"> at java.io.DataOutputStream.write(DataO
>>>>> ava:107)
>>>>>
">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>>>>
>>>>>> at
>>>>> padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>>>>
>>>>>>>
>>>>>>
>>>>>>> at
>>>>> olid; margin-left: 5px;
width: 100%;">
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> ... 3 more
>>>>> ote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans
>>>>>
te> < /dev/null 1>>
>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>> px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
Regards,
>>>>> --
>>>>> Indrashish Basu
>>>>> Graduate Student
>>>>>
Department o
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>
>>>>
CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>
>>> --
>>>
>>> Indrashish Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> --
>>
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer
Engineering
>> University of Florida
>
> CONFIDENTIALITY NOTICE
>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Volunteer
Posted by Mike <mi...@unitedrmr.com>.
Hi guys
I would like to volunteer and help with hadoop. Could you point me in the right direction?
Best regards
Mike
Volunteer
Posted by Mike <mi...@unitedrmr.com>.
Hi guys
I would like to volunteer and help with hadoop. Could you point me in the right direction?
Best regards
Mike
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Please find the below details requested:
A) ALL OF
THE TASKTRACKER LOG :
Same what I have provided in the earlier email.
B) THE TASK-LOGS :
SYSLOG :
2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer =
262144/327680
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
at
java.io.DataInputStream.readByte(DataInputStream.java:267)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
2013-11-06
16:37:23,157 INFO org.apache.hadoop.mapred.pipes.Application: Aborting
because of java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by:
java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
... 3 more
STDERR :
Command:0received before authentication.
Exiting..
STDOUT :
cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
deviceID:
C)
SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
attempt_201311061630_0001_m_000000_0
attempt_201311061630_0001_m_000001_0
attempt_201311061630_0001_m_000002_0
attempt_201311061630_0001_m_000000_1
attempt_201311061630_0001_m_000001_1
attempt_201311061630_0001_m_000003_0
attempt_201311061630_0001_m_000000_2
attempt_201311061630_0001_m_000001_2
attempt_201311061630_0001_m_000000_3
attempt_201311061630_0001_m_000001_3
Kindly let me know in case of any
more details required.
Thanks again for your time and patience
regarding this.
Regards,
Indrashish.
On Wed, 6 Nov 2013 16:58:53
-0800, Vinod Kumar Vavilapalli wrote:
> Don't see anything in the logs
that you pasted.
> Can you paste the following in say pastebin?
> -
All of the TaskTracker log
> - The task-logs. These are syslog, stderr,
stdout files for a specific TaskAttempt.
> - And specific TaskAttempt's
TaskAttemtpID that is failing.
>
> Thanks,
> +Vinod
>
> On Nov 6,
2013, at 2:50 PM, Basu,Indrashish wrote:
>
>> Can anyone please assist
regarding this ?
>>
>> Thanks in advance
>>
>> Regards,
>>
>>
Indra
>>
>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>
>>> Hi Vinod,
>>>
>>> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>>>
>>>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>
>>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>
>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>
>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>
>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>
>>> 636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>
>>> 0636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>
>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>
>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>
>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>
>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>
>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>
>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>
>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>
>>> state:UNASSIGNED
>>>
>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>
>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>
>>>
11060636_0001_m_000002_0
>>>
>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>
>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>
>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>
>>> 36_0001_m_000002_0
>>>
>>> Regards,
>>>
>>> Indrashish
>>>
>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>>
>>>> It seems like your pipes mapper is exiting before consuming
all the input. Did you check the task-logs on the web UI?
>>>>
>>>>
Thanks,
>>>> +Vinod
>>>>
>>>> On Nov 5, 2013, at 7:25 AM,
Basu,Indrashish wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can anyone kindly
assist on this ?
>>>>>
>>>>> Regards,
>>>>> Indrashish
>>>>>
>>>>> On
Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi
All,
>>>>>
>>>>>> Any update on the below post ?
>>>>> solution as "
The *nopipe* example needs more documentation. It is run with the
InputFormat from
>>>>> x; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">src/test/org/apache/*hadoop*/mapred/*p
>>>>>
>>>>>>
: 5px; width: 100%;">*WordCountInputFormat*.java
>>>>> very specific
input split probably be recoded *to* use libhdfs *too*, but t
>>>>>
blockquote> if you have nfs or something working across the cluster.
"
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"
>>>>>
>>>>>> ding-left: 5px;
border-left: #1010ff 2px solid; margin-l
>>>>> th: 100%;">could
elaborate a bit about the same as what needs to be done exactly.
>>>>>
>>>>>>> To mention, I am trying to run a sample KMeans algorithm on a
GPU
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">usin
>>>>>
>>>>>> eft: 5px;
width: 100%;">Thanks in advance.
>>>>> g-left: 5px; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;">Regards, ndrashish.
>>>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;">On Thu, 31 Oct
2013 20:00:10 -0400, Basu,Indrashish wrote: : 100%;">
>>>>> id;
margin-left: 5px; width: 100%;">Hi, I am trying to run a sample Hadoop
GPU source
>>>>> algorithm) M processor and getting the below error. Can
anyone please
>>>>> ckquote> 2px solid; marg
>>>>> width: 100%;">rmr:
cannot remove output: No such file or directory. uote sty
>>>>> eft:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">13/10/31 13:43:12 WARN mapred.
>>>>>
>>>>>> px; border-
>>>>>
2px solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to eft: 5px; width: 100%;">
>>>>>>
>>>>>>> process : 1
>>>>> 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:13 INFO
mapred.JobClient:
>>>>>
>>>>>> er-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:14
>>>>> JobClient: map
0% reduce 0% attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>> olid; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
idth: 100%;">
>>>>>>
>>>>>>> at
>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apach
>>>>>
>>>>>>
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">
>>>>> 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) ;at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> d;
margin-left: 5px; width: 100%;"> at org.apac
>>>>>
>>>>>> 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> Caused
by
>>>>> cketException: Broken pipe at
java.net.SocketOutputStream.socketWrite(Soc
>>>>>
>>>>>> #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at java.net.Socke
>>>>>
.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
100%;"> at java.io.DataOutputStream.write(DataO
>>>>> ava:107)
>>>>>
">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>>>>
>>>>>> at
>>>>> padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>>>>
>>>>>>>
>>>>>>
>>>>>>> at
>>>>> olid; margin-left: 5px;
width: 100%;">
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> ... 3 more
>>>>> ote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans
>>>>>
te> < /dev/null 1>>
>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>> px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
Regards,
>>>>> --
>>>>> Indrashish Basu
>>>>> Graduate Student
>>>>>
Department o
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>
>>>>
CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>
>>> --
>>>
>>> Indrashish Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> --
>>
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer
Engineering
>> University of Florida
>
> CONFIDENTIALITY NOTICE
>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Please find the below details requested:
A) ALL OF
THE TASKTRACKER LOG :
Same what I have provided in the earlier email.
B) THE TASK-LOGS :
SYSLOG :
2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer =
262144/327680
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
at
java.io.DataInputStream.readByte(DataInputStream.java:267)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
2013-11-06
16:37:23,157 INFO org.apache.hadoop.mapred.pipes.Application: Aborting
because of java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by:
java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
... 3 more
STDERR :
Command:0received before authentication.
Exiting..
STDOUT :
cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
deviceID:
C)
SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
attempt_201311061630_0001_m_000000_0
attempt_201311061630_0001_m_000001_0
attempt_201311061630_0001_m_000002_0
attempt_201311061630_0001_m_000000_1
attempt_201311061630_0001_m_000001_1
attempt_201311061630_0001_m_000003_0
attempt_201311061630_0001_m_000000_2
attempt_201311061630_0001_m_000001_2
attempt_201311061630_0001_m_000000_3
attempt_201311061630_0001_m_000001_3
Kindly let me know in case of any
more details required.
Thanks again for your time and patience
regarding this.
Regards,
Indrashish.
On Wed, 6 Nov 2013 16:58:53
-0800, Vinod Kumar Vavilapalli wrote:
> Don't see anything in the logs
that you pasted.
> Can you paste the following in say pastebin?
> -
All of the TaskTracker log
> - The task-logs. These are syslog, stderr,
stdout files for a specific TaskAttempt.
> - And specific TaskAttempt's
TaskAttemtpID that is failing.
>
> Thanks,
> +Vinod
>
> On Nov 6,
2013, at 2:50 PM, Basu,Indrashish wrote:
>
>> Can anyone please assist
regarding this ?
>>
>> Thanks in advance
>>
>> Regards,
>>
>>
Indra
>>
>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>
>>> Hi Vinod,
>>>
>>> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>>>
>>>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>
>>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>
>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>
>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>
>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>
>>> 636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>
>>> 0636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>
>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>
>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>
>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>
>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>
>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>
>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>
>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>
>>> state:UNASSIGNED
>>>
>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>
>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>
>>>
11060636_0001_m_000002_0
>>>
>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>
>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>
>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>
>>> 36_0001_m_000002_0
>>>
>>> Regards,
>>>
>>> Indrashish
>>>
>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>>
>>>> It seems like your pipes mapper is exiting before consuming
all the input. Did you check the task-logs on the web UI?
>>>>
>>>>
Thanks,
>>>> +Vinod
>>>>
>>>> On Nov 5, 2013, at 7:25 AM,
Basu,Indrashish wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can anyone kindly
assist on this ?
>>>>>
>>>>> Regards,
>>>>> Indrashish
>>>>>
>>>>> On
Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi
All,
>>>>>
>>>>>> Any update on the below post ?
>>>>> solution as "
The *nopipe* example needs more documentation. It is run with the
InputFormat from
>>>>> x; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">src/test/org/apache/*hadoop*/mapred/*p
>>>>>
>>>>>>
: 5px; width: 100%;">*WordCountInputFormat*.java
>>>>> very specific
input split probably be recoded *to* use libhdfs *too*, but t
>>>>>
blockquote> if you have nfs or something working across the cluster.
"
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"
>>>>>
>>>>>> ding-left: 5px;
border-left: #1010ff 2px solid; margin-l
>>>>> th: 100%;">could
elaborate a bit about the same as what needs to be done exactly.
>>>>>
>>>>>>> To mention, I am trying to run a sample KMeans algorithm on a
GPU
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">usin
>>>>>
>>>>>> eft: 5px;
width: 100%;">Thanks in advance.
>>>>> g-left: 5px; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;">Regards, ndrashish.
>>>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;">On Thu, 31 Oct
2013 20:00:10 -0400, Basu,Indrashish wrote: : 100%;">
>>>>> id;
margin-left: 5px; width: 100%;">Hi, I am trying to run a sample Hadoop
GPU source
>>>>> algorithm) M processor and getting the below error. Can
anyone please
>>>>> ckquote> 2px solid; marg
>>>>> width: 100%;">rmr:
cannot remove output: No such file or directory. uote sty
>>>>> eft:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">13/10/31 13:43:12 WARN mapred.
>>>>>
>>>>>> px; border-
>>>>>
2px solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to eft: 5px; width: 100%;">
>>>>>>
>>>>>>> process : 1
>>>>> 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:13 INFO
mapred.JobClient:
>>>>>
>>>>>> er-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:14
>>>>> JobClient: map
0% reduce 0% attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>> olid; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
idth: 100%;">
>>>>>>
>>>>>>> at
>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apach
>>>>>
>>>>>>
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">
>>>>> 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) ;at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> d;
margin-left: 5px; width: 100%;"> at org.apac
>>>>>
>>>>>> 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> Caused
by
>>>>> cketException: Broken pipe at
java.net.SocketOutputStream.socketWrite(Soc
>>>>>
>>>>>> #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at java.net.Socke
>>>>>
.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
100%;"> at java.io.DataOutputStream.write(DataO
>>>>> ava:107)
>>>>>
">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>>>>
>>>>>> at
>>>>> padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>>>>
>>>>>>>
>>>>>>
>>>>>>> at
>>>>> olid; margin-left: 5px;
width: 100%;">
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> ... 3 more
>>>>> ote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans
>>>>>
te> < /dev/null 1>>
>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>> px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
Regards,
>>>>> --
>>>>> Indrashish Basu
>>>>> Graduate Student
>>>>>
Department o
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>
>>>>
CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>
>>> --
>>>
>>> Indrashish Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> --
>>
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer
Engineering
>> University of Florida
>
> CONFIDENTIALITY NOTICE
>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Please find the below details requested:
A) ALL OF
THE TASKTRACKER LOG :
Same what I have provided in the earlier email.
B) THE TASK-LOGS :
SYSLOG :
2013-11-06 16:37:21,331 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=MAP, sessionId=
2013-11-06 16:37:22,245 INFO
org.apache.hadoop.mapred.MapTask: numReduceTasks: 2
2013-11-06
16:37:22,301 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-11-06 16:37:22,832 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 79691776/99614720
2013-11-06 16:37:22,833 INFO
org.apache.hadoop.mapred.MapTask: record buffer =
262144/327680
2013-11-06 16:37:23,157 ERROR
org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException
at
java.io.DataInputStream.readByte(DataInputStream.java:267)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
2013-11-06
16:37:23,157 INFO org.apache.hadoop.mapred.pipes.Application: Aborting
because of java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
2013-11-06
16:37:23,284 WARN org.apache.hadoop.mapred.TaskTracker: Error running
child
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at
org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by:
java.net.SocketException: Broken pipe
at
java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
... 3 more
STDERR :
Command:0received before authentication.
Exiting..
STDOUT :
cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
'0' < /dev/null 1>> /usr/lo
cal/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201311061630_0001_m_000001_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../log
s/userlogs/attempt_201311061630_0001_m_000001_0/stderr]
deviceID:
C)
SPECIFIC TASKATTEMPT'S TASKATTEMTPID THAT IS FAILING :
attempt_201311061630_0001_m_000000_0
attempt_201311061630_0001_m_000001_0
attempt_201311061630_0001_m_000002_0
attempt_201311061630_0001_m_000000_1
attempt_201311061630_0001_m_000001_1
attempt_201311061630_0001_m_000003_0
attempt_201311061630_0001_m_000000_2
attempt_201311061630_0001_m_000001_2
attempt_201311061630_0001_m_000000_3
attempt_201311061630_0001_m_000001_3
Kindly let me know in case of any
more details required.
Thanks again for your time and patience
regarding this.
Regards,
Indrashish.
On Wed, 6 Nov 2013 16:58:53
-0800, Vinod Kumar Vavilapalli wrote:
> Don't see anything in the logs
that you pasted.
> Can you paste the following in say pastebin?
> -
All of the TaskTracker log
> - The task-logs. These are syslog, stderr,
stdout files for a specific TaskAttempt.
> - And specific TaskAttempt's
TaskAttemtpID that is failing.
>
> Thanks,
> +Vinod
>
> On Nov 6,
2013, at 2:50 PM, Basu,Indrashish wrote:
>
>> Can anyone please assist
regarding this ?
>>
>> Thanks in advance
>>
>> Regards,
>>
>>
Indra
>>
>> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>>
>>> Hi Vinod,
>>>
>>> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>>>
>>>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>>>
>>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>>>
>>>
2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>>>
>>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager:
JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>>>
>>>
2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM
with ID: jvm_201311060636_0001_m_1100862588 given task:
attempt_201311060
>>>
>>> 636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>>>
>>> 0636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:09,361 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>>>
>>> 2013-11-06
06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>>>
>>> 2013-11-06
06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,021 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:13,021 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>>>
>>>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>>>
>>> 2013-11-06
06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>>>
>>>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>>>
>>> 2013-11-06
06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu
xx from status: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>>>
>>> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>>>
>>> 2013-11-06 06:40:14,894 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>>>
>>>
2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0
task's
>>>
>>> state:UNASSIGNED
>>>
>>> 2013-11-06 06:40:14,902 INFO
org.apache.hadoop.mapred.TaskTracker: Trying to launch :
attempt_201311060636_0001_m_000002_0 on CPU
>>>
>>> 2013-11-06
06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher,
current free CPU slots : 4 and trying to launch attempt_2013
>>>
>>>
11060636_0001_m_000002_0
>>>
>>> 2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
>>>
>>> 2013-11-06 06:40:14,920
INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for
task: attempt_201311060636_0001_m_000001_3
>>>
>>> 2013-11-06
06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_164532908
>>>
>>>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>>>
>>> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>>>
>>> 36_0001_m_000002_0
>>>
>>> Regards,
>>>
>>> Indrashish
>>>
>>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>>
>>>> It seems like your pipes mapper is exiting before consuming
all the input. Did you check the task-logs on the web UI?
>>>>
>>>>
Thanks,
>>>> +Vinod
>>>>
>>>> On Nov 5, 2013, at 7:25 AM,
Basu,Indrashish wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can anyone kindly
assist on this ?
>>>>>
>>>>> Regards,
>>>>> Indrashish
>>>>>
>>>>> On
Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>>>
>>>>>> Hi
All,
>>>>>
>>>>>> Any update on the below post ?
>>>>> solution as "
The *nopipe* example needs more documentation. It is run with the
InputFormat from
>>>>> x; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">src/test/org/apache/*hadoop*/mapred/*p
>>>>>
>>>>>>
: 5px; width: 100%;">*WordCountInputFormat*.java
>>>>> very specific
input split probably be recoded *to* use libhdfs *too*, but t
>>>>>
blockquote> if you have nfs or something working across the cluster.
"
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;"
>>>>>
>>>>>> ding-left: 5px;
border-left: #1010ff 2px solid; margin-l
>>>>> th: 100%;">could
elaborate a bit about the same as what needs to be done exactly.
>>>>>
>>>>>>> To mention, I am trying to run a sample KMeans algorithm on a
GPU
>>>>> blockquote style="padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">usin
>>>>>
>>>>>> eft: 5px;
width: 100%;">Thanks in advance.
>>>>> g-left: 5px; border-left: #1010ff
2px solid; margin-left: 5px; width: 100%;">Regards, ndrashish.
>>>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;">On Thu, 31 Oct
2013 20:00:10 -0400, Basu,Indrashish wrote: : 100%;">
>>>>> id;
margin-left: 5px; width: 100%;">Hi, I am trying to run a sample Hadoop
GPU source
>>>>> algorithm) M processor and getting the below error. Can
anyone please
>>>>> ckquote> 2px solid; marg
>>>>> width: 100%;">rmr:
cannot remove output: No such file or directory. uote sty
>>>>> eft:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">13/10/31 13:43:12 WARN mapred.
>>>>>
>>>>>> px; border-
>>>>>
2px solid; margin-left: 5px; width: 100%;"> 13/10/31 13:43:12 INFO
mapred.FileInputFormat: Total input paths to eft: 5px; width: 100%;">
>>>>>>
>>>>>>> process : 1
>>>>> 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:13 INFO
mapred.JobClient:
>>>>>
>>>>>> er-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> 13/10/31 13:43:14
>>>>> JobClient: map
0% reduce 0% attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>>>> olid; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
idth: 100%;">
>>>>>>
>>>>>>> at
>>>>> border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;"> org.apach
>>>>>
>>>>>>
style="padding-left: 5px; border-left: #1010ff 2px solid; margin-left:
5px; width: 100%;">
>>>>> 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) ;at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> d;
margin-left: 5px; width: 100%;"> at org.apac
>>>>>
>>>>>> 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;"> Caused
by
>>>>> cketException: Broken pipe at
java.net.SocketOutputStream.socketWrite(Soc
>>>>>
>>>>>> #1010ff 2px
solid; margin-left: 5px; width: 100%;"> at java.net.Socke
>>>>>
.write(SocketOutputStream.java:159) at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
100%;"> at java.io.DataOutputStream.write(DataO
>>>>> ava:107)
>>>>>
">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>>>>
>>>>>> at
>>>>> padding-left: 5px; border-left: #1010ff 2px
solid; margin-left: 5px; width:
100%;">org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>>>>
>>>>>>>
>>>>>>
>>>>>>> at
>>>>> olid; margin-left: 5px;
width: 100%;">
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> ... 3 more
>>>>> ote style="padding-left: 5px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans
>>>>>
te> < /dev/null 1>>
>>>>> 1010ff 2px solid; margin-left: 5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>>> px;
border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
Regards,
>>>>> --
>>>>> Indrashish Basu
>>>>> Graduate Student
>>>>>
Department o
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>
>>>>
CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use
of the individual or entity to which it is addressed and may contain
information that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended
recipient, you are hereby notified that any printing, copying,
dissemination, distribution, disclosure or forwarding of this
communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete
it from your system. Thank You.
>>>
>>> --
>>>
>>> Indrashish Basu
>>> Graduate Student
>>> Department of Electrical and Computer
Engineering
>>> University of Florida
>>
>> --
>>
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer
Engineering
>> University of Florida
>
> CONFIDENTIALITY NOTICE
>
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Don't see anything in the logs that you pasted.
Can you paste the following in say pastebin?
- All of the TaskTracker log
- The task-logs. These are syslog, stderr, stdout files for a specific TaskAttempt.
- And specific TaskAttempt's TaskAttemtpID that is failing.
Thanks,
+Vinod
On Nov 6, 2013, at 2:50 PM, Basu,Indrashish wrote:
>
> Can anyone please assist regarding this ?
>
> Thanks in advance
>
> Regards,
>
> Indra
>
>
>
> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>
>>
>> Hi Vinod,
>>
>> Thanks for your help regarding this. I checked the task logs, this is what it is giving as output.
>>
>>
>> 2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>> 2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>> 2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>> 636_0001_m_000001_3
>> 2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>> 0636_0001_m_000000_3
>> 2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000001_3 0.0%
>> 2013-11-06 06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000000_3 0.0%
>> 2013-11-06 06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000001_3 done; removing files.
>> 2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 3
>> 2013-11-06 06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000000_3 done; removing files.
>> 2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 4
>> 2013-11-06 06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>> 2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0 task's
>> state:UNASSIGNED
>> 2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>> 2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free CPU slots : 4 and trying to launch attempt_2013
>> 11060636_0001_m_000002_0
>> 2013-11-06 06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>> 2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>> 2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_164532908 spawned.
>> 2013-11-06 06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>> 36_0001_m_000002_0
>>
>> Regards,
>>
>> Indrashish
>>
>>
>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>
>> It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>
>> Hi,
>>
>> Can anyone kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>>
>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>> Any update on the below post ?
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>> Thanks in advance.
>> Regards,
>> Indrashish.
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> Regards,
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Don't see anything in the logs that you pasted.
Can you paste the following in say pastebin?
- All of the TaskTracker log
- The task-logs. These are syslog, stderr, stdout files for a specific TaskAttempt.
- And specific TaskAttempt's TaskAttemtpID that is failing.
Thanks,
+Vinod
On Nov 6, 2013, at 2:50 PM, Basu,Indrashish wrote:
>
> Can anyone please assist regarding this ?
>
> Thanks in advance
>
> Regards,
>
> Indra
>
>
>
> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>
>>
>> Hi Vinod,
>>
>> Thanks for your help regarding this. I checked the task logs, this is what it is giving as output.
>>
>>
>> 2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>> 2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>> 2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>> 636_0001_m_000001_3
>> 2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>> 0636_0001_m_000000_3
>> 2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000001_3 0.0%
>> 2013-11-06 06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000000_3 0.0%
>> 2013-11-06 06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000001_3 done; removing files.
>> 2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 3
>> 2013-11-06 06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000000_3 done; removing files.
>> 2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 4
>> 2013-11-06 06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>> 2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0 task's
>> state:UNASSIGNED
>> 2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>> 2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free CPU slots : 4 and trying to launch attempt_2013
>> 11060636_0001_m_000002_0
>> 2013-11-06 06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>> 2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>> 2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_164532908 spawned.
>> 2013-11-06 06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>> 36_0001_m_000002_0
>>
>> Regards,
>>
>> Indrashish
>>
>>
>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>
>> It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>
>> Hi,
>>
>> Can anyone kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>>
>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>> Any update on the below post ?
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>> Thanks in advance.
>> Regards,
>> Indrashish.
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> Regards,
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Don't see anything in the logs that you pasted.
Can you paste the following in say pastebin?
- All of the TaskTracker log
- The task-logs. These are syslog, stderr, stdout files for a specific TaskAttempt.
- And specific TaskAttempt's TaskAttemtpID that is failing.
Thanks,
+Vinod
On Nov 6, 2013, at 2:50 PM, Basu,Indrashish wrote:
>
> Can anyone please assist regarding this ?
>
> Thanks in advance
>
> Regards,
>
> Indra
>
>
>
> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>
>>
>> Hi Vinod,
>>
>> Thanks for your help regarding this. I checked the task logs, this is what it is giving as output.
>>
>>
>> 2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>> 2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>> 2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>> 636_0001_m_000001_3
>> 2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>> 0636_0001_m_000000_3
>> 2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000001_3 0.0%
>> 2013-11-06 06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000000_3 0.0%
>> 2013-11-06 06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000001_3 done; removing files.
>> 2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 3
>> 2013-11-06 06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000000_3 done; removing files.
>> 2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 4
>> 2013-11-06 06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>> 2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0 task's
>> state:UNASSIGNED
>> 2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>> 2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free CPU slots : 4 and trying to launch attempt_2013
>> 11060636_0001_m_000002_0
>> 2013-11-06 06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>> 2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>> 2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_164532908 spawned.
>> 2013-11-06 06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>> 36_0001_m_000002_0
>>
>> Regards,
>>
>> Indrashish
>>
>>
>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>
>> It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>
>> Hi,
>>
>> Can anyone kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>>
>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>> Any update on the below post ?
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>> Thanks in advance.
>> Regards,
>> Indrashish.
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> Regards,
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Don't see anything in the logs that you pasted.
Can you paste the following in say pastebin?
- All of the TaskTracker log
- The task-logs. These are syslog, stderr, stdout files for a specific TaskAttempt.
- And specific TaskAttempt's TaskAttemtpID that is failing.
Thanks,
+Vinod
On Nov 6, 2013, at 2:50 PM, Basu,Indrashish wrote:
>
> Can anyone please assist regarding this ?
>
> Thanks in advance
>
> Regards,
>
> Indra
>
>
>
> On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish wrote:
>
>>
>> Hi Vinod,
>>
>> Thanks for your help regarding this. I checked the task logs, this is what it is giving as output.
>>
>>
>> 2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>> 2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_1100862588 spawned.
>> 2013-11-06 06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>> 2013-11-06 06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_-1960039766 spawned.
>> 2013-11-06 06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>> 636_0001_m_000001_3
>> 2013-11-06 06:40:07,618 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>> 0636_0001_m_000000_3
>> 2013-11-06 06:40:08,013 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,014 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000001_3 0.0%
>> 2013-11-06 06:40:09,735 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201311060636_0001_m_000000_3 0.0%
>> 2013-11-06 06:40:10,018 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000001_3 done; removing files.
>> 2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 3
>> 2013-11-06 06:40:14,025 INFO org.apache.hadoop.mapred.TaskRunner: attempt_201311060636_0001_m_000000_3 done; removing files.
>> 2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot : current free slots : 4
>> 2013-11-06 06:40:14,476 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status: false
>> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker: running on gpu : false
>> 2013-11-06 06:40:14,900 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201311060636_0001_m_000002_0 task's
>> state:UNASSIGNED
>> 2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>> 2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free CPU slots : 4 and trying to launch attempt_2013
>> 11060636_0001_m_000002_0
>> 2013-11-06 06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>> 2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>> 2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201311060636_0001_m_164532908 spawned.
>> 2013-11-06 06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>> 36_0001_m_000002_0
>>
>> Regards,
>>
>> Indrashish
>>
>>
>> On Tue, 5 Nov 2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>>
>> It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>
>> Hi,
>>
>> Can anyone kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>>
>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>> Any update on the below post ?
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>> Thanks in advance.
>> Regards,
>> Indrashish.
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> Regards,
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Can anyone please assist regarding this ?
Thanks in advance
Regards,
Indra
On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
> Hi Vinod,
>
> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>
>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>
>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>
> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>
> 2013-11-06
06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
>
> 2013-11-06
06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>
>
636_0001_m_000001_3
>
> 2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>
>
0636_0001_m_000000_3
>
> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>
> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>
> 2013-11-06 06:40:09,735
INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>
> 2013-11-06 06:40:10,018
INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>
>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>
> 2013-11-06 06:40:14,025
INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>
>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>
> 2013-11-06 06:40:14,476
INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from
status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
>
> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>
> state:UNASSIGNED
>
>
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>
>
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
>
> 11060636_0001_m_000002_0
>
> 2013-11-06
06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>
>
2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker:
Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>
> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>
>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>
> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>
>
36_0001_m_000002_0
>
> Regards,
>
> Indrashish
>
> On Tue, 5 Nov
2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>
>> It seems like
your pipes mapper is exiting before consuming all the input. Did you
check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>> Hi,
>>>
>>>
Can anyone kindly assist on this ?
>>>
>>> Regards,
>>> Indrashish
>>>
>>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi All,
>>>
>>>> Any update on the below post ?
>>>
*WordCountInputFor
>>> ich has a very specific input split probably be
recoded *to* use libhdfs
>>> hat is more C++ example is using local file
reads, it will only work on
>>>
>>>> eft: 5px; width: 100%;">if you
have nfs or something working across
>>> " I would need some more light
on the ab
>>>
>>>> 2px solid; margin-left: 5px; width: 100%;">could
elaborate
>>> the same as what needs to be done exactly. using Hadoop.
in advance.
>>> 0ff 2px solid; margin-left: 5px; width: 100%;">Regards,
On
>>> 013 20:00:10 -0400, Basu,Indrashish wrote:
>>>
>>>> ">Hi,
>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;"> rmr: cannot
r
>>>
>>>> px; border-left: #
>>> solid; margin-left: 5px; width:
100%;"> : 5px; widt
>>> sses may not be found. See JobConf(Class) or
JobConf#setJar(String). 13/10/31 13:43:12 INFO mapred.FileInputFormat:
Total input pat
>>> uote> process : 1
>>> px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">13/10/31 13:43:13 INFO mapr
>>>
>>>> -left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/
>>> 4 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>> id; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
left: 5px; width: 100%;">
>>>>
>>>>> at
>>> g-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> r-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>>> ">
>>>>
>>>>> Caused by: java.net.SocketException: Broken pipe
>>>
le="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) width:
100%;"> at java.io.DataOutputStream.write
>>> ream.java:107) at left:
5px; width: 100%;"> org.apache.hadoop.mapred.pipes.BinaryProtoc
>>>
t(BinaryProtocol.java:333)
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>> g-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
n-left: 5px; width: 100%;"> &n
>>> 3 more attempt_201310
>>>
>>>> t:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
'/app/hadoop/tm
>>>
l/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>>
>>>> '0' < /dev/null 1>>
>>> er-left: #1010ff 2px solid; margin-left:
5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;">2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> Regards,
>>> --
>>> Indrashish Basu
>>> Graduate
Stu
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This
message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the
reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution,
disclosure or forwarding of this communication is strictly prohibited.
If you have received this communication in error, please contact the
sender immediately and delete it from your system. Thank You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of Electrical and
Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Can anyone please assist regarding this ?
Thanks in advance
Regards,
Indra
On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
> Hi Vinod,
>
> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>
>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>
>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>
> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>
> 2013-11-06
06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
>
> 2013-11-06
06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>
>
636_0001_m_000001_3
>
> 2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>
>
0636_0001_m_000000_3
>
> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>
> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>
> 2013-11-06 06:40:09,735
INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>
> 2013-11-06 06:40:10,018
INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>
>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>
> 2013-11-06 06:40:14,025
INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>
>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>
> 2013-11-06 06:40:14,476
INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from
status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
>
> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>
> state:UNASSIGNED
>
>
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>
>
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
>
> 11060636_0001_m_000002_0
>
> 2013-11-06
06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>
>
2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker:
Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>
> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>
>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>
> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>
>
36_0001_m_000002_0
>
> Regards,
>
> Indrashish
>
> On Tue, 5 Nov
2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>
>> It seems like
your pipes mapper is exiting before consuming all the input. Did you
check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>> Hi,
>>>
>>>
Can anyone kindly assist on this ?
>>>
>>> Regards,
>>> Indrashish
>>>
>>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi All,
>>>
>>>> Any update on the below post ?
>>>
*WordCountInputFor
>>> ich has a very specific input split probably be
recoded *to* use libhdfs
>>> hat is more C++ example is using local file
reads, it will only work on
>>>
>>>> eft: 5px; width: 100%;">if you
have nfs or something working across
>>> " I would need some more light
on the ab
>>>
>>>> 2px solid; margin-left: 5px; width: 100%;">could
elaborate
>>> the same as what needs to be done exactly. using Hadoop.
in advance.
>>> 0ff 2px solid; margin-left: 5px; width: 100%;">Regards,
On
>>> 013 20:00:10 -0400, Basu,Indrashish wrote:
>>>
>>>> ">Hi,
>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;"> rmr: cannot
r
>>>
>>>> px; border-left: #
>>> solid; margin-left: 5px; width:
100%;"> : 5px; widt
>>> sses may not be found. See JobConf(Class) or
JobConf#setJar(String). 13/10/31 13:43:12 INFO mapred.FileInputFormat:
Total input pat
>>> uote> process : 1
>>> px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">13/10/31 13:43:13 INFO mapr
>>>
>>>> -left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/
>>> 4 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>> id; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
left: 5px; width: 100%;">
>>>>
>>>>> at
>>> g-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> r-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>>> ">
>>>>
>>>>> Caused by: java.net.SocketException: Broken pipe
>>>
le="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) width:
100%;"> at java.io.DataOutputStream.write
>>> ream.java:107) at left:
5px; width: 100%;"> org.apache.hadoop.mapred.pipes.BinaryProtoc
>>>
t(BinaryProtocol.java:333)
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>> g-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
n-left: 5px; width: 100%;"> &n
>>> 3 more attempt_201310
>>>
>>>> t:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
'/app/hadoop/tm
>>>
l/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>>
>>>> '0' < /dev/null 1>>
>>> er-left: #1010ff 2px solid; margin-left:
5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;">2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> Regards,
>>> --
>>> Indrashish Basu
>>> Graduate
Stu
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This
message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the
reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution,
disclosure or forwarding of this communication is strictly prohibited.
If you have received this communication in error, please contact the
sender immediately and delete it from your system. Thank You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of Electrical and
Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Can anyone please assist regarding this ?
Thanks in advance
Regards,
Indra
On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
> Hi Vinod,
>
> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>
>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>
>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>
> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>
> 2013-11-06
06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
>
> 2013-11-06
06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>
>
636_0001_m_000001_3
>
> 2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>
>
0636_0001_m_000000_3
>
> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>
> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>
> 2013-11-06 06:40:09,735
INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>
> 2013-11-06 06:40:10,018
INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>
>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>
> 2013-11-06 06:40:14,025
INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>
>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>
> 2013-11-06 06:40:14,476
INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from
status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
>
> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>
> state:UNASSIGNED
>
>
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>
>
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
>
> 11060636_0001_m_000002_0
>
> 2013-11-06
06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>
>
2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker:
Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>
> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>
>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>
> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>
>
36_0001_m_000002_0
>
> Regards,
>
> Indrashish
>
> On Tue, 5 Nov
2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>
>> It seems like
your pipes mapper is exiting before consuming all the input. Did you
check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>> Hi,
>>>
>>>
Can anyone kindly assist on this ?
>>>
>>> Regards,
>>> Indrashish
>>>
>>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi All,
>>>
>>>> Any update on the below post ?
>>>
*WordCountInputFor
>>> ich has a very specific input split probably be
recoded *to* use libhdfs
>>> hat is more C++ example is using local file
reads, it will only work on
>>>
>>>> eft: 5px; width: 100%;">if you
have nfs or something working across
>>> " I would need some more light
on the ab
>>>
>>>> 2px solid; margin-left: 5px; width: 100%;">could
elaborate
>>> the same as what needs to be done exactly. using Hadoop.
in advance.
>>> 0ff 2px solid; margin-left: 5px; width: 100%;">Regards,
On
>>> 013 20:00:10 -0400, Basu,Indrashish wrote:
>>>
>>>> ">Hi,
>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;"> rmr: cannot
r
>>>
>>>> px; border-left: #
>>> solid; margin-left: 5px; width:
100%;"> : 5px; widt
>>> sses may not be found. See JobConf(Class) or
JobConf#setJar(String). 13/10/31 13:43:12 INFO mapred.FileInputFormat:
Total input pat
>>> uote> process : 1
>>> px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">13/10/31 13:43:13 INFO mapr
>>>
>>>> -left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/
>>> 4 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>> id; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
left: 5px; width: 100%;">
>>>>
>>>>> at
>>> g-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> r-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>>> ">
>>>>
>>>>> Caused by: java.net.SocketException: Broken pipe
>>>
le="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) width:
100%;"> at java.io.DataOutputStream.write
>>> ream.java:107) at left:
5px; width: 100%;"> org.apache.hadoop.mapred.pipes.BinaryProtoc
>>>
t(BinaryProtocol.java:333)
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>> g-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
n-left: 5px; width: 100%;"> &n
>>> 3 more attempt_201310
>>>
>>>> t:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
'/app/hadoop/tm
>>>
l/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>>
>>>> '0' < /dev/null 1>>
>>> er-left: #1010ff 2px solid; margin-left:
5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;">2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> Regards,
>>> --
>>> Indrashish Basu
>>> Graduate
Stu
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This
message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the
reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution,
disclosure or forwarding of this communication is strictly prohibited.
If you have received this communication in error, please contact the
sender immediately and delete it from your system. Thank You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of Electrical and
Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Can anyone please assist regarding this ?
Thanks in advance
Regards,
Indra
On Wed, 06 Nov 2013 09:50:02 -0500, Basu,Indrashish
wrote:
> Hi Vinod,
>
> Thanks for your help regarding this. I
checked the task logs, this is what it is giving as output.
>
>
2013-11-06 06:40:05,541 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201311060636_0001_m_1100862588
>
>
2013-11-06 06:40:05,553 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_1100862588 spawned.
>
> 2013-11-06
06:40:05,650 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID: jvm_201311060636_0001_m_-1960039766
>
> 2013-11-06
06:40:05,651 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
>
> 2013-11-06
06:40:07,496 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
>
>
636_0001_m_000001_3
>
> 2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
>
>
0636_0001_m_000000_3
>
> 2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
>
> 2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
>
> 2013-11-06 06:40:09,735
INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
>
> 2013-11-06 06:40:10,018
INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
>
> 2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
>
>
2013-11-06 06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 3
>
> 2013-11-06 06:40:14,025
INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
>
>
2013-11-06 06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker:
addCPUFreeSlot : current free slots : 4
>
> 2013-11-06 06:40:14,476
INFO org.apache.hadoop.mapred.TaskTracker: running on gpu xx from
status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
>
> 2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
>
> 2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
>
> 2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
>
> state:UNASSIGNED
>
>
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
>
>
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
>
> 11060636_0001_m_000002_0
>
> 2013-11-06
06:40:14,909 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201311060636_0001_m_000000_3
>
>
2013-11-06 06:40:14,920 INFO org.apache.hadoop.mapred.TaskTracker:
Received KillTaskAction for task: attempt_201311060636_0001_m_000001_3
>
> 2013-11-06 06:40:15,161 INFO org.apache.hadoop.mapred.JvmManager:
In JvmRunner constructed JVM ID: jvm_201311060636_0001_m_164532908
>
>
2013-11-06 06:40:15,162 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_201311060636_0001_m_164532908 spawned.
>
> 2013-11-06
06:40:17,216 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
>
>
36_0001_m_000002_0
>
> Regards,
>
> Indrashish
>
> On Tue, 5 Nov
2013 10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
>
>> It seems like
your pipes mapper is exiting before consuming all the input. Did you
check the task-logs on the web UI?
>>
>> Thanks,
>> +Vinod
>>
>> On
Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>>
>>> Hi,
>>>
>>>
Can anyone kindly assist on this ?
>>>
>>> Regards,
>>> Indrashish
>>>
>>> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>>
>>>> Hi All,
>>>
>>>> Any update on the below post ?
>>>
*WordCountInputFor
>>> ich has a very specific input split probably be
recoded *to* use libhdfs
>>> hat is more C++ example is using local file
reads, it will only work on
>>>
>>>> eft: 5px; width: 100%;">if you
have nfs or something working across
>>> " I would need some more light
on the ab
>>>
>>>> 2px solid; margin-left: 5px; width: 100%;">could
elaborate
>>> the same as what needs to be done exactly. using Hadoop.
in advance.
>>> 0ff 2px solid; margin-left: 5px; width: 100%;">Regards,
On
>>> 013 20:00:10 -0400, Basu,Indrashish wrote:
>>>
>>>> ">Hi,
>>>
eft: #1010ff 2px solid; margin-left: 5px; width: 100%;"> rmr: cannot
r
>>>
>>>> px; border-left: #
>>> solid; margin-left: 5px; width:
100%;"> : 5px; widt
>>> sses may not be found. See JobConf(Class) or
JobConf#setJar(String). 13/10/31 13:43:12 INFO mapred.FileInputFormat:
Total input pat
>>> uote> process : 1
>>> px; border-left: #1010ff 2px
solid; margin-left: 5px; width: 100%;">13/10/31 13:43:13 INFO mapr
>>>
>>>> -left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> 13/
>>> 4 INFO mapred.JobClient: map 0% reduce 0%
attempt_201310311320_0001_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
>>> id; margin-left: 5px;
width: 100%;"> at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
left: 5px; width: 100%;">
>>>>
>>>>> at
>>> g-left: 5px; border-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> r-left:
#1010ff 2px solid; margin-left: 5px; width: 100%;"> at
org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>>> ">
>>>>
>>>>> Caused by: java.net.SocketException: Broken pipe
>>>
le="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) width:
100%;"> at java.io.DataOutputStream.write
>>> ream.java:107) at left:
5px; width: 100%;"> org.apache.hadoop.mapred.pipes.BinaryProtoc
>>>
t(BinaryProtocol.java:333)
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
at
>>> g-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width:
100%;">org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
n-left: 5px; width: 100%;"> &n
>>> 3 more attempt_201310
>>>
>>>> t:
5px; border-left: #1010ff 2px solid; margin-left: 5px; width: 100%;">
'/app/hadoop/tm
>>>
l/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>>
>>>> '0' < /dev/null 1>>
>>> er-left: #1010ff 2px solid; margin-left:
5px; width:
100%;">/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
ote style="padding-left: 5px; border-left: #1010ff 2px solid;
margin-left: 5px; width: 100%;">2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
e="padding-left: 5px; border-left: #1010ff 2px solid; margin-left: 5px;
width: 100%;"> Regards,
>>> --
>>> Indrashish Basu
>>> Graduate
Stu
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This
message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the
reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution,
disclosure or forwarding of this communication is strictly prohibited.
If you have received this communication in error, please contact the
sender immediately and delete it from your system. Thank You.
>
> --
>
> Indrashish Basu
> Graduate Student
> Department of Electrical and
Computer Engineering
> University of Florida
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
2013-11-06 06:40:05,541
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_1100862588
2013-11-06 06:40:05,553 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
2013-11-06 06:40:05,650
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_-1960039766
2013-11-06 06:40:05,651 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
2013-11-06 06:40:07,496
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
636_0001_m_000001_3
2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
0636_0001_m_000000_3
2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
2013-11-06 06:40:09,735 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
2013-11-06 06:40:10,018 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
2013-11-06
06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 3
2013-11-06 06:40:14,025 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
2013-11-06
06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 4
2013-11-06 06:40:14,476 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
state:UNASSIGNED
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
11060636_0001_m_000002_0
2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
2013-11-06 06:40:14,920 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000001_3
2013-11-06 06:40:15,161 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_164532908
2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
2013-11-06 06:40:17,216
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
36_0001_m_000002_0
Regards,
Indrashish
On Tue, 5 Nov 2013
10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
> It seems like your
pipes mapper is exiting before consuming all the input. Did you check
the task-logs on the web UI?
>
> Thanks,
> +Vinod
>
> On Nov 5,
2013, at 7:25 AM, Basu,Indrashish wrote:
>
>> Hi,
>>
>> Can anyone
kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>> On Mon, 04
Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>
>>> Hi All,
>>
>>>
Any update on the below post ?
>> solution as " The *nopipe* example
needs more documentation. It src/test/org/apache/*hadoop
>>
>>> d;
margin-left:5px; width:100%">*WordCountInputF
>> which has a very
specific input split probably be recoded *to* use libhdfs *too*,
>> ore
C++ example is using local file reads, it will only work on a cl
>>
>>>
ft:5px; width:100%">if you have nfs or something working across the
>>
lockquote> I would need some more light on the above exp
>>
>>> x
solid; margin-left:5px; width:100%">could elaborate a bit
>> e as what
needs to be done exactly. using Hadoop. .
>> ff 2px solid;
margin-left:5px; width:100%">Regards, On Thu, 31 Oct 2013
>> 0,
Basu,Indrashish wrote: quote>
>> solid; margin-left:5px; width:100%"> I
am tryi
>> ample Hadoop GPU source code (kmeans algorithm) on an ARM
processor and getting the below er
>> ne please throw some light on this
? px solid; m
>> x; width:100%">13/10/31 13:43:12 WARN mapred.JobClient:
No job jar file set. User h:100%">
>> #1010ff 2px solid;
margin-left:5px; width:100%">classes may not be found. See
JobConf(Class) or
>>
>>> 5px; width:100%"> J
>> (String).
>>
>>>
>>
3:12 INFO mapred.FileInputFormat: Total input paths to ockquote>
>> x
solid; margin-left:5px; width:100%">13/10/31 13:43:13 INFO
mapred.JobClient: Running job: job_201310311320
>>
>>> olid;
margin-left:5px; width:100%"> 13/10/31 13:43:14 INFO mapred.JobClient:
 
>> e 0% 13/10/31 13:43:39 INFO mapred.JobClie
>>
>>> ff 2px
solid; margin-left:5px; width:100%">
>> 310311320_0001_m_000000_0,
Status : FAILED
>> ption
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapR
>> 3) &
>>
>>> lockquote type="cite" style="padding-left:5px; border-left:#1010ff
2px solid; margin-left:5px; width:100%">
>> d; margin-left:5px;
width:100%"> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> "
style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px;
width:100%"> at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStre
>> e(SocketOutputStream.java:113) at
>>
>>> "cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">
>> ; at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
%"> at java.io.BufferedOutputStream.write(BufferedOutputStrea
>>
blockquote> :5px;
width:100%">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
1010ff 2px solid; margin-left:5px; width:100%"> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
pe="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%"> org.apache.hadoop.mapred.pipe
>>
ner.run(PipesMapRunner.java:92) ... 3 more
>> :#1010ff 2px solid;
margin-left:5px; width:100%">
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">'0' < /dev/null 1>>
>> r-left:#1010ff 2px
solid; margin-left:5px; width:100%">
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
idth:100%"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> te>
Regards,
>>> --
>>> Indrashish Basu
>>> Graduate Student
>>>
Department of Electrical and C
>> eering
>> University of Florida
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>
> CONFIDENTIALITY
NOTICE
> NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
2013-11-06 06:40:05,541
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_1100862588
2013-11-06 06:40:05,553 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
2013-11-06 06:40:05,650
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_-1960039766
2013-11-06 06:40:05,651 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
2013-11-06 06:40:07,496
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
636_0001_m_000001_3
2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
0636_0001_m_000000_3
2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
2013-11-06 06:40:09,735 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
2013-11-06 06:40:10,018 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
2013-11-06
06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 3
2013-11-06 06:40:14,025 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
2013-11-06
06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 4
2013-11-06 06:40:14,476 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
state:UNASSIGNED
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
11060636_0001_m_000002_0
2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
2013-11-06 06:40:14,920 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000001_3
2013-11-06 06:40:15,161 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_164532908
2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
2013-11-06 06:40:17,216
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
36_0001_m_000002_0
Regards,
Indrashish
On Tue, 5 Nov 2013
10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
> It seems like your
pipes mapper is exiting before consuming all the input. Did you check
the task-logs on the web UI?
>
> Thanks,
> +Vinod
>
> On Nov 5,
2013, at 7:25 AM, Basu,Indrashish wrote:
>
>> Hi,
>>
>> Can anyone
kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>> On Mon, 04
Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>
>>> Hi All,
>>
>>>
Any update on the below post ?
>> solution as " The *nopipe* example
needs more documentation. It src/test/org/apache/*hadoop
>>
>>> d;
margin-left:5px; width:100%">*WordCountInputF
>> which has a very
specific input split probably be recoded *to* use libhdfs *too*,
>> ore
C++ example is using local file reads, it will only work on a cl
>>
>>>
ft:5px; width:100%">if you have nfs or something working across the
>>
lockquote> I would need some more light on the above exp
>>
>>> x
solid; margin-left:5px; width:100%">could elaborate a bit
>> e as what
needs to be done exactly. using Hadoop. .
>> ff 2px solid;
margin-left:5px; width:100%">Regards, On Thu, 31 Oct 2013
>> 0,
Basu,Indrashish wrote: quote>
>> solid; margin-left:5px; width:100%"> I
am tryi
>> ample Hadoop GPU source code (kmeans algorithm) on an ARM
processor and getting the below er
>> ne please throw some light on this
? px solid; m
>> x; width:100%">13/10/31 13:43:12 WARN mapred.JobClient:
No job jar file set. User h:100%">
>> #1010ff 2px solid;
margin-left:5px; width:100%">classes may not be found. See
JobConf(Class) or
>>
>>> 5px; width:100%"> J
>> (String).
>>
>>>
>>
3:12 INFO mapred.FileInputFormat: Total input paths to ockquote>
>> x
solid; margin-left:5px; width:100%">13/10/31 13:43:13 INFO
mapred.JobClient: Running job: job_201310311320
>>
>>> olid;
margin-left:5px; width:100%"> 13/10/31 13:43:14 INFO mapred.JobClient:
 
>> e 0% 13/10/31 13:43:39 INFO mapred.JobClie
>>
>>> ff 2px
solid; margin-left:5px; width:100%">
>> 310311320_0001_m_000000_0,
Status : FAILED
>> ption
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapR
>> 3) &
>>
>>> lockquote type="cite" style="padding-left:5px; border-left:#1010ff
2px solid; margin-left:5px; width:100%">
>> d; margin-left:5px;
width:100%"> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> "
style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px;
width:100%"> at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStre
>> e(SocketOutputStream.java:113) at
>>
>>> "cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">
>> ; at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
%"> at java.io.BufferedOutputStream.write(BufferedOutputStrea
>>
blockquote> :5px;
width:100%">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
1010ff 2px solid; margin-left:5px; width:100%"> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
pe="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%"> org.apache.hadoop.mapred.pipe
>>
ner.run(PipesMapRunner.java:92) ... 3 more
>> :#1010ff 2px solid;
margin-left:5px; width:100%">
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">'0' < /dev/null 1>>
>> r-left:#1010ff 2px
solid; margin-left:5px; width:100%">
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
idth:100%"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> te>
Regards,
>>> --
>>> Indrashish Basu
>>> Graduate Student
>>>
Department of Electrical and C
>> eering
>> University of Florida
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>
> CONFIDENTIALITY
NOTICE
> NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
2013-11-06 06:40:05,541
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_1100862588
2013-11-06 06:40:05,553 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
2013-11-06 06:40:05,650
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_-1960039766
2013-11-06 06:40:05,651 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
2013-11-06 06:40:07,496
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
636_0001_m_000001_3
2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
0636_0001_m_000000_3
2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
2013-11-06 06:40:09,735 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
2013-11-06 06:40:10,018 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
2013-11-06
06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 3
2013-11-06 06:40:14,025 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
2013-11-06
06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 4
2013-11-06 06:40:14,476 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
state:UNASSIGNED
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
11060636_0001_m_000002_0
2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
2013-11-06 06:40:14,920 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000001_3
2013-11-06 06:40:15,161 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_164532908
2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
2013-11-06 06:40:17,216
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
36_0001_m_000002_0
Regards,
Indrashish
On Tue, 5 Nov 2013
10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
> It seems like your
pipes mapper is exiting before consuming all the input. Did you check
the task-logs on the web UI?
>
> Thanks,
> +Vinod
>
> On Nov 5,
2013, at 7:25 AM, Basu,Indrashish wrote:
>
>> Hi,
>>
>> Can anyone
kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>> On Mon, 04
Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>
>>> Hi All,
>>
>>>
Any update on the below post ?
>> solution as " The *nopipe* example
needs more documentation. It src/test/org/apache/*hadoop
>>
>>> d;
margin-left:5px; width:100%">*WordCountInputF
>> which has a very
specific input split probably be recoded *to* use libhdfs *too*,
>> ore
C++ example is using local file reads, it will only work on a cl
>>
>>>
ft:5px; width:100%">if you have nfs or something working across the
>>
lockquote> I would need some more light on the above exp
>>
>>> x
solid; margin-left:5px; width:100%">could elaborate a bit
>> e as what
needs to be done exactly. using Hadoop. .
>> ff 2px solid;
margin-left:5px; width:100%">Regards, On Thu, 31 Oct 2013
>> 0,
Basu,Indrashish wrote: quote>
>> solid; margin-left:5px; width:100%"> I
am tryi
>> ample Hadoop GPU source code (kmeans algorithm) on an ARM
processor and getting the below er
>> ne please throw some light on this
? px solid; m
>> x; width:100%">13/10/31 13:43:12 WARN mapred.JobClient:
No job jar file set. User h:100%">
>> #1010ff 2px solid;
margin-left:5px; width:100%">classes may not be found. See
JobConf(Class) or
>>
>>> 5px; width:100%"> J
>> (String).
>>
>>>
>>
3:12 INFO mapred.FileInputFormat: Total input paths to ockquote>
>> x
solid; margin-left:5px; width:100%">13/10/31 13:43:13 INFO
mapred.JobClient: Running job: job_201310311320
>>
>>> olid;
margin-left:5px; width:100%"> 13/10/31 13:43:14 INFO mapred.JobClient:
 
>> e 0% 13/10/31 13:43:39 INFO mapred.JobClie
>>
>>> ff 2px
solid; margin-left:5px; width:100%">
>> 310311320_0001_m_000000_0,
Status : FAILED
>> ption
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapR
>> 3) &
>>
>>> lockquote type="cite" style="padding-left:5px; border-left:#1010ff
2px solid; margin-left:5px; width:100%">
>> d; margin-left:5px;
width:100%"> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> "
style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px;
width:100%"> at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStre
>> e(SocketOutputStream.java:113) at
>>
>>> "cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">
>> ; at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
%"> at java.io.BufferedOutputStream.write(BufferedOutputStrea
>>
blockquote> :5px;
width:100%">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
1010ff 2px solid; margin-left:5px; width:100%"> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
pe="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%"> org.apache.hadoop.mapred.pipe
>>
ner.run(PipesMapRunner.java:92) ... 3 more
>> :#1010ff 2px solid;
margin-left:5px; width:100%">
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">'0' < /dev/null 1>>
>> r-left:#1010ff 2px
solid; margin-left:5px; width:100%">
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
idth:100%"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> te>
Regards,
>>> --
>>> Indrashish Basu
>>> Graduate Student
>>>
Department of Electrical and C
>> eering
>> University of Florida
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>
> CONFIDENTIALITY
NOTICE
> NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi Vinod,
Thanks for your help regarding this. I checked the task
logs, this is what it is giving as output.
2013-11-06 06:40:05,541
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_1100862588
2013-11-06 06:40:05,553 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_1100862588 spawned.
2013-11-06 06:40:05,650
INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM
ID: jvm_201311060636_0001_m_-1960039766
2013-11-06 06:40:05,651 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_-1960039766 spawned.
2013-11-06 06:40:07,496
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_1100862588 given task: attempt_201311060
636_0001_m_000001_3
2013-11-06 06:40:07,618 INFO
org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_-1960039766 given task: attempt_20131106
0636_0001_m_000000_3
2013-11-06 06:40:08,013 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:08,014 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:08,015 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:08,015 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:09,361 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000001_3 0.0%
2013-11-06 06:40:09,735 INFO
org.apache.hadoop.mapred.TaskTracker:
attempt_201311060636_0001_m_000000_3 0.0%
2013-11-06 06:40:10,018 INFO
org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201311060636_0001_m_1100862588 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,021 INFO org.apache.hadoop.mapred.JvmManager: JVM
: jvm_201311060636_0001_m_-1960039766 exited. Number of tasks it ran: 0
2013-11-06 06:40:11,442 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,442 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:11,443 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:11,443 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:13,021 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000001_3 done; removing files.
2013-11-06
06:40:13,037 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 3
2013-11-06 06:40:14,025 INFO
org.apache.hadoop.mapred.TaskRunner:
attempt_201311060636_0001_m_000000_3 done; removing files.
2013-11-06
06:40:14,028 INFO org.apache.hadoop.mapred.TaskTracker: addCPUFreeSlot :
current free slots : 4
2013-11-06 06:40:14,476 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from status:
false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,477 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu xx from status: false
2013-11-06 06:40:14,477 INFO
org.apache.hadoop.mapred.TaskTracker: running on gpu xx from task: false
2013-11-06 06:40:14,894 INFO org.apache.hadoop.mapred.TaskTracker:
running on gpu : false
2013-11-06 06:40:14,900 INFO
org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask):
attempt_201311060636_0001_m_000002_0 task's
state:UNASSIGNED
2013-11-06 06:40:14,902 INFO org.apache.hadoop.mapred.TaskTracker:
Trying to launch : attempt_201311060636_0001_m_000002_0 on CPU
2013-11-06 06:40:14,904 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free CPU slots : 4 and trying to launch
attempt_2013
11060636_0001_m_000002_0
2013-11-06 06:40:14,909 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000000_3
2013-11-06 06:40:14,920 INFO
org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task:
attempt_201311060636_0001_m_000001_3
2013-11-06 06:40:15,161 INFO
org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
jvm_201311060636_0001_m_164532908
2013-11-06 06:40:15,162 INFO
org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201311060636_0001_m_164532908 spawned.
2013-11-06 06:40:17,216
INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID:
jvm_201311060636_0001_m_164532908 given task: attempt_2013110606
36_0001_m_000002_0
Regards,
Indrashish
On Tue, 5 Nov 2013
10:09:36 -0800, Vinod Kumar Vavilapalli wrote:
> It seems like your
pipes mapper is exiting before consuming all the input. Did you check
the task-logs on the web UI?
>
> Thanks,
> +Vinod
>
> On Nov 5,
2013, at 7:25 AM, Basu,Indrashish wrote:
>
>> Hi,
>>
>> Can anyone
kindly assist on this ?
>>
>> Regards,
>> Indrashish
>>
>> On Mon, 04
Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>>
>>> Hi All,
>>
>>>
Any update on the below post ?
>> solution as " The *nopipe* example
needs more documentation. It src/test/org/apache/*hadoop
>>
>>> d;
margin-left:5px; width:100%">*WordCountInputF
>> which has a very
specific input split probably be recoded *to* use libhdfs *too*,
>> ore
C++ example is using local file reads, it will only work on a cl
>>
>>>
ft:5px; width:100%">if you have nfs or something working across the
>>
lockquote> I would need some more light on the above exp
>>
>>> x
solid; margin-left:5px; width:100%">could elaborate a bit
>> e as what
needs to be done exactly. using Hadoop. .
>> ff 2px solid;
margin-left:5px; width:100%">Regards, On Thu, 31 Oct 2013
>> 0,
Basu,Indrashish wrote: quote>
>> solid; margin-left:5px; width:100%"> I
am tryi
>> ample Hadoop GPU source code (kmeans algorithm) on an ARM
processor and getting the below er
>> ne please throw some light on this
? px solid; m
>> x; width:100%">13/10/31 13:43:12 WARN mapred.JobClient:
No job jar file set. User h:100%">
>> #1010ff 2px solid;
margin-left:5px; width:100%">classes may not be found. See
JobConf(Class) or
>>
>>> 5px; width:100%"> J
>> (String).
>>
>>>
>>
3:12 INFO mapred.FileInputFormat: Total input paths to ockquote>
>> x
solid; margin-left:5px; width:100%">13/10/31 13:43:13 INFO
mapred.JobClient: Running job: job_201310311320
>>
>>> olid;
margin-left:5px; width:100%"> 13/10/31 13:43:14 INFO mapred.JobClient:
 
>> e 0% 13/10/31 13:43:39 INFO mapred.JobClie
>>
>>> ff 2px
solid; margin-left:5px; width:100%">
>> 310311320_0001_m_000000_0,
Status : FAILED
>> ption
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapR
>> 3) &
>>
>>> lockquote type="cite" style="padding-left:5px; border-left:#1010ff
2px solid; margin-left:5px; width:100%">
>> d; margin-left:5px;
width:100%"> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> "
style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px;
width:100%"> at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStre
>> e(SocketOutputStream.java:113) at
>>
>>> "cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">
>> ; at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
%"> at java.io.BufferedOutputStream.write(BufferedOutputStrea
>>
blockquote> :5px;
width:100%">org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>
1010ff 2px solid; margin-left:5px; width:100%"> at
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>
pe="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%"> org.apache.hadoop.mapred.pipe
>>
ner.run(PipesMapRunner.java:92) ... 3 more
>> :#1010ff 2px solid;
margin-left:5px; width:100%">
'/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid;
margin-left:5px; width:100%">'0' < /dev/null 1>>
>> r-left:#1010ff 2px
solid; margin-left:5px; width:100%">
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
idth:100%"> 2>>
/usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>> te>
Regards,
>>> --
>>> Indrashish Basu
>>> Graduate Student
>>>
Department of Electrical and C
>> eering
>> University of Florida
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>>
>>>
>
> CONFIDENTIALITY
NOTICE
> NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information that is
confidential, privileged and exempt from disclosure under applicable
law. If the reader of this message is not the intended recipient, you
are hereby notified that any printing, copying, dissemination,
distribution, disclosure or forwarding of this communication is strictly
prohibited. If you have received this communication in error, please
contact the sender immediately and delete it from your system. Thank
You.
--
Indrashish Basu
Graduate Student
Department of Electrical
and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
Thanks,
+Vinod
On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>
> Hi,
>
> Can anyone kindly assist on this ?
>
> Regards,
> Indrashish
>
>
> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>>
>> Any update on the below post ?
>>
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>>
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>>
>> Thanks in advance.
>>
>> Regards,
>> Indrashish.
>>
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>>> Hi,
>>>
>>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>>> on an ARM processor and getting the below error. Can anyone please
>>> throw some light on this ?
>>>
>>> rmr: cannot remove output: No such file or directory.
>>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>>> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>>> java.io.IOException: pipe child exception
>>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.net.SocketException: Broken pipe
>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>> ... 3 more
>>>
>>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>> '0' < /dev/null 1>>
>>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
>>> Regards,
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
Thanks,
+Vinod
On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>
> Hi,
>
> Can anyone kindly assist on this ?
>
> Regards,
> Indrashish
>
>
> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>>
>> Any update on the below post ?
>>
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>>
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>>
>> Thanks in advance.
>>
>> Regards,
>> Indrashish.
>>
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>>> Hi,
>>>
>>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>>> on an ARM processor and getting the below error. Can anyone please
>>> throw some light on this ?
>>>
>>> rmr: cannot remove output: No such file or directory.
>>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>>> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>>> java.io.IOException: pipe child exception
>>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.net.SocketException: Broken pipe
>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>> ... 3 more
>>>
>>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>> '0' < /dev/null 1>>
>>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
>>> Regards,
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
Thanks,
+Vinod
On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>
> Hi,
>
> Can anyone kindly assist on this ?
>
> Regards,
> Indrashish
>
>
> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>>
>> Any update on the below post ?
>>
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>>
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>>
>> Thanks in advance.
>>
>> Regards,
>> Indrashish.
>>
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>>> Hi,
>>>
>>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>>> on an ARM processor and getting the below error. Can anyone please
>>> throw some light on this ?
>>>
>>> rmr: cannot remove output: No such file or directory.
>>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>>> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>>> java.io.IOException: pipe child exception
>>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.net.SocketException: Broken pipe
>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>> ... 3 more
>>>
>>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>> '0' < /dev/null 1>>
>>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
>>> Regards,
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It seems like your pipes mapper is exiting before consuming all the input. Did you check the task-logs on the web UI?
Thanks,
+Vinod
On Nov 5, 2013, at 7:25 AM, Basu,Indrashish wrote:
>
> Hi,
>
> Can anyone kindly assist on this ?
>
> Regards,
> Indrashish
>
>
> On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
>> Hi All,
>>
>> Any update on the below post ?
>>
>> I came across some old post regarding the same issue. It explains the
>> solution as " The *nopipe* example needs more documentation. It
>> assumes that it is run with the InputFormat from
>> src/test/org/apache/*hadoop*/mapred/*pipes*/
>> *WordCountInputFormat*.java, which has a very specific input split
>> format. By running with a TextInputFormat, it will send binary bytes
>> as the input split and won't work right. The *nopipe* example should
>> probably be recoded *to* use libhdfs *too*, but that is more
>> complicated *to* get running as a unit test. Also note that since the
>> C++ example is using local file reads, it will only work on a cluster
>> if you have nfs or something working across the cluster. "
>>
>> I would need some more light on the above explanation , so if anyone
>> could elaborate a bit about the same as what needs to be done exactly.
>> To mention, I am trying to run a sample KMeans algorithm on a GPU
>> using Hadoop.
>>
>> Thanks in advance.
>>
>> Regards,
>> Indrashish.
>>
>> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>>> Hi,
>>>
>>> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
>>> on an ARM processor and getting the below error. Can anyone please
>>> throw some light on this ?
>>>
>>> rmr: cannot remove output: No such file or directory.
>>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>>> classes may not be found. See JobConf(Class) or
>>> JobConf#setJar(String).
>>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job: job_201310311320_0001
>>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>>> java.io.IOException: pipe child exception
>>> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.net.SocketException: Broken pipe
>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>>> at
>>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>>> at
>>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>>> ... 3 more
>>>
>>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>>> '0' < /dev/null 1>>
>>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>>
>>> Regards,
>
> --
> Indrashish Basu
> Graduate Student
> Department of Electrical and Computer Engineering
> University of Florida
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone kindly assist on this ?
Regards,
Indrashish
On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
> Hi All,
>
> Any update on the below post ?
>
> I came across some old post regarding the same issue. It explains the
> solution as " The *nopipe* example needs more documentation. It
> assumes that it is run with the InputFormat from
> src/test/org/apache/*hadoop*/mapred/*pipes*/
> *WordCountInputFormat*.java, which has a very specific input split
> format. By running with a TextInputFormat, it will send binary bytes
> as the input split and won't work right. The *nopipe* example should
> probably be recoded *to* use libhdfs *too*, but that is more
> complicated *to* get running as a unit test. Also note that since the
> C++ example is using local file reads, it will only work on a cluster
> if you have nfs or something working across the cluster. "
>
> I would need some more light on the above explanation , so if anyone
> could elaborate a bit about the same as what needs to be done
> exactly.
> To mention, I am trying to run a sample KMeans algorithm on a GPU
> using Hadoop.
>
> Thanks in advance.
>
> Regards,
> Indrashish.
>
> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>>
>> I am trying to run a sample Hadoop GPU source code (kmeans
>> algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>>
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
>> job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at
>> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at
>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at
>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>>
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>
>> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone kindly assist on this ?
Regards,
Indrashish
On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
> Hi All,
>
> Any update on the below post ?
>
> I came across some old post regarding the same issue. It explains the
> solution as " The *nopipe* example needs more documentation. It
> assumes that it is run with the InputFormat from
> src/test/org/apache/*hadoop*/mapred/*pipes*/
> *WordCountInputFormat*.java, which has a very specific input split
> format. By running with a TextInputFormat, it will send binary bytes
> as the input split and won't work right. The *nopipe* example should
> probably be recoded *to* use libhdfs *too*, but that is more
> complicated *to* get running as a unit test. Also note that since the
> C++ example is using local file reads, it will only work on a cluster
> if you have nfs or something working across the cluster. "
>
> I would need some more light on the above explanation , so if anyone
> could elaborate a bit about the same as what needs to be done
> exactly.
> To mention, I am trying to run a sample KMeans algorithm on a GPU
> using Hadoop.
>
> Thanks in advance.
>
> Regards,
> Indrashish.
>
> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>>
>> I am trying to run a sample Hadoop GPU source code (kmeans
>> algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>>
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
>> job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at
>> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at
>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at
>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>>
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>
>> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone kindly assist on this ?
Regards,
Indrashish
On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
> Hi All,
>
> Any update on the below post ?
>
> I came across some old post regarding the same issue. It explains the
> solution as " The *nopipe* example needs more documentation. It
> assumes that it is run with the InputFormat from
> src/test/org/apache/*hadoop*/mapred/*pipes*/
> *WordCountInputFormat*.java, which has a very specific input split
> format. By running with a TextInputFormat, it will send binary bytes
> as the input split and won't work right. The *nopipe* example should
> probably be recoded *to* use libhdfs *too*, but that is more
> complicated *to* get running as a unit test. Also note that since the
> C++ example is using local file reads, it will only work on a cluster
> if you have nfs or something working across the cluster. "
>
> I would need some more light on the above explanation , so if anyone
> could elaborate a bit about the same as what needs to be done
> exactly.
> To mention, I am trying to run a sample KMeans algorithm on a GPU
> using Hadoop.
>
> Thanks in advance.
>
> Regards,
> Indrashish.
>
> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>>
>> I am trying to run a sample Hadoop GPU source code (kmeans
>> algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>>
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
>> job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at
>> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at
>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at
>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>>
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>
>> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi,
Can anyone kindly assist on this ?
Regards,
Indrashish
On Mon, 04 Nov 2013 10:23:23 -0500, Basu,Indrashish wrote:
> Hi All,
>
> Any update on the below post ?
>
> I came across some old post regarding the same issue. It explains the
> solution as " The *nopipe* example needs more documentation. It
> assumes that it is run with the InputFormat from
> src/test/org/apache/*hadoop*/mapred/*pipes*/
> *WordCountInputFormat*.java, which has a very specific input split
> format. By running with a TextInputFormat, it will send binary bytes
> as the input split and won't work right. The *nopipe* example should
> probably be recoded *to* use libhdfs *too*, but that is more
> complicated *to* get running as a unit test. Also note that since the
> C++ example is using local file reads, it will only work on a cluster
> if you have nfs or something working across the cluster. "
>
> I would need some more light on the above explanation , so if anyone
> could elaborate a bit about the same as what needs to be done
> exactly.
> To mention, I am trying to run a sample KMeans algorithm on a GPU
> using Hadoop.
>
> Thanks in advance.
>
> Regards,
> Indrashish.
>
> On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
>> Hi,
>>
>> I am trying to run a sample Hadoop GPU source code (kmeans
>> algorithm)
>> on an ARM processor and getting the below error. Can anyone please
>> throw some light on this ?
>>
>> rmr: cannot remove output: No such file or directory.
>> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
>> job_201310311320_0001
>> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
>> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
>> attempt_201310311320_0001_m_000000_0, Status : FAILED
>> java.io.IOException: pipe child exception
>> at
>> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
>> at
>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>> at
>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
>> at
>>
>> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
>> at
>>
>> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
>> ... 3 more
>>
>> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>>
>> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
>> '0' < /dev/null 1>>
>>
>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
>> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>>
>> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi All,
Any update on the below post ?
I came across some old post regarding the same issue. It explains the
solution as " The *nopipe* example needs more documentation. It assumes
that it is run with the InputFormat from
src/test/org/apache/*hadoop*/mapred/*pipes*/
*WordCountInputFormat*.java, which has a very specific input split
format. By running with a TextInputFormat, it will send binary bytes as
the input split and won't work right. The *nopipe* example should
probably be recoded *to* use libhdfs *too*, but that is more complicated
*to* get running as a unit test. Also note that since the C++ example is
using local file reads, it will only work on a cluster if you have nfs
or something working across the cluster. "
I would need some more light on the above explanation , so if anyone
could elaborate a bit about the same as what needs to be done exactly.
To mention, I am trying to run a sample KMeans algorithm on a GPU using
Hadoop.
Thanks in advance.
Regards,
Indrashish.
On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
> Hi,
>
> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
> on an ARM processor and getting the below error. Can anyone please
> throw some light on this ?
>
> rmr: cannot remove output: No such file or directory.
> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
> job_201310311320_0001
> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
> attempt_201310311320_0001_m_000000_0, Status : FAILED
> java.io.IOException: pipe child exception
> at
> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
> at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
> ... 3 more
>
> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>
> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
> '0' < /dev/null 1>>
>
> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>
> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi All,
Any update on the below post ?
I came across some old post regarding the same issue. It explains the
solution as " The *nopipe* example needs more documentation. It assumes
that it is run with the InputFormat from
src/test/org/apache/*hadoop*/mapred/*pipes*/
*WordCountInputFormat*.java, which has a very specific input split
format. By running with a TextInputFormat, it will send binary bytes as
the input split and won't work right. The *nopipe* example should
probably be recoded *to* use libhdfs *too*, but that is more complicated
*to* get running as a unit test. Also note that since the C++ example is
using local file reads, it will only work on a cluster if you have nfs
or something working across the cluster. "
I would need some more light on the above explanation , so if anyone
could elaborate a bit about the same as what needs to be done exactly.
To mention, I am trying to run a sample KMeans algorithm on a GPU using
Hadoop.
Thanks in advance.
Regards,
Indrashish.
On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
> Hi,
>
> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
> on an ARM processor and getting the below error. Can anyone please
> throw some light on this ?
>
> rmr: cannot remove output: No such file or directory.
> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
> job_201310311320_0001
> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
> attempt_201310311320_0001_m_000000_0, Status : FAILED
> java.io.IOException: pipe child exception
> at
> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
> at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
> ... 3 more
>
> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>
> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
> '0' < /dev/null 1>>
>
> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>
> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi All,
Any update on the below post ?
I came across some old post regarding the same issue. It explains the
solution as " The *nopipe* example needs more documentation. It assumes
that it is run with the InputFormat from
src/test/org/apache/*hadoop*/mapred/*pipes*/
*WordCountInputFormat*.java, which has a very specific input split
format. By running with a TextInputFormat, it will send binary bytes as
the input split and won't work right. The *nopipe* example should
probably be recoded *to* use libhdfs *too*, but that is more complicated
*to* get running as a unit test. Also note that since the C++ example is
using local file reads, it will only work on a cluster if you have nfs
or something working across the cluster. "
I would need some more light on the above explanation , so if anyone
could elaborate a bit about the same as what needs to be done exactly.
To mention, I am trying to run a sample KMeans algorithm on a GPU using
Hadoop.
Thanks in advance.
Regards,
Indrashish.
On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
> Hi,
>
> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
> on an ARM processor and getting the below error. Can anyone please
> throw some light on this ?
>
> rmr: cannot remove output: No such file or directory.
> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
> job_201310311320_0001
> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
> attempt_201310311320_0001_m_000000_0, Status : FAILED
> java.io.IOException: pipe child exception
> at
> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
> at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
> ... 3 more
>
> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>
> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
> '0' < /dev/null 1>>
>
> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>
> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida
Re: Error while running Hadoop Source Code
Posted by "Basu,Indrashish" <in...@ufl.edu>.
Hi All,
Any update on the below post ?
I came across some old post regarding the same issue. It explains the
solution as " The *nopipe* example needs more documentation. It assumes
that it is run with the InputFormat from
src/test/org/apache/*hadoop*/mapred/*pipes*/
*WordCountInputFormat*.java, which has a very specific input split
format. By running with a TextInputFormat, it will send binary bytes as
the input split and won't work right. The *nopipe* example should
probably be recoded *to* use libhdfs *too*, but that is more complicated
*to* get running as a unit test. Also note that since the C++ example is
using local file reads, it will only work on a cluster if you have nfs
or something working across the cluster. "
I would need some more light on the above explanation , so if anyone
could elaborate a bit about the same as what needs to be done exactly.
To mention, I am trying to run a sample KMeans algorithm on a GPU using
Hadoop.
Thanks in advance.
Regards,
Indrashish.
On Thu, 31 Oct 2013 20:00:10 -0400, Basu,Indrashish wrote:
> Hi,
>
> I am trying to run a sample Hadoop GPU source code (kmeans algorithm)
> on an ARM processor and getting the below error. Can anyone please
> throw some light on this ?
>
> rmr: cannot remove output: No such file or directory.
> 13/10/31 13:43:12 WARN mapred.JobClient: No job jar file set. User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 13/10/31 13:43:12 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 13/10/31 13:43:13 INFO mapred.JobClient: Running job:
> job_201310311320_0001
> 13/10/31 13:43:14 INFO mapred.JobClient: map 0% reduce 0%
> 13/10/31 13:43:39 INFO mapred.JobClient: Task Id :
> attempt_201310311320_0001_m_000000_0, Status : FAILED
> java.io.IOException: pipe child exception
> at
> org.apache.hadoop.mapred.pipes.Application.abort(Application.java:191)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:103)
> at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
> at
>
> org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
> at
>
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:92)
> ... 3 more
>
> attempt_201310311320_0001_m_000000_0: cmd: [bash, -c, exec
>
> '/app/hadoop/tmp/mapred/local/taskTracker/archive/10.227.56.195bin/cpu-kmeans2D/cpu-kmeans2D'
> '0' < /dev/null 1>>
>
> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/attempt_201310311320_0001_m_000000_0/stdout
> 2>> /usr/local/hadoop/hadoop-gpu-0.20.1/bin/../logs/userlogs/
>
> Regards,
--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida