You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Adarsh Sharma <ad...@orkash.com> on 2011/03/31 06:11:12 UTC
How to apply Patch
Dear all,
Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
I attached the patch.
Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set Configurations in site.xml files
Thanks & best Regards,
Adarsh Sharma
Re: How to apply Patch
Posted by Adarsh Sharma <ad...@orkash.com>.
Thanks Steve , U helped me to clear my doubts several times.
I explain U What my Problem is :
I am trying to run *wordcount-nopipe.cc* program in
*/home/hadoop/project/hadoop-0.20.2/src/examples/pipes/impl* directory.
I am able to run a simple wordcount.cpp program in Hadoop Cluster but
whebn I am going to run this program, ifaced below exception :
*bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -input gutenberg -output
gutenberg-out1101 -program bin/wordcount-nopipe2*
11/03/31 14:59:07 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/03/31 14:59:07 INFO mapred.FileInputFormat: Total input paths to
process : 3
11/03/31 14:59:08 INFO mapred.JobClient: Running job: job_201103310903_0007
11/03/31 14:59:09 INFO mapred.JobClient: map 0% reduce 0%
11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
attempt_201103310903_0007_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
at
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
... 3 more
attempt_201103310903_0007_m_000000_0: Hadoop Pipes Exception: failed to
open at wordcount-nopipe2.cc:86 in
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/03/31 14:59:18 INFO mapred.JobClient: Task Id :
attempt_201103310903_0007_m_000001_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
at
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
... 3 more
After some R&D , i find the below links quite useful :
http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html
http://stackoverflow.com/questions/4395140/eofexception-thrown-by-a-hadoop-pipes-program
But don't know how to resolve this. I think my program try to open the
file as file://gutenberg but it requires as hdfs://.....
Here is the contents of my Makefile :
CC = g++
HADOOP_INSTALL =/home/hadoop/project/hadoop-0.20.2
PLATFORM = Linux-amd64-64
CPPFLAGS = -m64
-I/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/include
-I/usr/local/cuda/include
wordcount-nopipe2 : wordcount-nopipe2.cc
$(CC) $(CPPFLAGS) $< -Wall
-L/home/hadoop/project/hadoop-0.20.2/c++/Linux-amd64-64/lib
-L/usr/local/cuda/lib64 -lhadooppipes \
-lhadooputils -lpthread -g -O2 -o $@
Would it be a bug in hadoop-0.20.2 and if not Please guide me how to
debug it.
Thanks & best Regards,
Adarsh Sharma
Steve Loughran wrote:
> On 31/03/11 07:37, Adarsh Sharma wrote:
>> Thanks a lot for such deep explanation :
>>
>> I have done it now, but it doesn't help me in my original problem for
>> which I'm doing this.
>>
>> Please if you have some idea comment on it. I attached the problem.
>>
>
> Sadly. Matt's deep explanation is what you need, low-level that it is
>
> -patches are designed to be applied to source, so you need the apache
> source tree, not any binary installations.
>
> -you need to be sure that the source version you have matches that the
> patch is designed to be applied against, unless you want to get into
> the problem of understanding the source enough to fix inconsistencies.
>
> -you need to rebuild hadoop afterwards.
>
> Because Apache code is open source, patches and the like are all bits
> of source. This is not windows where the source is secret and all os
> updates are bits of binary code. The view is that if you want to apply
> patches, then yes, you do have to play at the source level.
>
> The good news is once you can do that for one patch, you can apply
> others, and you will be in a position to find and fix bugs yourself.
>
> -steve
Re: How to apply Patch
Posted by Steve Loughran <st...@apache.org>.
On 31/03/11 07:37, Adarsh Sharma wrote:
> Thanks a lot for such deep explanation :
>
> I have done it now, but it doesn't help me in my original problem for
> which I'm doing this.
>
> Please if you have some idea comment on it. I attached the problem.
>
Sadly. Matt's deep explanation is what you need, low-level that it is
-patches are designed to be applied to source, so you need the apache
source tree, not any binary installations.
-you need to be sure that the source version you have matches that the
patch is designed to be applied against, unless you want to get into the
problem of understanding the source enough to fix inconsistencies.
-you need to rebuild hadoop afterwards.
Because Apache code is open source, patches and the like are all bits of
source. This is not windows where the source is secret and all os
updates are bits of binary code. The view is that if you want to apply
patches, then yes, you do have to play at the source level.
The good news is once you can do that for one patch, you can apply
others, and you will be in a position to find and fix bugs yourself.
-steve
Re: How to apply Patch
Posted by Adarsh Sharma <ad...@orkash.com>.
Thanks a lot for such deep explanation :
I have done it now, but it doesn't help me in my original problem for
which I'm doing this.
Please if you have some idea comment on it. I attached the problem.
Thanks & best Regards,
Adarsh Sharma
Matthew Foley wrote:
> Hi Adarsh,
> see if the information at http://wiki.apache.org/hadoop/HowToContribute is helpful to you. I'll walk you thru the typical process, but first a couple questions:
>
> Did you get a tar file for the whole source tree of Hadoop, or only the binary distribution? To apply patches you must get the source tree (usually from the Apache svn server), apply the patch, then re-build.
>
> Also, if you're just starting out, you might consider using a newer version like 0.22 instead of 0.20.
>
> And finally, is the patch you are looking at intended to work with v0.20.2? A patch targeted for 0.22 is only maybe 50% likely to work as intended with a v0.20 version.
>
> So, here's a sketch of how to apply a patch (to v0.20.2) and rebuild:
>
> Per the instructions at http://wiki.apache.org/hadoop/HowToContribute, create a directory for your hadoop projects, and from within that directory do:
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ hadoop-0.20.2
>
> In the same projects directory create a "patches" subdirectory (as a sibling to the "hadoop-0.20.2" subdirectory) and save the patch into it (perhaps from your browser while viewing the Jira)
>
> Now cd into the HADOOP_HOME directory:
> cd hadoop-0.20.2
> Observe that it has subdirectories like build/, conf/, ivy/, and src/.
>
> Now apply the patch:
> patch -p 0 < ../patches/fix-test-pipes.patch
>
> Let's take a look at that last line:
> patch -p 0 < ../patches/fix-test-pipes.patch
> ^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> | | |_where you stored the patch file
> | |
> | |_the patch command is funny, you locate yourself at the root of
> | the source tree, then pipe the patch file contents into the "patch"
> | command. The target source directory name is not an argument.
> |
> |_this "-p 0" argument is generally applicable for patches generated the way we like to do patches in the hadoop dev community. Read the unix man page for "patch", and http://wiki.apache.org/hadoop/HowToContribute if you're interested.
>
> Now, re-build hadoop:
> ant veryclean compile bin-package
>
> Of course, that assumes you have installed ant and ivy. Each of those utilities have home pages that are very informative, along with the info on the above HowToContribute page. You can also look at the docs at http://hadoop.apache.org/common/docs/current/
>
> You should now have all the pieces necessary to install and run the patched version of Hadoop. (The jars are in the "build" subdirectory in hadoop-0.20.2) In my experience it is always easier to get a vanilla system installed and running, and then overlay that installation with the new jars, rather than try to install from your build directory, but that's just my personal preference.
>
> If you want to work in v21 or higher, you'll have to deal with three hadoop subdirectories instead of one, and use the "ant -Dresolvers=internal mvn-install" invocation to get them to build the patch right across components. But in v20 the above should work fine.
>
> If you are going to work with Hadoop long term, you'll want an IDE like Eclipse. So web pages like http://wiki.apache.org/hadoop/EclipseEnvironment may be helpful.
>
> Hope this helps,
> --Matt
>
>
Re: How to apply Patch
Posted by Matthew Foley <ma...@yahoo-inc.com>.
Hi Adarsh,
see if the information at http://wiki.apache.org/hadoop/HowToContribute is helpful to you. I'll walk you thru the typical process, but first a couple questions:
Did you get a tar file for the whole source tree of Hadoop, or only the binary distribution? To apply patches you must get the source tree (usually from the Apache svn server), apply the patch, then re-build.
Also, if you're just starting out, you might consider using a newer version like 0.22 instead of 0.20.
And finally, is the patch you are looking at intended to work with v0.20.2? A patch targeted for 0.22 is only maybe 50% likely to work as intended with a v0.20 version.
So, here's a sketch of how to apply a patch (to v0.20.2) and rebuild:
Per the instructions at http://wiki.apache.org/hadoop/HowToContribute, create a directory for your hadoop projects, and from within that directory do:
svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ hadoop-0.20.2
In the same projects directory create a "patches" subdirectory (as a sibling to the "hadoop-0.20.2" subdirectory) and save the patch into it (perhaps from your browser while viewing the Jira)
Now cd into the HADOOP_HOME directory:
cd hadoop-0.20.2
Observe that it has subdirectories like build/, conf/, ivy/, and src/.
Now apply the patch:
patch -p 0 < ../patches/fix-test-pipes.patch
Let's take a look at that last line:
patch -p 0 < ../patches/fix-test-pipes.patch
^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| | |_where you stored the patch file
| |
| |_the patch command is funny, you locate yourself at the root of
| the source tree, then pipe the patch file contents into the "patch"
| command. The target source directory name is not an argument.
|
|_this "-p 0" argument is generally applicable for patches generated the way we like to do patches in the hadoop dev community. Read the unix man page for "patch", and http://wiki.apache.org/hadoop/HowToContribute if you're interested.
Now, re-build hadoop:
ant veryclean compile bin-package
Of course, that assumes you have installed ant and ivy. Each of those utilities have home pages that are very informative, along with the info on the above HowToContribute page. You can also look at the docs at http://hadoop.apache.org/common/docs/current/
You should now have all the pieces necessary to install and run the patched version of Hadoop. (The jars are in the "build" subdirectory in hadoop-0.20.2) In my experience it is always easier to get a vanilla system installed and running, and then overlay that installation with the new jars, rather than try to install from your build directory, but that's just my personal preference.
If you want to work in v21 or higher, you'll have to deal with three hadoop subdirectories instead of one, and use the "ant -Dresolvers=internal mvn-install" invocation to get them to build the patch right across components. But in v20 the above should work fine.
If you are going to work with Hadoop long term, you'll want an IDE like Eclipse. So web pages like http://wiki.apache.org/hadoop/EclipseEnvironment may be helpful.
Hope this helps,
--Matt
On Mar 30, 2011, at 9:24 PM, Adarsh Sharma wrote:
Thanks Harsh,
I am trying the patch command but below error exists :
[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#
Best regards,
Adarsh
Harsh J wrote:
There is a utility available for Unix called 'patch'. You can use that
with a suitable -p(num) argument (man patch, for more info).
On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma <ad...@orkash.com>> wrote:
Dear all,
Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
I attached the patch.
Please find the attachment. I just follow below steps for Hadoop :
1. Download Hadoop-0.20.2.tar.gz
2. Extract the file.
3. Set Configurations in site.xml files
Thanks & best Regards,
Adarsh Sharma
Re: How to apply Patch
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
Adarsh,
Your command should be :
patch -p0 < fix-test-pipes.patch
See http://wiki.apache.org/hadoop/HowToContribute for details on how to contribute.
Thanks
Amareshwari
On 3/31/11 9:54 AM, "Adarsh Sharma" <ad...@orkash.com> wrote:
Thanks Harsh,
I am trying the patch command but below error exists :
[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#
Best regards,
Adarsh
Harsh J wrote:
> There is a utility available for Unix called 'patch'. You can use that
> with a suitable -p(num) argument (man patch, for more info).
>
> On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma <ad...@orkash.com> wrote:
>
>> Dear all,
>>
>> Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
>>
>> I attached the patch.
>>
>> Please find the attachment. I just follow below steps for Hadoop :
>> 1. Download Hadoop-0.20.2.tar.gz
>> 2. Extract the file.
>> 3. Set Configurations in site.xml files
>>
>> Thanks & best Regards,
>> Adarsh Sharma
>>
>>
>
>
>
>
Re: How to apply Patch
Posted by Adarsh Sharma <ad...@orkash.com>.
Thanks Harsh,
I am trying the patch command but below error exists :
[root@ws-test project]# patch hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -R hadoop-0.20.2 fix-test-pipes.patch
patch: **** File hadoop-0.20.2 is not a regular file -- can't patch
[root@ws-test project]# patch -p hadoop-0.20.2 fix-test-pipes.patch
patch: **** strip count hadoop-0.20.2 is not a number
[root@ws-test project]#
Best regards,
Adarsh
Harsh J wrote:
> There is a utility available for Unix called 'patch'. You can use that
> with a suitable -p(num) argument (man patch, for more info).
>
> On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma <ad...@orkash.com> wrote:
>
>> Dear all,
>>
>> Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
>>
>> I attached the patch.
>>
>> Please find the attachment. I just follow below steps for Hadoop :
>> 1. Download Hadoop-0.20.2.tar.gz
>> 2. Extract the file.
>> 3. Set Configurations in site.xml files
>>
>> Thanks & best Regards,
>> Adarsh Sharma
>>
>>
>
>
>
>
Re: How to apply Patch
Posted by Harsh J <qw...@gmail.com>.
There is a utility available for Unix called 'patch'. You can use that
with a suitable -p(num) argument (man patch, for more info).
On Thu, Mar 31, 2011 at 9:41 AM, Adarsh Sharma <ad...@orkash.com> wrote:
> Dear all,
>
> Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
>
> I attached the patch.
>
> Please find the attachment. I just follow below steps for Hadoop :
> 1. Download Hadoop-0.20.2.tar.gz
> 2. Extract the file.
> 3. Set Configurations in site.xml files
>
> Thanks & best Regards,
> Adarsh Sharma
>
--
Harsh J
http://harshj.com
Re: How to apply Patch
Posted by Adarsh Sharma <ad...@orkash.com>.
Sorry, Just check the attachment now.
Adarsh Sharma wrote:
> Dear all,
>
> Can Someone Please tell me how to apply a patch on hadoop-0.20.2 package.
>
> I attached the patch.
>
> Please find the attachment. I just follow below steps for Hadoop :
> 1. Download Hadoop-0.20.2.tar.gz
> 2. Extract the file.
> 3. Set Configurations in site.xml files
>
> Thanks & best Regards,
> Adarsh Sharma
>