You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Madhvaraj Shetty <ma...@gmail.com> on 2016/11/15 11:36:23 UTC
Hadoop source code modification.
Hello,
I am working in the area of hadoop, especially i want to improve hadoop
security.
So i need procedure for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part
of the code.
Is it compulsory to compile the whole hadoop, can i compile/debug a part of
it. Any steps for that.
Thank you.
Re: Hadoop source code modification.
Posted by Suraj Nayak <sn...@gmail.com>.
Follow https://wiki.apache.org/hadoop/EclipseEnvironment.
You need to add generated sources of the project to include generated Proto
classes.
This blog
<http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/>
is old, but will give you some idea on how to import the generated java
classes.
*The generated sources (e.g. *Proto.java files that are generated using
protoc) might not be directly linked and can show up as errors. To fix
them, select the project and configure the build path to include the java
files under target/generated-sources and target/generated-test-sources. For
inclusion pattern, select “**/*.java”.*
*- Source
: http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
<http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/>*
On Fri, Nov 18, 2016 at 8:19 AM, Madhvaraj Shetty <madresearch2015@gmail.com
> wrote:
> Hi,
>
> After producing eclipse files from maven, i successfully imported hadoop
> project to eclipse.(hadoop-common-project)
>
> But it has few errors in eclipse:
>
> AvroRecord cannot be resolved to a type
> EchoRequestProto cannot be resolved to a type
> TestProtobufRpcProto cannot be resolved
> TestProtobufRpcProto cannot be resolved to a type
>
> I think it is related protocol buffer, but i can't figure out where it is.
>
> Thanks
>
>
> On Tue, Nov 15, 2016 at 6:27 PM, Mallanagouda Patil <
> mallanagouda.c.patil@gmail.com> wrote:
>
> > Hi,
> > Some time back I have written an article on how to setup development
> > environment for Hadoop and also how to debug.
> > http://www.codeproject.com/Articles/1067129/Debugging-
> > Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux
> >
> > You can reach me for any issues.
> > Thanks
> > Mallan
> >
> > On Nov 15, 2016 5:49 PM, "Brahma Reddy Battula" <
> > brahmareddy.battula@huawei.com> wrote:
> >
> >> (Keeping user-mailing list in loop.)
> >>
> >> You can compile corresponding module which you modified.
> >>
> >> Please refer "Where to run Maven from?" from the following.
> >>
> >> https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
> >>
> >>
> >>
> >> Regards
> >> Brahma Reddy Battula
> >>
> >> -----Original Message-----
> >> From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
> >> Sent: 15 November 2016 19:36
> >> To: general@hadoop.apache.org
> >> Subject: Hadoop source code modification.
> >>
> >> Hello,
> >>
> >> I am working in the area of hadoop, especially i want to improve hadoop
> >> security.
> >>
> >> So i need procedure for modifying hadoop source code.
> >> I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
> >> able to compile hadoop from the source code, but i need to modify the
> >> part of the code.
> >>
> >> Is it compulsory to compile the whole hadoop, can i compile/debug a part
> >> of it. Any steps for that.
> >>
> >>
> >> Thank you.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: user-help@hadoop.apache.org
> >>
> >
>
>
> --
> --
>
>
>
>
>
> *Madhvaraj M Shetty,Research Scholar,Department of Computer
> Science,Mangalore University, Mangalagangothri, Mangalore, Karnataka -
> 574199.*
> *E-mail: Madhvarajj@gmail.com <dp...@gmail.com>*
> *Phone: 9741449181*
>
--
Thanks
Suraj Nayak M
Re: Hadoop source code modification.
Posted by zhangjc <61...@qq.com>.
Hi,
I have the same error "Type AvroRecord cannot be resolved to a type”, and the solution is:
(1) Download the latest avro-tools from http://mirrors.cnnic.cn/apache/avro/avro-1.7.7/java/avro-tools-1.7.7.jar <http://mirrors.cnnic.cn/apache/avro/avro-1.7.7/java/avro-tools-1.7.7.jar>.
(2) Change directory to hadoop-2.7.3-src/hadoop-common-project/hadoop-common/src/test/avro, and perform command:
$ java -jar <avro-tools-jar-path>/avro-tools-1.7.7.jar compile schema avroRecord.avsc ../java/
Note that this command will generate the corresponding java source file with avroRecord.avsc.
(3)Refresh the hadoop-common project in Eclipse.
My hadoop is 2.7.3, it works with avro-tools-1.7.7.jar, but not works with latest avro-tools-1.8.1.jar.
> 在 2016年11月19日，上午12:19，Madhvaraj Shetty <ma...@gmail.com> 写道：
>
> Hi,
>
> After producing eclipse files from maven, i successfully imported hadoop project to eclipse.(hadoop-common-project)
>
> But it has few errors in eclipse:
>
> AvroRecord cannot be resolved to a type
> EchoRequestProto cannot be resolved to a type
> TestProtobufRpcProto cannot be resolved
> TestProtobufRpcProto cannot be resolved to a type
>
> I think it is related protocol buffer, but i can't figure out where it is.
>
> Thanks
>
>
> On Tue, Nov 15, 2016 at 6:27 PM, Mallanagouda Patil <mallanagouda.c.patil@gmail.com <ma...@gmail.com>> wrote:
> Hi,
> Some time back I have written an article on how to setup development environment for Hadoop and also how to debug.
> http://www.codeproject.com/Articles/1067129/Debugging-Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux <http://www.codeproject.com/Articles/1067129/Debugging-Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux>
> You can reach me for any issues.
> Thanks
> Mallan
>
>
> On Nov 15, 2016 5:49 PM, "Brahma Reddy Battula" <brahmareddy.battula@huawei.com <ma...@huawei.com>> wrote:
> (Keeping user-mailing list in loop.)
>
> You can compile corresponding module which you modified.
>
> Please refer "Where to run Maven from?" from the following.
>
> https://github.com/apache/hadoop/blob/trunk/BUILDING.txt <https://github.com/apache/hadoop/blob/trunk/BUILDING.txt>
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com <ma...@gmail.com>]
> Sent: 15 November 2016 19:36
> To: general@hadoop.apache.org <ma...@hadoop.apache.org>
> Subject: Hadoop source code modification.
>
> Hello,
>
> I am working in the area of hadoop, especially i want to improve hadoop security.
>
> So i need procedure for modifying hadoop source code.
> I have visited this <https://wiki.apache.org/hadoop/HowToContribute <https://wiki.apache.org/hadoop/HowToContribute>>and
> able to compile hadoop from the source code, but i need to modify the part of the code.
>
> Is it compulsory to compile the whole hadoop, can i compile/debug a part of it. Any steps for that.
>
>
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org <ma...@hadoop.apache.org>
> For additional commands, e-mail: user-help@hadoop.apache.org <ma...@hadoop.apache.org>
>
>
>
> --
> --
> Madhvaraj M Shetty,
> Research Scholar,
> Department of Computer Science,
> Mangalore University, Mangalagangothri,
> Mangalore, Karnataka - 574199.
> E-mail: Madhvarajj@gmail.com <ma...@gmail.com>
> Phone: 9741449181
Re: Hadoop source code modification.
Posted by Madhvaraj Shetty <ma...@gmail.com>.
Hi,
After producing eclipse files from maven, i successfully imported hadoop
project to eclipse.(hadoop-common-project)
But it has few errors in eclipse:
AvroRecord cannot be resolved to a type
EchoRequestProto cannot be resolved to a type
TestProtobufRpcProto cannot be resolved
TestProtobufRpcProto cannot be resolved to a type
I think it is related protocol buffer, but i can't figure out where it is.
Thanks
On Tue, Nov 15, 2016 at 6:27 PM, Mallanagouda Patil <
mallanagouda.c.patil@gmail.com> wrote:
> Hi,
> Some time back I have written an article on how to setup development
> environment for Hadoop and also how to debug.
> http://www.codeproject.com/Articles/1067129/Debugging-
> Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux
>
> You can reach me for any issues.
> Thanks
> Mallan
>
> On Nov 15, 2016 5:49 PM, "Brahma Reddy Battula" <
> brahmareddy.battula@huawei.com> wrote:
>
>> (Keeping user-mailing list in loop.)
>>
>> You can compile corresponding module which you modified.
>>
>> Please refer "Where to run Maven from?" from the following.
>>
>> https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
>>
>>
>>
>> Regards
>> Brahma Reddy Battula
>>
>> -----Original Message-----
>> From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
>> Sent: 15 November 2016 19:36
>> To: general@hadoop.apache.org
>> Subject: Hadoop source code modification.
>>
>> Hello,
>>
>> I am working in the area of hadoop, especially i want to improve hadoop
>> security.
>>
>> So i need procedure for modifying hadoop source code.
>> I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
>> able to compile hadoop from the source code, but i need to modify the
>> part of the code.
>>
>> Is it compulsory to compile the whole hadoop, can i compile/debug a part
>> of it. Any steps for that.
>>
>>
>> Thank you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>
--
--
*Madhvaraj M Shetty,Research Scholar,Department of Computer
Science,Mangalore University, Mangalagangothri, Mangalore, Karnataka -
574199.*
*E-mail: Madhvarajj@gmail.com <dp...@gmail.com>*
*Phone: 9741449181*
Re: Hadoop source code modification.
Posted by Madhvaraj Shetty <ma...@gmail.com>.
Hi,
After producing eclipse files from maven, i successfully imported hadoop
project to eclipse.(hadoop-common-project)
But it has few errors in eclipse:
AvroRecord cannot be resolved to a type
EchoRequestProto cannot be resolved to a type
TestProtobufRpcProto cannot be resolved
TestProtobufRpcProto cannot be resolved to a type
I think it is related protocol buffer, but i can't figure out where it is.
Thanks
On Tue, Nov 15, 2016 at 6:27 PM, Mallanagouda Patil <
mallanagouda.c.patil@gmail.com> wrote:
> Hi,
> Some time back I have written an article on how to setup development
> environment for Hadoop and also how to debug.
> http://www.codeproject.com/Articles/1067129/Debugging-
> Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux
>
> You can reach me for any issues.
> Thanks
> Mallan
>
> On Nov 15, 2016 5:49 PM, "Brahma Reddy Battula" <
> brahmareddy.battula@huawei.com> wrote:
>
>> (Keeping user-mailing list in loop.)
>>
>> You can compile corresponding module which you modified.
>>
>> Please refer "Where to run Maven from?" from the following.
>>
>> https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
>>
>>
>>
>> Regards
>> Brahma Reddy Battula
>>
>> -----Original Message-----
>> From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
>> Sent: 15 November 2016 19:36
>> To: general@hadoop.apache.org
>> Subject: Hadoop source code modification.
>>
>> Hello,
>>
>> I am working in the area of hadoop, especially i want to improve hadoop
>> security.
>>
>> So i need procedure for modifying hadoop source code.
>> I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
>> able to compile hadoop from the source code, but i need to modify the
>> part of the code.
>>
>> Is it compulsory to compile the whole hadoop, can i compile/debug a part
>> of it. Any steps for that.
>>
>>
>> Thank you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>
--
--
*Madhvaraj M Shetty,Research Scholar,Department of Computer
Science,Mangalore University, Mangalagangothri, Mangalore, Karnataka -
574199.*
*E-mail: Madhvarajj@gmail.com <dp...@gmail.com>*
*Phone: 9741449181*
RE: Hadoop source code modification.
Posted by Mallanagouda Patil <ma...@gmail.com>.
Hi,
Some time back I have written an article on how to setup development
environment for Hadoop and also how to debug.
http://www.codeproject.com/Articles/1067129/Debugging-Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux
You can reach me for any issues.
Thanks
Mallan
On Nov 15, 2016 5:49 PM, "Brahma Reddy Battula" <
brahmareddy.battula@huawei.com> wrote:
> (Keeping user-mailing list in loop.)
>
> You can compile corresponding module which you modified.
>
> Please refer "Where to run Maven from?" from the following.
>
> https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
>
>
>
> Regards
> Brahma Reddy Battula
>
> -----Original Message-----
> From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
> Sent: 15 November 2016 19:36
> To: general@hadoop.apache.org
> Subject: Hadoop source code modification.
>
> Hello,
>
> I am working in the area of hadoop, especially i want to improve hadoop
> security.
>
> So i need procedure for modifying hadoop source code.
> I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
> able to compile hadoop from the source code, but i need to modify the part
> of the code.
>
> Is it compulsory to compile the whole hadoop, can i compile/debug a part
> of it. Any steps for that.
>
>
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
RE: Hadoop source code modification.
Posted by Brahma Reddy Battula <br...@huawei.com>.
Following links might be useful fro you.
https://wiki.apache.org/hadoop/EclipseEnvironment
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
https://www.quora.com/What-are-the-best-ways-to-learn-about-Hadoop-source
Regards
Brahma Reddy Battula
From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
Sent: 15 November 2016 20:27
To: Brahma Reddy Battula
Subject: Re: Hadoop source code modification.
Thank you so much sir
Any links for understanding hadoop code and classes?
Any quick way to debug OR any setup like eclipse.. to do these work?
On Tue, Nov 15, 2016 at 5:49 PM, Brahma Reddy Battula <br...@huawei.com>> wrote:
(Keeping user-mailing list in loop.)
You can compile corresponding module which you modified.
Please refer "Where to run Maven from?" from the following.
https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
Regards
Brahma Reddy Battula
-----Original Message-----
From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com<ma...@gmail.com>]
Sent: 15 November 2016 19:36
To: general@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop source code modification.
Hello,
I am working in the area of hadoop, especially i want to improve hadoop security.
So i need procedure for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part of the code.
Is it compulsory to compile the whole hadoop, can i compile/debug a part of it. Any steps for that.
Thank you.
RE: Hadoop source code modification.
Posted by Brahma Reddy Battula <br...@huawei.com>.
(Keeping user-mailing list in loop.)
You can compile corresponding module which you modified.
Please refer "Where to run Maven from?" from the following.
https://github.com/apache/hadoop/blob/trunk/BUILDING.txt
Regards
Brahma Reddy Battula
-----Original Message-----
From: Madhvaraj Shetty [mailto:madresearch2015@gmail.com]
Sent: 15 November 2016 19:36
To: general@hadoop.apache.org
Subject: Hadoop source code modification.
Hello,
I am working in the area of hadoop, especially i want to improve hadoop security.
So i need procedure for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part of the code.
Is it compulsory to compile the whole hadoop, can i compile/debug a part of it. Any steps for that.
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
RE: Hadoop source code modification.
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Madhvaraj,
Its not required to compile everytime, as you have followed the steps in <https://wiki.apache.org/hadoop/HowToContribute>.
You can go to specific project and run " mvn clean install"
Regards,
+ Naga
________________________________________
From: Madhvaraj Shetty [madresearch2015@gmail.com]
Sent: Tuesday, November 15, 2016 17:06
To: general@hadoop.apache.org
Subject: Hadoop source code modification.
Hello,
I am working in the area of hadoop, especially i want to improve hadoop
security.
So i need procedure for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part
of the code.
Is it compulsory to compile the whole hadoop, can i compile/debug a part of
it. Any steps for that.
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@hadoop.apache.org
For additional commands, e-mail: general-help@hadoop.apache.org