You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Niels Basjes <Ni...@basjes.nl> on 2013/04/24 22:45:46 UTC
Building pig and piggybank from CDH 4.2.0 sources?
Hi,
[ I know this question is probably CDH specific, yet I'm hoping one of you
may be able to point me in the right direction. ]
I want to make a small change to the piggybank for pig 0.10 that is in CDH
4.2.0 and recreate an RPM from that for easier deployment.
So I downloaded
http://archive.cloudera.com/cdh4/cdh/4/pig-0.10.0-cdh4.2.0.tar.gz and tried
to simply build pig using ant.
When I do that on CentOS 6.4 x86_64 it starts downloading and compiling and
then it fails with:
*compile-sources:*
* [javac] /home/niels/tmp/pig-0.10.0-cdh4.2.0/build.xml:575: warning:
'includeantruntime' was not set, defaulting to build.sysclasspath=last; set
to false for repeatable builds*
* [javac] Compiling 788 source files to
/home/niels/tmp/pig-0.10.0-cdh4.2.0/build/classes*
* [javac]
/home/niels/tmp/pig-0.10.0-cdh4.2.0/src/org/apache/pig/data/Tuple.java:23:
cannot find symbol*
* [javac] symbol : class WritableComparable*
* [javac] location: package org.apache.hadoop.io*
* [javac] import org.apache.hadoop.io.WritableComparable;*
* [javac] ^*
* [javac]
/home/niels/tmp/pig-0.10.0-cdh4.2.0/src/org/apache/pig/data/Tuple.java:47:
cannot find symbol*
* [javac] symbol: class WritableComparable*
* [javac] public interface Tuple extends WritableComparable,
Serializable, Iterable<Object> {*
* [javac] ^*
Apparently it is unable to find the most basic Hadoop dependencies
This is strange because during the download phase I see:
*[ivy:resolve] found org.apache.hadoop#hadoop-annotations;2.0.0-cdh4.2.0
in cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-auth;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-common;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-hdfs;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-core;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-yarn-server-tests;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-app;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-common;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-yarn-api;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-yarn-client;2.0.0-cdh4.2.0
in cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-yarn-common;2.0.0-cdh4.2.0
in cloudera-releases*
*[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server;2.0.0-cdh4.2.0
in cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-yarn-server-web-proxy;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-yarn-server-common;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-yarn-server-nodemanager;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.0.0-cdh4.2.0 in
cloudera-releases*
*[ivy:resolve] found
org.apache.hadoop#hadoop-mapreduce-client-hs;2.0.0-cdh4.2.0 in
cloudera-releases*
I've also tried to just build the piggybank in the contrib folder and when
I do that I get similar (dependency) errors:
* [javac]
/home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:23:
cannot find symbol*
* [javac] symbol : class Accumulator*
* [javac] location: package org.apache.pig*
* [javac] import org.apache.pig.Accumulator;*
* [javac] ^*
* [javac]
/home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:24:
cannot find symbol*
* [javac] symbol : class Algebraic*
* [javac] location: package org.apache.pig*
* [javac] import org.apache.pig.Algebraic;*
* [javac] ^*
* [javac]
/home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:25:
cannot find symbol*
* [javac] symbol : class EvalFunc*
* [javac] location: package org.apache.pig*
* [javac] import org.apache.pig.EvalFunc;*
* [javac] ^*
What should I do to fix this?
Is there a 'better' manual on building pig/piggybank than what is described
in the RELEASE_NOTES.txt ?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Building pig and piggybank from CDH 4.2.0 sources?
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
On Fri, Apr 26, 2013 at 1:20 AM, Cheolsoo Park <pi...@gmail.com> wrote:
> >> Perhaps this should be part of a "Developer / Build Howto" ?
>
> You probably want to ask it to CDH user group. I don't think Apache
> projects document about specific distros.
>
After your first email I kicked the cloudera sources and went for the plain
pig sources instead.
Regarding the other error, I can't reproduce it with Apache Pig. I am using
> Ant 1.8.2.
>
Hmm, I should try this with 1.8.2 then. Perhaps there is a bug in ant 1.9.0
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Building pig and piggybank from CDH 4.2.0 sources?
Posted by Cheolsoo Park <pi...@gmail.com>.
Hi Niels,
>> Perhaps this should be part of a "Developer / Build Howto" ?
You probably want to ask it to CDH user group. I don't think Apache
projects document about specific distros.
Regarding the other error, I can't reproduce it with Apache Pig. I am using
Ant 1.8.2.
Thanks,
Cheolsoo
On Thu, Apr 25, 2013 at 2:01 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi Cheolsoo,
>
> On Wed, Apr 24, 2013 at 11:53 PM, Cheolsoo Park <piaozhexiu@gmail.com
> >wrote:
>
> > Please try:
> > ant clean jar-all piggybank -Dhadoopversion=23
> >
>
> Thanks, this really helped.
> Perhaps this should be part of a "Developer / Build Howto" ?
>
>
> I downloaded the current trunk of pig (via github) and ran into a
> completely different hurdle:
> With hadoopversion=23 this target is fired
> <target name="include-meta" if="isHadoop23">
>
> And even though I installed ant 1.9.0 (the latest) it causes an NPE in ant
> itself.
>
> *BUILD FAILED
> *
> */home/nbasjes/workspace/pig/build.xml:658: The following error occurred
> while executing this line:*
> */home/nbasjes/workspace/pig/build.xml:667: java.lang.NullPointerException*
> * at org.apache.tools.ant.taskdefs.Zip.zipFile(Zip.java:1827)*
> * at org.apache.tools.ant.taskdefs.Jar.zipFile(Jar.java:702)*
> * at org.apache.tools.ant.taskdefs.Zip.zipFile(Zip.java:1856)*
> *...*
>
> Has anyone seen this too? Is this a bug in ant or am I doing something
> wrong?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Re: Building pig and piggybank from CDH 4.2.0 sources?
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi Cheolsoo,
On Wed, Apr 24, 2013 at 11:53 PM, Cheolsoo Park <pi...@gmail.com>wrote:
> Please try:
> ant clean jar-all piggybank -Dhadoopversion=23
>
Thanks, this really helped.
Perhaps this should be part of a "Developer / Build Howto" ?
I downloaded the current trunk of pig (via github) and ran into a
completely different hurdle:
With hadoopversion=23 this target is fired
<target name="include-meta" if="isHadoop23">
And even though I installed ant 1.9.0 (the latest) it causes an NPE in ant
itself.
*BUILD FAILED
*
*/home/nbasjes/workspace/pig/build.xml:658: The following error occurred
while executing this line:*
*/home/nbasjes/workspace/pig/build.xml:667: java.lang.NullPointerException*
* at org.apache.tools.ant.taskdefs.Zip.zipFile(Zip.java:1827)*
* at org.apache.tools.ant.taskdefs.Jar.zipFile(Jar.java:702)*
* at org.apache.tools.ant.taskdefs.Zip.zipFile(Zip.java:1856)*
*...*
Has anyone seen this too? Is this a bug in ant or am I doing something
wrong?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Building pig and piggybank from CDH 4.2.0 sources?
Posted by Cheolsoo Park <pi...@gmail.com>.
Hi,
Please try:
ant clean jar-all piggybank -Dhadoopversion=23
Thanks,
Cheolsoo
On Wed, Apr 24, 2013 at 1:45 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> [ I know this question is probably CDH specific, yet I'm hoping one of you
> may be able to point me in the right direction. ]
>
> I want to make a small change to the piggybank for pig 0.10 that is in CDH
> 4.2.0 and recreate an RPM from that for easier deployment.
>
> So I downloaded
> http://archive.cloudera.com/cdh4/cdh/4/pig-0.10.0-cdh4.2.0.tar.gz and
> tried
> to simply build pig using ant.
>
> When I do that on CentOS 6.4 x86_64 it starts downloading and compiling and
> then it fails with:
>
> *compile-sources:*
> * [javac] /home/niels/tmp/pig-0.10.0-cdh4.2.0/build.xml:575: warning:
> 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set
> to false for repeatable builds*
> * [javac] Compiling 788 source files to
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/build/classes*
> * [javac]
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/src/org/apache/pig/data/Tuple.java:23:
> cannot find symbol*
> * [javac] symbol : class WritableComparable*
> * [javac] location: package org.apache.hadoop.io*
> * [javac] import org.apache.hadoop.io.WritableComparable;*
> * [javac] ^*
> * [javac]
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/src/org/apache/pig/data/Tuple.java:47:
> cannot find symbol*
> * [javac] symbol: class WritableComparable*
> * [javac] public interface Tuple extends WritableComparable,
> Serializable, Iterable<Object> {*
> * [javac] ^*
>
> Apparently it is unable to find the most basic Hadoop dependencies
>
> This is strange because during the download phase I see:
> *[ivy:resolve] found org.apache.hadoop#hadoop-annotations;2.0.0-cdh4.2.0
> in cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-auth;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-common;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-hdfs;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-core;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-yarn-server-tests;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-app;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-common;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-yarn-api;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-yarn-client;2.0.0-cdh4.2.0
> in cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-yarn-common;2.0.0-cdh4.2.0
> in cloudera-releases*
> *[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server;2.0.0-cdh4.2.0
> in cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-yarn-server-web-proxy;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-yarn-server-common;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-yarn-server-nodemanager;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.0.0-cdh4.2.0 in
> cloudera-releases*
> *[ivy:resolve] found
> org.apache.hadoop#hadoop-mapreduce-client-hs;2.0.0-cdh4.2.0 in
> cloudera-releases*
>
> I've also tried to just build the piggybank in the contrib folder and when
> I do that I get similar (dependency) errors:
>
> * [javac]
>
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:23:
> cannot find symbol*
> * [javac] symbol : class Accumulator*
> * [javac] location: package org.apache.pig*
> * [javac] import org.apache.pig.Accumulator;*
> * [javac] ^*
> * [javac]
>
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:24:
> cannot find symbol*
> * [javac] symbol : class Algebraic*
> * [javac] location: package org.apache.pig*
> * [javac] import org.apache.pig.Algebraic;*
> * [javac] ^*
> * [javac]
>
> /home/niels/tmp/pig-0.10.0-cdh4.2.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:25:
> cannot find symbol*
> * [javac] symbol : class EvalFunc*
> * [javac] location: package org.apache.pig*
> * [javac] import org.apache.pig.EvalFunc;*
> * [javac] ^*
>
> What should I do to fix this?
> Is there a 'better' manual on building pig/piggybank than what is described
> in the RELEASE_NOTES.txt ?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>