You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Turner Kunkel <th...@gmail.com> on 2009/08/18 19:37:26 UTC
Pig 0.3.0 and Hadoop 0.20.0
Hello,
I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
connects to the HDFS.
I tried rebuilding and applying patch PIG660.
I have a script that I run that exports these environment tables correctly:
JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
Pig still connects to file:/// instead of HDFS.
I rebuilt the Pig package so that there are no patches applied at the
moment, as if I had just unpacked PIG-0.3.0 for the first time.
So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the start?
Thanks!
--
-Turner Kunkel
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by George Pang <p0...@gmail.com>.
Yes, I use PIG-660_4.patch. And I check out from svn.
I will try again. Thank you.
George
2009/9/4 Turner Kunkel <th...@gmail.com>
> Did you patch using PIG-660_4.patch?
>
> It looks like it's not looking in the correct spot for Java classes.
> Did you download everything from the svn Pig package correctly?
>
> I also ran into the issue of running ant with certain permissions. I
> dedicated full rights to user 'hadoop' on all of my Hadoop folders
> (pig, hadoop, etc.) and ant only succeeded when I was logged in as
> user 'hadoop' and ran ant.
>
> -Turner
>
> On Fri, Sep 4, 2009 at 3:15 PM, George Pang<p0...@gmail.com> wrote:
> > Hi Turner,
> >
> > I use Alex's tutorial on building Pig with Ant but get errors like this,
> >
> > ........
> > er.java:36: package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.LogFactory;
> > [javac] ^
> > [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
> > package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.Log;
> > [javac] ^
> > [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
> > package org.apache.commons.logging does not exist
> > [javac] import org.apache.commons.logging.LogFactory;
> > [javac] ^
> > [javac] Note: Some input files use unchecked or unsafe operations.
> > [javac] Note: Recompile with -Xlint:unchecked for details.
> > [javac] 100 errors
> >
> > BUILD FAILED
> > /home/hadoop20/pig/build.xml:246: The following error occurred while
> > executing this line:
> > /home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
> > output for details.
> >
> > Where did I get wrong? Or where to find the error output file?
> >
> > Thank you.
> >
> > George
> >
> >
> >
> > 2009/8/20 Turner Kunkel <th...@gmail.com>
> >
> >> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
> >> download with the patch provided on Alex's how-to site.
> >> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
> >> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
> >> ant.
> >> I set my classpath to include the hadoop conf folder and pig.jar
> >> (along with the other environment settings).
> >> I execute pig with this command:
> >>
> >> java -Dfs.default.name=hdfs://localhost:9000
> >> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
> >> and mapred settings of course)
> >>
> >> So I was doing multiple things differently after coming from Hadoop
> >> 18, but this seems to work.
> >>
> >> -Turner
> >>
> >> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >
> >> wrote:
> >> >
> >> > Turner,
> >> > Before you went through those steps, did you use a vanilla hadoop 20
> >> > jar or did you get one from the zebra ticket?
> >> > I am still at a loss as to why it wasn't working for you before.
> >> > -D
> >> >
> >> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<th...@gmail.com>
> >> wrote:
> >> > > It's finally working!
> >> > >
> >> > > Using your environment table specs and
> >> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>
> <http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
> >> I got it to work.
> >> > >
> >> > > Thanks much, this helps me a lot. Have a nice day.
> >> > >
> >> > > -Turner
> >> > >
> >> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <
> dvryaboy@cloudera.com
> >> >wrote:
> >> > >
> >> > >> Tumer,
> >> > >> That error means you dropped pig.jar from your classpath.
> >> > >>
> >> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<thkunkel@gmail.com
> >
> >> wrote:
> >> > >> > Ok, I know I'm really close now, I ran into this when running Pig
> on
> >> 18.
> >> > >> > "Could not find the main class: org.apache.pig.Main"
> >> > >> >
> >> > >> > I forgot how to resolve this.
> >> > >> >
> >> > >> > Thanks a lot so far, by the way :)
> >> > >> >
> >> > >> > -Turner
> >> > >> >
> >> > >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
> >> sms@yahoo-inc.com
> >> > >> >wrote:
> >> > >> >
> >> > >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop
> the
> >> hod
> >> > >> >> parameters and thing should start working
> >> > >> >>
> >> > >> >> -----Original Message-----
> >> > >> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
> >> > >> >> Sent: Wednesday, August 19, 2009 11:52 AM
> >> > >> >> To: pig-user@hadoop.apache.org
> >> > >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> >> > >> >>
> >> > >> >> If you are not using Hadoop on Demand (hod), you will need to
> >> comment
> >> > >> >> out the hod properties in pig's conf dir.
> >> > >> >>
> >> > >> >> -----Original Message-----
> >> > >> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> > >> >> Sent: Wednesday, August 19, 2009 11:50 AM
> >> > >> >> To: pig-user@hadoop.apache.org
> >> > >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> > >> >>
> >> > >> >> Getting closer...
> >> > >> >> Actually got and error this time:
> >> > >> >> ERROR 6012: Unable to run command: hod allocate ....
> >> > >> >>
> >> > >> >> -Turner
> >> > >> >>
> >> > >> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> >> > >> >> <sm...@yahoo-inc.com>wrote:
> >> > >> >>
> >> > >> >> > You need a special hadoop20.jar to build the pig sources.
> Check
> >> out
> >> > >> >> PIG-660
> >> > >> >> >
> >> > >> >> > Santhosh
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > -----Original Message-----
> >> > >> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> > >> >> > Sent: Wed 8/19/2009 9:53 AM
> >> > >> >> > To: pig-user@hadoop.apache.org
> >> > >> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> > >> >> >
> >> > >> >> > Hm, still nothing. Maybe I have to build it differently? I
> will
> >> play
> >> > >> >> > around with the environment settings, but any more input is
> >> > >> >> appreciated.
> >> > >> >> >
> >> > >> >> > -Turner
> >> > >> >> >
> >> > >> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> >> > >> >> <dvryaboy@cloudera.com
> >> > >> >> > >wrote:
> >> > >> >> >
> >> > >> >> > > Don't point it at mapred-site.xml directly, but the conf
> >> directory,
> >> > >> >> > > instead.
> >> > >> >> > > (it is smart enough to pick up the new config files when
> it's
> >> > >> >> running on
> >> > >> >> > > 20).
> >> > >> >> > >
> >> > >> >> > > I tried setting the path directly to mapred-site.xml like
> you
> >> have
> >> > >> >> it,
> >> > >> >> > > and indeed, that doesn't work. Changing to the directory
> should
> >> fix
> >> > >> >> > > your issue.
> >> > >> >> > >
> >> > >> >> > > -D
> >> > >> >> > >
> >> > >> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<
> >> thkunkel@gmail.com>
> >> > >> >> > wrote:
> >> > >> >> > > > I am running off of unpacked tar.gz files of Hadoop and
> Pig.
> >> Was
> >> > >> >> > working
> >> > >> >> > > > with 0.18.0 and had it working fine.
> >> > >> >> > > > With 0.20.0 I imagine you're correct with changing
> >> PIG_CLASSPATH
> >> > >> >> since
> >> > >> >> > > Pig
> >> > >> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> >> > >> >> > > > My environment looks like this now:
> >> > >> >> > > >
> >> > >> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> >> > >> >> conf/mapred-site.xml
> >> > >> >> > > > PIGDIR=/usr/local/pig-0.3.0
> >> > >> >> > > > HADOOP_HOME=/usr/local/hadoop
> >> > >> >> > > > HADOOPDIR=/usr/local/hadoop/conf
> >> > >> >> > > > PIG_HADOOP_VERSION=20
> >> > >> >> > > >
> >> > >> >> > > > Pig still isn't connecting correctly.
> >> > >> >> > > >
> >> > >> >> > > > -Turner
> >> > >> >> > > >
> >> > >> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> >> > >> >> <dvryaboy@cloudera.com
> >> > >> >> > > >wrote:
> >> > >> >> > > >
> >> > >> >> > > >> Nipun and Turner,
> >> > >> >> > > >> What are you setting PIG_CLASSPATH to?
> >> > >> >> > > >>
> >> > >> >> > > >> My environment works if I set it to
> >> > >> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> >> > >> >> > > >>
> >> > >> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both
> >> lead to
> >> > >> >> > > >> breakage -- I haven't quite decided if that's a bug or
> not.)
> >> > >> >> > > >>
> >> > >> >> > > >> For completeness, a full set of environment variables
> that
> >> make
> >> > >> >> it
> >> > >> >> > > >> work for me. I am running the Cloudera CDH2 testing
> package
> >> for
> >> > >> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a
> >> patched
> >> > >> >> > version
> >> > >> >> > > >> of Pig from trunk.
> >> > >> >> > > >>
> >> > >> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> > >> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> > >> >> > > >> PIG_HADOOP_VERSION=20
> >> > >> >> > > >> PIG_CONF_DIR=./conf/
> >> > >> >> > > >>
> >> > >> >> > > >>
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> >> > >> >> /pig.jar
> >> > >> >> > > >>
> >> > >> >> > > >>
> >> > >> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> >> > >> >> Kunkel<th...@gmail.com>
> >> > >> >> > > wrote:
> >> > >> >> > > >> > Hello,
> >> > >> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working)
> and
> >> Pig
> >> > >> >> no
> >> > >> >> > > longer
> >> > >> >> > > >> > connects to the HDFS.
> >> > >> >> > > >> > I tried rebuilding and applying patch PIG660.
> >> > >> >> > > >> > I have a script that I run that exports these
> environment
> >> > >> >> tables
> >> > >> >> > > >> correctly:
> >> > >> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > >> >> > > >> > Pig still connects to file:/// instead of HDFS.
> >> > >> >> > > >> > I rebuilt the Pig package so that there are no patches
> >> applied
> >> > >> >> at
> >> > >> >> > the
> >> > >> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the
> first
> >> time.
> >> > >> >> > > >> >
> >> > >> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop
> >> 0.20.0
> >> > >> >> from
> >> > >> >> > the
> >> > >> >> > > >> start?
> >> > >> >> > > >> >
> >> > >> >> > > >> > Thanks!
> >> > >> >> > > >> >
> >> > >> >> > > >> > --
> >> > >> >> > > >> >
> >> > >> >> > > >> > -Turner Kunkel
> >> > >> >> > > >> >
> >> > >> >> > > >>
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >>
> >> > >> >
> >> > >>
> >> > >
> >>
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
Did you patch using PIG-660_4.patch?
It looks like it's not looking in the correct spot for Java classes.
Did you download everything from the svn Pig package correctly?
I also ran into the issue of running ant with certain permissions. I
dedicated full rights to user 'hadoop' on all of my Hadoop folders
(pig, hadoop, etc.) and ant only succeeded when I was logged in as
user 'hadoop' and ran ant.
-Turner
On Fri, Sep 4, 2009 at 3:15 PM, George Pang<p0...@gmail.com> wrote:
> Hi Turner,
>
> I use Alex's tutorial on building Pig with Ant but get errors like this,
>
> ........
> er.java:36: package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.LogFactory;
> [javac] ^
> [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
> package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.Log;
> [javac] ^
> [javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
> package org.apache.commons.logging does not exist
> [javac] import org.apache.commons.logging.LogFactory;
> [javac] ^
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 100 errors
>
> BUILD FAILED
> /home/hadoop20/pig/build.xml:246: The following error occurred while
> executing this line:
> /home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
> output for details.
>
> Where did I get wrong? Or where to find the error output file?
>
> Thank you.
>
> George
>
>
>
> 2009/8/20 Turner Kunkel <th...@gmail.com>
>
>> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
>> download with the patch provided on Alex's how-to site.
>> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
>> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
>> ant.
>> I set my classpath to include the hadoop conf folder and pig.jar
>> (along with the other environment settings).
>> I execute pig with this command:
>>
>> java -Dfs.default.name=hdfs://localhost:9000
>> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
>> and mapred settings of course)
>>
>> So I was doing multiple things differently after coming from Hadoop
>> 18, but this seems to work.
>>
>> -Turner
>>
>> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <dv...@cloudera.com>
>> wrote:
>> >
>> > Turner,
>> > Before you went through those steps, did you use a vanilla hadoop 20
>> > jar or did you get one from the zebra ticket?
>> > I am still at a loss as to why it wasn't working for you before.
>> > -D
>> >
>> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<th...@gmail.com>
>> wrote:
>> > > It's finally working!
>> > >
>> > > Using your environment table specs and
>> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
>> I got it to work.
>> > >
>> > > Thanks much, this helps me a lot. Have a nice day.
>> > >
>> > > -Turner
>> > >
>> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
>> >wrote:
>> > >
>> > >> Tumer,
>> > >> That error means you dropped pig.jar from your classpath.
>> > >>
>> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com>
>> wrote:
>> > >> > Ok, I know I'm really close now, I ran into this when running Pig on
>> 18.
>> > >> > "Could not find the main class: org.apache.pig.Main"
>> > >> >
>> > >> > I forgot how to resolve this.
>> > >> >
>> > >> > Thanks a lot so far, by the way :)
>> > >> >
>> > >> > -Turner
>> > >> >
>> > >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
>> sms@yahoo-inc.com
>> > >> >wrote:
>> > >> >
>> > >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the
>> hod
>> > >> >> parameters and thing should start working
>> > >> >>
>> > >> >> -----Original Message-----
>> > >> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
>> > >> >> Sent: Wednesday, August 19, 2009 11:52 AM
>> > >> >> To: pig-user@hadoop.apache.org
>> > >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>> > >> >>
>> > >> >> If you are not using Hadoop on Demand (hod), you will need to
>> comment
>> > >> >> out the hod properties in pig's conf dir.
>> > >> >>
>> > >> >> -----Original Message-----
>> > >> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> > >> >> Sent: Wednesday, August 19, 2009 11:50 AM
>> > >> >> To: pig-user@hadoop.apache.org
>> > >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> > >> >>
>> > >> >> Getting closer...
>> > >> >> Actually got and error this time:
>> > >> >> ERROR 6012: Unable to run command: hod allocate ....
>> > >> >>
>> > >> >> -Turner
>> > >> >>
>> > >> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
>> > >> >> <sm...@yahoo-inc.com>wrote:
>> > >> >>
>> > >> >> > You need a special hadoop20.jar to build the pig sources. Check
>> out
>> > >> >> PIG-660
>> > >> >> >
>> > >> >> > Santhosh
>> > >> >> >
>> > >> >> >
>> > >> >> > -----Original Message-----
>> > >> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> > >> >> > Sent: Wed 8/19/2009 9:53 AM
>> > >> >> > To: pig-user@hadoop.apache.org
>> > >> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> > >> >> >
>> > >> >> > Hm, still nothing. Maybe I have to build it differently? I will
>> play
>> > >> >> > around with the environment settings, but any more input is
>> > >> >> appreciated.
>> > >> >> >
>> > >> >> > -Turner
>> > >> >> >
>> > >> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
>> > >> >> <dvryaboy@cloudera.com
>> > >> >> > >wrote:
>> > >> >> >
>> > >> >> > > Don't point it at mapred-site.xml directly, but the conf
>> directory,
>> > >> >> > > instead.
>> > >> >> > > (it is smart enough to pick up the new config files when it's
>> > >> >> running on
>> > >> >> > > 20).
>> > >> >> > >
>> > >> >> > > I tried setting the path directly to mapred-site.xml like you
>> have
>> > >> >> it,
>> > >> >> > > and indeed, that doesn't work. Changing to the directory should
>> fix
>> > >> >> > > your issue.
>> > >> >> > >
>> > >> >> > > -D
>> > >> >> > >
>> > >> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<
>> thkunkel@gmail.com>
>> > >> >> > wrote:
>> > >> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig.
>> Was
>> > >> >> > working
>> > >> >> > > > with 0.18.0 and had it working fine.
>> > >> >> > > > With 0.20.0 I imagine you're correct with changing
>> PIG_CLASSPATH
>> > >> >> since
>> > >> >> > > Pig
>> > >> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
>> > >> >> > > > My environment looks like this now:
>> > >> >> > > >
>> > >> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
>> > >> >> conf/mapred-site.xml
>> > >> >> > > > PIGDIR=/usr/local/pig-0.3.0
>> > >> >> > > > HADOOP_HOME=/usr/local/hadoop
>> > >> >> > > > HADOOPDIR=/usr/local/hadoop/conf
>> > >> >> > > > PIG_HADOOP_VERSION=20
>> > >> >> > > >
>> > >> >> > > > Pig still isn't connecting correctly.
>> > >> >> > > >
>> > >> >> > > > -Turner
>> > >> >> > > >
>> > >> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
>> > >> >> <dvryaboy@cloudera.com
>> > >> >> > > >wrote:
>> > >> >> > > >
>> > >> >> > > >> Nipun and Turner,
>> > >> >> > > >> What are you setting PIG_CLASSPATH to?
>> > >> >> > > >>
>> > >> >> > > >> My environment works if I set it to
>> > >> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
>> > >> >> > > >>
>> > >> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both
>> lead to
>> > >> >> > > >> breakage -- I haven't quite decided if that's a bug or not.)
>> > >> >> > > >>
>> > >> >> > > >> For completeness, a full set of environment variables that
>> make
>> > >> >> it
>> > >> >> > > >> work for me. I am running the Cloudera CDH2 testing package
>> for
>> > >> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a
>> patched
>> > >> >> > version
>> > >> >> > > >> of Pig from trunk.
>> > >> >> > > >>
>> > >> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> > >> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> > >> >> > > >> PIG_HADOOP_VERSION=20
>> > >> >> > > >> PIG_CONF_DIR=./conf/
>> > >> >> > > >>
>> > >> >> > > >>
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
>> > >> >> /pig.jar
>> > >> >> > > >>
>> > >> >> > > >>
>> > >> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
>> > >> >> Kunkel<th...@gmail.com>
>> > >> >> > > wrote:
>> > >> >> > > >> > Hello,
>> > >> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and
>> Pig
>> > >> >> no
>> > >> >> > > longer
>> > >> >> > > >> > connects to the HDFS.
>> > >> >> > > >> > I tried rebuilding and applying patch PIG660.
>> > >> >> > > >> > I have a script that I run that exports these environment
>> > >> >> tables
>> > >> >> > > >> correctly:
>> > >> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
>> > >> >> > > >> > Pig still connects to file:/// instead of HDFS.
>> > >> >> > > >> > I rebuilt the Pig package so that there are no patches
>> applied
>> > >> >> at
>> > >> >> > the
>> > >> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first
>> time.
>> > >> >> > > >> >
>> > >> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop
>> 0.20.0
>> > >> >> from
>> > >> >> > the
>> > >> >> > > >> start?
>> > >> >> > > >> >
>> > >> >> > > >> > Thanks!
>> > >> >> > > >> >
>> > >> >> > > >> > --
>> > >> >> > > >> >
>> > >> >> > > >> > -Turner Kunkel
>> > >> >> > > >> >
>> > >> >> > > >>
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >>
>> > >> >
>> > >>
>> > >
>>
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by George Pang <p0...@gmail.com>.
Hi Turner,
I use Alex's tutorial on building Pig with Ant but get errors like this,
........
er.java:36: package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.LogFactory;
[javac] ^
[javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.Log;
[javac] ^
[javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.LogFactory;
[javac] ^
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
BUILD FAILED
/home/hadoop20/pig/build.xml:246: The following error occurred while
executing this line:
/home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
output for details.
Where did I get wrong? Or where to find the error output file?
Thank you.
George
2009/8/20 Turner Kunkel <th...@gmail.com>
> I built hadoop20.jar using ant after patching the Hadoop 0.20.0
> download with the patch provided on Alex's how-to site.
> I grabbed Pig using svn and patched it with PIG-660_4.patch, then
> moved hadoop20.jar into Pig's lib directory, then built pig.jar with
> ant.
> I set my classpath to include the hadoop conf folder and pig.jar
> (along with the other environment settings).
> I execute pig with this command:
>
> java -Dfs.default.name=hdfs://localhost:9000
> -Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
> and mapred settings of course)
>
> So I was doing multiple things differently after coming from Hadoop
> 18, but this seems to work.
>
> -Turner
>
> On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <dv...@cloudera.com>
> wrote:
> >
> > Turner,
> > Before you went through those steps, did you use a vanilla hadoop 20
> > jar or did you get one from the zebra ticket?
> > I am still at a loss as to why it wasn't working for you before.
> > -D
> >
> > On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> > > It's finally working!
> > >
> > > Using your environment table specs and
> > > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
> I got it to work.
> > >
> > > Thanks much, this helps me a lot. Have a nice day.
> > >
> > > -Turner
> > >
> > > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >wrote:
> > >
> > >> Tumer,
> > >> That error means you dropped pig.jar from your classpath.
> > >>
> > >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com>
> wrote:
> > >> > Ok, I know I'm really close now, I ran into this when running Pig on
> 18.
> > >> > "Could not find the main class: org.apache.pig.Main"
> > >> >
> > >> > I forgot how to resolve this.
> > >> >
> > >> > Thanks a lot so far, by the way :)
> > >> >
> > >> > -Turner
> > >> >
> > >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
> sms@yahoo-inc.com
> > >> >wrote:
> > >> >
> > >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the
> hod
> > >> >> parameters and thing should start working
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
> > >> >> Sent: Wednesday, August 19, 2009 11:52 AM
> > >> >> To: pig-user@hadoop.apache.org
> > >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> > >> >>
> > >> >> If you are not using Hadoop on Demand (hod), you will need to
> comment
> > >> >> out the hod properties in pig's conf dir.
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> > >> >> Sent: Wednesday, August 19, 2009 11:50 AM
> > >> >> To: pig-user@hadoop.apache.org
> > >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> > >> >>
> > >> >> Getting closer...
> > >> >> Actually got and error this time:
> > >> >> ERROR 6012: Unable to run command: hod allocate ....
> > >> >>
> > >> >> -Turner
> > >> >>
> > >> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> > >> >> <sm...@yahoo-inc.com>wrote:
> > >> >>
> > >> >> > You need a special hadoop20.jar to build the pig sources. Check
> out
> > >> >> PIG-660
> > >> >> >
> > >> >> > Santhosh
> > >> >> >
> > >> >> >
> > >> >> > -----Original Message-----
> > >> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
> > >> >> > Sent: Wed 8/19/2009 9:53 AM
> > >> >> > To: pig-user@hadoop.apache.org
> > >> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> > >> >> >
> > >> >> > Hm, still nothing. Maybe I have to build it differently? I will
> play
> > >> >> > around with the environment settings, but any more input is
> > >> >> appreciated.
> > >> >> >
> > >> >> > -Turner
> > >> >> >
> > >> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> > >> >> <dvryaboy@cloudera.com
> > >> >> > >wrote:
> > >> >> >
> > >> >> > > Don't point it at mapred-site.xml directly, but the conf
> directory,
> > >> >> > > instead.
> > >> >> > > (it is smart enough to pick up the new config files when it's
> > >> >> running on
> > >> >> > > 20).
> > >> >> > >
> > >> >> > > I tried setting the path directly to mapred-site.xml like you
> have
> > >> >> it,
> > >> >> > > and indeed, that doesn't work. Changing to the directory should
> fix
> > >> >> > > your issue.
> > >> >> > >
> > >> >> > > -D
> > >> >> > >
> > >> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<
> thkunkel@gmail.com>
> > >> >> > wrote:
> > >> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig.
> Was
> > >> >> > working
> > >> >> > > > with 0.18.0 and had it working fine.
> > >> >> > > > With 0.20.0 I imagine you're correct with changing
> PIG_CLASSPATH
> > >> >> since
> > >> >> > > Pig
> > >> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > >> >> > > > My environment looks like this now:
> > >> >> > > >
> > >> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> > >> >> conf/mapred-site.xml
> > >> >> > > > PIGDIR=/usr/local/pig-0.3.0
> > >> >> > > > HADOOP_HOME=/usr/local/hadoop
> > >> >> > > > HADOOPDIR=/usr/local/hadoop/conf
> > >> >> > > > PIG_HADOOP_VERSION=20
> > >> >> > > >
> > >> >> > > > Pig still isn't connecting correctly.
> > >> >> > > >
> > >> >> > > > -Turner
> > >> >> > > >
> > >> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> > >> >> <dvryaboy@cloudera.com
> > >> >> > > >wrote:
> > >> >> > > >
> > >> >> > > >> Nipun and Turner,
> > >> >> > > >> What are you setting PIG_CLASSPATH to?
> > >> >> > > >>
> > >> >> > > >> My environment works if I set it to
> > >> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >> >> > > >>
> > >> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both
> lead to
> > >> >> > > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >> >> > > >>
> > >> >> > > >> For completeness, a full set of environment variables that
> make
> > >> >> it
> > >> >> > > >> work for me. I am running the Cloudera CDH2 testing package
> for
> > >> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a
> patched
> > >> >> > version
> > >> >> > > >> of Pig from trunk.
> > >> >> > > >>
> > >> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> >> > > >> PIG_HADOOP_VERSION=20
> > >> >> > > >> PIG_CONF_DIR=./conf/
> > >> >> > > >>
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> > >> >> /pig.jar
> > >> >> > > >>
> > >> >> > > >>
> > >> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> > >> >> Kunkel<th...@gmail.com>
> > >> >> > > wrote:
> > >> >> > > >> > Hello,
> > >> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and
> Pig
> > >> >> no
> > >> >> > > longer
> > >> >> > > >> > connects to the HDFS.
> > >> >> > > >> > I tried rebuilding and applying patch PIG660.
> > >> >> > > >> > I have a script that I run that exports these environment
> > >> >> tables
> > >> >> > > >> correctly:
> > >> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> >> > > >> > Pig still connects to file:/// instead of HDFS.
> > >> >> > > >> > I rebuilt the Pig package so that there are no patches
> applied
> > >> >> at
> > >> >> > the
> > >> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first
> time.
> > >> >> > > >> >
> > >> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop
> 0.20.0
> > >> >> from
> > >> >> > the
> > >> >> > > >> start?
> > >> >> > > >> >
> > >> >> > > >> > Thanks!
> > >> >> > > >> >
> > >> >> > > >> > --
> > >> >> > > >> >
> > >> >> > > >> > -Turner Kunkel
> > >> >> > > >> >
> > >> >> > > >>
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >>
> > >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
I built hadoop20.jar using ant after patching the Hadoop 0.20.0
download with the patch provided on Alex's how-to site.
I grabbed Pig using svn and patched it with PIG-660_4.patch, then
moved hadoop20.jar into Pig's lib directory, then built pig.jar with
ant.
I set my classpath to include the hadoop conf folder and pig.jar
(along with the other environment settings).
I execute pig with this command:
java -Dfs.default.name=hdfs://localhost:9000
-Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
and mapred settings of course)
So I was doing multiple things differently after coming from Hadoop
18, but this seems to work.
-Turner
On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <dv...@cloudera.com> wrote:
>
> Turner,
> Before you went through those steps, did you use a vanilla hadoop 20
> jar or did you get one from the zebra ticket?
> I am still at a loss as to why it wasn't working for you before.
> -D
>
> On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<th...@gmail.com> wrote:
> > It's finally working!
> >
> > Using your environment table specs and
> > http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.
> >
> > Thanks much, this helps me a lot. Have a nice day.
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> >
> >> Tumer,
> >> That error means you dropped pig.jar from your classpath.
> >>
> >> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com> wrote:
> >> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
> >> > "Could not find the main class: org.apache.pig.Main"
> >> >
> >> > I forgot how to resolve this.
> >> >
> >> > Thanks a lot so far, by the way :)
> >> >
> >> > -Turner
> >> >
> >> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <sms@yahoo-inc.com
> >> >wrote:
> >> >
> >> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> >> >> parameters and thing should start working
> >> >>
> >> >> -----Original Message-----
> >> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
> >> >> Sent: Wednesday, August 19, 2009 11:52 AM
> >> >> To: pig-user@hadoop.apache.org
> >> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> >> >>
> >> >> If you are not using Hadoop on Demand (hod), you will need to comment
> >> >> out the hod properties in pig's conf dir.
> >> >>
> >> >> -----Original Message-----
> >> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> >> Sent: Wednesday, August 19, 2009 11:50 AM
> >> >> To: pig-user@hadoop.apache.org
> >> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >>
> >> >> Getting closer...
> >> >> Actually got and error this time:
> >> >> ERROR 6012: Unable to run command: hod allocate ....
> >> >>
> >> >> -Turner
> >> >>
> >> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> >> >> <sm...@yahoo-inc.com>wrote:
> >> >>
> >> >> > You need a special hadoop20.jar to build the pig sources. Check out
> >> >> PIG-660
> >> >> >
> >> >> > Santhosh
> >> >> >
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> >> > Sent: Wed 8/19/2009 9:53 AM
> >> >> > To: pig-user@hadoop.apache.org
> >> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >> >
> >> >> > Hm, still nothing. Maybe I have to build it differently? I will play
> >> >> > around with the environment settings, but any more input is
> >> >> appreciated.
> >> >> >
> >> >> > -Turner
> >> >> >
> >> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> >> >> <dvryaboy@cloudera.com
> >> >> > >wrote:
> >> >> >
> >> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
> >> >> > > instead.
> >> >> > > (it is smart enough to pick up the new config files when it's
> >> >> running on
> >> >> > > 20).
> >> >> > >
> >> >> > > I tried setting the path directly to mapred-site.xml like you have
> >> >> it,
> >> >> > > and indeed, that doesn't work. Changing to the directory should fix
> >> >> > > your issue.
> >> >> > >
> >> >> > > -D
> >> >> > >
> >> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> >> >> > wrote:
> >> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> >> >> > working
> >> >> > > > with 0.18.0 and had it working fine.
> >> >> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
> >> >> since
> >> >> > > Pig
> >> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> >> >> > > > My environment looks like this now:
> >> >> > > >
> >> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> >> >> conf/mapred-site.xml
> >> >> > > > PIGDIR=/usr/local/pig-0.3.0
> >> >> > > > HADOOP_HOME=/usr/local/hadoop
> >> >> > > > HADOOPDIR=/usr/local/hadoop/conf
> >> >> > > > PIG_HADOOP_VERSION=20
> >> >> > > >
> >> >> > > > Pig still isn't connecting correctly.
> >> >> > > >
> >> >> > > > -Turner
> >> >> > > >
> >> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> >> >> <dvryaboy@cloudera.com
> >> >> > > >wrote:
> >> >> > > >
> >> >> > > >> Nipun and Turner,
> >> >> > > >> What are you setting PIG_CLASSPATH to?
> >> >> > > >>
> >> >> > > >> My environment works if I set it to
> >> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> >> >> > > >>
> >> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> >> > > >> breakage -- I haven't quite decided if that's a bug or not.)
> >> >> > > >>
> >> >> > > >> For completeness, a full set of environment variables that make
> >> >> it
> >> >> > > >> work for me. I am running the Cloudera CDH2 testing package for
> >> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> >> >> > version
> >> >> > > >> of Pig from trunk.
> >> >> > > >>
> >> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> >> > > >> PIG_HADOOP_VERSION=20
> >> >> > > >> PIG_CONF_DIR=./conf/
> >> >> > > >>
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> >> >> /pig.jar
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> >> >> Kunkel<th...@gmail.com>
> >> >> > > wrote:
> >> >> > > >> > Hello,
> >> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
> >> >> no
> >> >> > > longer
> >> >> > > >> > connects to the HDFS.
> >> >> > > >> > I tried rebuilding and applying patch PIG660.
> >> >> > > >> > I have a script that I run that exports these environment
> >> >> tables
> >> >> > > >> correctly:
> >> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> >> > > >> > Pig still connects to file:/// instead of HDFS.
> >> >> > > >> > I rebuilt the Pig package so that there are no patches applied
> >> >> at
> >> >> > the
> >> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >> > > >> >
> >> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
> >> >> from
> >> >> > the
> >> >> > > >> start?
> >> >> > > >> >
> >> >> > > >> > Thanks!
> >> >> > > >> >
> >> >> > > >> > --
> >> >> > > >> >
> >> >> > > >> > -Turner Kunkel
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >
> >>
> >
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Dmitriy Ryaboy <dv...@cloudera.com>.
Turner,
Before you went through those steps, did you use a vanilla hadoop 20
jar or did you get one from the zebra ticket?
I am still at a loss as to why it wasn't working for you before.
-D
On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<th...@gmail.com> wrote:
> It's finally working!
>
> Using your environment table specs and
> http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.
>
> Thanks much, this helps me a lot. Have a nice day.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
>
>> Tumer,
>> That error means you dropped pig.jar from your classpath.
>>
>> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com> wrote:
>> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
>> > "Could not find the main class: org.apache.pig.Main"
>> >
>> > I forgot how to resolve this.
>> >
>> > Thanks a lot so far, by the way :)
>> >
>> > -Turner
>> >
>> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <sms@yahoo-inc.com
>> >wrote:
>> >
>> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
>> >> parameters and thing should start working
>> >>
>> >> -----Original Message-----
>> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
>> >> Sent: Wednesday, August 19, 2009 11:52 AM
>> >> To: pig-user@hadoop.apache.org
>> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>> >>
>> >> If you are not using Hadoop on Demand (hod), you will need to comment
>> >> out the hod properties in pig's conf dir.
>> >>
>> >> -----Original Message-----
>> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> >> Sent: Wednesday, August 19, 2009 11:50 AM
>> >> To: pig-user@hadoop.apache.org
>> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >>
>> >> Getting closer...
>> >> Actually got and error this time:
>> >> ERROR 6012: Unable to run command: hod allocate ....
>> >>
>> >> -Turner
>> >>
>> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
>> >> <sm...@yahoo-inc.com>wrote:
>> >>
>> >> > You need a special hadoop20.jar to build the pig sources. Check out
>> >> PIG-660
>> >> >
>> >> > Santhosh
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> >> > Sent: Wed 8/19/2009 9:53 AM
>> >> > To: pig-user@hadoop.apache.org
>> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >> >
>> >> > Hm, still nothing. Maybe I have to build it differently? I will play
>> >> > around with the environment settings, but any more input is
>> >> appreciated.
>> >> >
>> >> > -Turner
>> >> >
>> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
>> >> <dvryaboy@cloudera.com
>> >> > >wrote:
>> >> >
>> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
>> >> > > instead.
>> >> > > (it is smart enough to pick up the new config files when it's
>> >> running on
>> >> > > 20).
>> >> > >
>> >> > > I tried setting the path directly to mapred-site.xml like you have
>> >> it,
>> >> > > and indeed, that doesn't work. Changing to the directory should fix
>> >> > > your issue.
>> >> > >
>> >> > > -D
>> >> > >
>> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
>> >> > wrote:
>> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
>> >> > working
>> >> > > > with 0.18.0 and had it working fine.
>> >> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
>> >> since
>> >> > > Pig
>> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
>> >> > > > My environment looks like this now:
>> >> > > >
>> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
>> >> > > >
>> >> > >
>> >> >
>> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
>> >> conf/mapred-site.xml
>> >> > > > PIGDIR=/usr/local/pig-0.3.0
>> >> > > > HADOOP_HOME=/usr/local/hadoop
>> >> > > > HADOOPDIR=/usr/local/hadoop/conf
>> >> > > > PIG_HADOOP_VERSION=20
>> >> > > >
>> >> > > > Pig still isn't connecting correctly.
>> >> > > >
>> >> > > > -Turner
>> >> > > >
>> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
>> >> <dvryaboy@cloudera.com
>> >> > > >wrote:
>> >> > > >
>> >> > > >> Nipun and Turner,
>> >> > > >> What are you setting PIG_CLASSPATH to?
>> >> > > >>
>> >> > > >> My environment works if I set it to
>> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
>> >> > > >>
>> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
>> >> > > >> breakage -- I haven't quite decided if that's a bug or not.)
>> >> > > >>
>> >> > > >> For completeness, a full set of environment variables that make
>> >> it
>> >> > > >> work for me. I am running the Cloudera CDH2 testing package for
>> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
>> >> > version
>> >> > > >> of Pig from trunk.
>> >> > > >>
>> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> >> > > >> PIG_HADOOP_VERSION=20
>> >> > > >> PIG_CONF_DIR=./conf/
>> >> > > >>
>> >> > > >>
>> >> > >
>> >> >
>> >> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
>> >> /pig.jar
>> >> > > >>
>> >> > > >>
>> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
>> >> Kunkel<th...@gmail.com>
>> >> > > wrote:
>> >> > > >> > Hello,
>> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
>> >> no
>> >> > > longer
>> >> > > >> > connects to the HDFS.
>> >> > > >> > I tried rebuilding and applying patch PIG660.
>> >> > > >> > I have a script that I run that exports these environment
>> >> tables
>> >> > > >> correctly:
>> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
>> >> > > >> > Pig still connects to file:/// instead of HDFS.
>> >> > > >> > I rebuilt the Pig package so that there are no patches applied
>> >> at
>> >> > the
>> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
>> >> > > >> >
>> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
>> >> from
>> >> > the
>> >> > > >> start?
>> >> > > >> >
>> >> > > >> > Thanks!
>> >> > > >> >
>> >> > > >> > --
>> >> > > >> >
>> >> > > >> > -Turner Kunkel
>> >> > > >> >
>> >> > > >>
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >>
>> >
>>
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
It's finally working!
Using your environment table specs and
http://behemoth.strlen.net/~alex/hadoop20-pig-howto.txt, I got it to work.
Thanks much, this helps me a lot. Have a nice day.
-Turner
On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> Tumer,
> That error means you dropped pig.jar from your classpath.
>
> On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com> wrote:
> > Ok, I know I'm really close now, I ran into this when running Pig on 18.
> > "Could not find the main class: org.apache.pig.Main"
> >
> > I forgot how to resolve this.
> >
> > Thanks a lot so far, by the way :)
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <sms@yahoo-inc.com
> >wrote:
> >
> >> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> >> parameters and thing should start working
> >>
> >> -----Original Message-----
> >> From: Naber, Chad [mailto:CNaber@edmunds.com]
> >> Sent: Wednesday, August 19, 2009 11:52 AM
> >> To: pig-user@hadoop.apache.org
> >> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
> >>
> >> If you are not using Hadoop on Demand (hod), you will need to comment
> >> out the hod properties in pig's conf dir.
> >>
> >> -----Original Message-----
> >> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> Sent: Wednesday, August 19, 2009 11:50 AM
> >> To: pig-user@hadoop.apache.org
> >> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >>
> >> Getting closer...
> >> Actually got and error this time:
> >> ERROR 6012: Unable to run command: hod allocate ....
> >>
> >> -Turner
> >>
> >> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> >> <sm...@yahoo-inc.com>wrote:
> >>
> >> > You need a special hadoop20.jar to build the pig sources. Check out
> >> PIG-660
> >> >
> >> > Santhosh
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
> >> > Sent: Wed 8/19/2009 9:53 AM
> >> > To: pig-user@hadoop.apache.org
> >> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >> >
> >> > Hm, still nothing. Maybe I have to build it differently? I will play
> >> > around with the environment settings, but any more input is
> >> appreciated.
> >> >
> >> > -Turner
> >> >
> >> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> >> <dvryaboy@cloudera.com
> >> > >wrote:
> >> >
> >> > > Don't point it at mapred-site.xml directly, but the conf directory,
> >> > > instead.
> >> > > (it is smart enough to pick up the new config files when it's
> >> running on
> >> > > 20).
> >> > >
> >> > > I tried setting the path directly to mapred-site.xml like you have
> >> it,
> >> > > and indeed, that doesn't work. Changing to the directory should fix
> >> > > your issue.
> >> > >
> >> > > -D
> >> > >
> >> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> >> > wrote:
> >> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> >> > working
> >> > > > with 0.18.0 and had it working fine.
> >> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
> >> since
> >> > > Pig
> >> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> >> > > > My environment looks like this now:
> >> > > >
> >> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >> > > >
> >> > >
> >> >
> >> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> >> conf/mapred-site.xml
> >> > > > PIGDIR=/usr/local/pig-0.3.0
> >> > > > HADOOP_HOME=/usr/local/hadoop
> >> > > > HADOOPDIR=/usr/local/hadoop/conf
> >> > > > PIG_HADOOP_VERSION=20
> >> > > >
> >> > > > Pig still isn't connecting correctly.
> >> > > >
> >> > > > -Turner
> >> > > >
> >> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> >> <dvryaboy@cloudera.com
> >> > > >wrote:
> >> > > >
> >> > > >> Nipun and Turner,
> >> > > >> What are you setting PIG_CLASSPATH to?
> >> > > >>
> >> > > >> My environment works if I set it to
> >> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> >> > > >>
> >> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> > > >> breakage -- I haven't quite decided if that's a bug or not.)
> >> > > >>
> >> > > >> For completeness, a full set of environment variables that make
> >> it
> >> > > >> work for me. I am running the Cloudera CDH2 testing package for
> >> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> >> > version
> >> > > >> of Pig from trunk.
> >> > > >>
> >> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> > > >> PIG_HADOOP_VERSION=20
> >> > > >> PIG_CONF_DIR=./conf/
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> >> /pig.jar
> >> > > >>
> >> > > >>
> >> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> >> Kunkel<th...@gmail.com>
> >> > > wrote:
> >> > > >> > Hello,
> >> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
> >> no
> >> > > longer
> >> > > >> > connects to the HDFS.
> >> > > >> > I tried rebuilding and applying patch PIG660.
> >> > > >> > I have a script that I run that exports these environment
> >> tables
> >> > > >> correctly:
> >> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > > >> > Pig still connects to file:/// instead of HDFS.
> >> > > >> > I rebuilt the Pig package so that there are no patches applied
> >> at
> >> > the
> >> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> > > >> >
> >> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
> >> from
> >> > the
> >> > > >> start?
> >> > > >> >
> >> > > >> > Thanks!
> >> > > >> >
> >> > > >> > --
> >> > > >> >
> >> > > >> > -Turner Kunkel
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >>
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Dmitriy Ryaboy <dv...@cloudera.com>.
Tumer,
That error means you dropped pig.jar from your classpath.
On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<th...@gmail.com> wrote:
> Ok, I know I'm really close now, I ran into this when running Pig on 18.
> "Could not find the main class: org.apache.pig.Main"
>
> I forgot how to resolve this.
>
> Thanks a lot so far, by the way :)
>
> -Turner
>
> On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <sm...@yahoo-inc.com>wrote:
>
>> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
>> parameters and thing should start working
>>
>> -----Original Message-----
>> From: Naber, Chad [mailto:CNaber@edmunds.com]
>> Sent: Wednesday, August 19, 2009 11:52 AM
>> To: pig-user@hadoop.apache.org
>> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>>
>> If you are not using Hadoop on Demand (hod), you will need to comment
>> out the hod properties in pig's conf dir.
>>
>> -----Original Message-----
>> From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> Sent: Wednesday, August 19, 2009 11:50 AM
>> To: pig-user@hadoop.apache.org
>> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>>
>> Getting closer...
>> Actually got and error this time:
>> ERROR 6012: Unable to run command: hod allocate ....
>>
>> -Turner
>>
>> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
>> <sm...@yahoo-inc.com>wrote:
>>
>> > You need a special hadoop20.jar to build the pig sources. Check out
>> PIG-660
>> >
>> > Santhosh
>> >
>> >
>> > -----Original Message-----
>> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
>> > Sent: Wed 8/19/2009 9:53 AM
>> > To: pig-user@hadoop.apache.org
>> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>> >
>> > Hm, still nothing. Maybe I have to build it differently? I will play
>> > around with the environment settings, but any more input is
>> appreciated.
>> >
>> > -Turner
>> >
>> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
>> <dvryaboy@cloudera.com
>> > >wrote:
>> >
>> > > Don't point it at mapred-site.xml directly, but the conf directory,
>> > > instead.
>> > > (it is smart enough to pick up the new config files when it's
>> running on
>> > > 20).
>> > >
>> > > I tried setting the path directly to mapred-site.xml like you have
>> it,
>> > > and indeed, that doesn't work. Changing to the directory should fix
>> > > your issue.
>> > >
>> > > -D
>> > >
>> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
>> > wrote:
>> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
>> > working
>> > > > with 0.18.0 and had it working fine.
>> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
>> since
>> > > Pig
>> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
>> > > > My environment looks like this now:
>> > > >
>> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
>> > > >
>> > >
>> >
>> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
>> conf/mapred-site.xml
>> > > > PIGDIR=/usr/local/pig-0.3.0
>> > > > HADOOP_HOME=/usr/local/hadoop
>> > > > HADOOPDIR=/usr/local/hadoop/conf
>> > > > PIG_HADOOP_VERSION=20
>> > > >
>> > > > Pig still isn't connecting correctly.
>> > > >
>> > > > -Turner
>> > > >
>> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
>> <dvryaboy@cloudera.com
>> > > >wrote:
>> > > >
>> > > >> Nipun and Turner,
>> > > >> What are you setting PIG_CLASSPATH to?
>> > > >>
>> > > >> My environment works if I set it to
>> > > >> /path/to/pig.jar:path/to/mapred-site.xml
>> > > >>
>> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
>> > > >> breakage -- I haven't quite decided if that's a bug or not.)
>> > > >>
>> > > >> For completeness, a full set of environment variables that make
>> it
>> > > >> work for me. I am running the Cloudera CDH2 testing package for
>> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
>> > version
>> > > >> of Pig from trunk.
>> > > >>
>> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> > > >> PIG_HADOOP_VERSION=20
>> > > >> PIG_CONF_DIR=./conf/
>> > > >>
>> > > >>
>> > >
>> >
>> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
>> /pig.jar
>> > > >>
>> > > >>
>> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
>> Kunkel<th...@gmail.com>
>> > > wrote:
>> > > >> > Hello,
>> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
>> no
>> > > longer
>> > > >> > connects to the HDFS.
>> > > >> > I tried rebuilding and applying patch PIG660.
>> > > >> > I have a script that I run that exports these environment
>> tables
>> > > >> correctly:
>> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
>> > > >> > Pig still connects to file:/// instead of HDFS.
>> > > >> > I rebuilt the Pig package so that there are no patches applied
>> at
>> > the
>> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
>> > > >> >
>> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
>> from
>> > the
>> > > >> start?
>> > > >> >
>> > > >> > Thanks!
>> > > >> >
>> > > >> > --
>> > > >> >
>> > > >> > -Turner Kunkel
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>>
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
Ok, I know I'm really close now, I ran into this when running Pig on 18.
"Could not find the main class: org.apache.pig.Main"
I forgot how to resolve this.
Thanks a lot so far, by the way :)
-Turner
On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <sm...@yahoo-inc.com>wrote:
> AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
> parameters and thing should start working
>
> -----Original Message-----
> From: Naber, Chad [mailto:CNaber@edmunds.com]
> Sent: Wednesday, August 19, 2009 11:52 AM
> To: pig-user@hadoop.apache.org
> Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
>
> If you are not using Hadoop on Demand (hod), you will need to comment
> out the hod properties in pig's conf dir.
>
> -----Original Message-----
> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> Sent: Wednesday, August 19, 2009 11:50 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Getting closer...
> Actually got and error this time:
> ERROR 6012: Unable to run command: hod allocate ....
>
> -Turner
>
> On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
> <sm...@yahoo-inc.com>wrote:
>
> > You need a special hadoop20.jar to build the pig sources. Check out
> PIG-660
> >
> > Santhosh
> >
> >
> > -----Original Message-----
> > From: Turner Kunkel [mailto:thkunkel@gmail.com]
> > Sent: Wed 8/19/2009 9:53 AM
> > To: pig-user@hadoop.apache.org
> > Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
> >
> > Hm, still nothing. Maybe I have to build it differently? I will play
> > around with the environment settings, but any more input is
> appreciated.
> >
> > -Turner
> >
> > On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
> <dvryaboy@cloudera.com
> > >wrote:
> >
> > > Don't point it at mapred-site.xml directly, but the conf directory,
> > > instead.
> > > (it is smart enough to pick up the new config files when it's
> running on
> > > 20).
> > >
> > > I tried setting the path directly to mapred-site.xml like you have
> it,
> > > and indeed, that doesn't work. Changing to the directory should fix
> > > your issue.
> > >
> > > -D
> > >
> > > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> > wrote:
> > > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> > working
> > > > with 0.18.0 and had it working fine.
> > > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
> since
> > > Pig
> > > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > > My environment looks like this now:
> > > >
> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > >
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
> conf/mapred-site.xml
> > > > PIGDIR=/usr/local/pig-0.3.0
> > > > HADOOP_HOME=/usr/local/hadoop
> > > > HADOOPDIR=/usr/local/hadoop/conf
> > > > PIG_HADOOP_VERSION=20
> > > >
> > > > Pig still isn't connecting correctly.
> > > >
> > > > -Turner
> > > >
> > > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
> <dvryaboy@cloudera.com
> > > >wrote:
> > > >
> > > >> Nipun and Turner,
> > > >> What are you setting PIG_CLASSPATH to?
> > > >>
> > > >> My environment works if I set it to
> > > >> /path/to/pig.jar:path/to/mapred-site.xml
> > > >>
> > > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > > >> breakage -- I haven't quite decided if that's a bug or not.)
> > > >>
> > > >> For completeness, a full set of environment variables that make
> it
> > > >> work for me. I am running the Cloudera CDH2 testing package for
> > > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> > version
> > > >> of Pig from trunk.
> > > >>
> > > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > > >> PIG_HADOOP_VERSION=20
> > > >> PIG_CONF_DIR=./conf/
> > > >>
> > > >>
> > >
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
> /pig.jar
> > > >>
> > > >>
> > > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
> Kunkel<th...@gmail.com>
> > > wrote:
> > > >> > Hello,
> > > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
> no
> > > longer
> > > >> > connects to the HDFS.
> > > >> > I tried rebuilding and applying patch PIG660.
> > > >> > I have a script that I run that exports these environment
> tables
> > > >> correctly:
> > > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > > >> > Pig still connects to file:/// instead of HDFS.
> > > >> > I rebuilt the Pig package so that there are no patches applied
> at
> > the
> > > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > > >> >
> > > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
> from
> > the
> > > >> start?
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> > --
> > > >> >
> > > >> > -Turner Kunkel
> > > >> >
> > > >>
> > > >
> > >
> >
>
>
RE: Pig 0.3.0 and Hadoop 0.20.0
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
AFAIK, hadoop-20 does not support hod. As Chad suggested drop the hod
parameters and thing should start working
-----Original Message-----
From: Naber, Chad [mailto:CNaber@edmunds.com]
Sent: Wednesday, August 19, 2009 11:52 AM
To: pig-user@hadoop.apache.org
Subject: RE: Pig 0.3.0 and Hadoop 0.20.0
If you are not using Hadoop on Demand (hod), you will need to comment
out the hod properties in pig's conf dir.
-----Original Message-----
From: Turner Kunkel [mailto:thkunkel@gmail.com]
Sent: Wednesday, August 19, 2009 11:50 AM
To: pig-user@hadoop.apache.org
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....
-Turner
On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
<sm...@yahoo-inc.com>wrote:
> You need a special hadoop20.jar to build the pig sources. Check out
PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> Sent: Wed 8/19/2009 9:53 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is
appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
<dvryaboy@cloudera.com
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's
running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have
it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH
since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
>
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
<dvryaboy@cloudera.com
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make
it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
>
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner
Kunkel<th...@gmail.com>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig
no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment
tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied
at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > >> >
> > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0
from
> the
> > >> start?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > --
> > >> >
> > >> > -Turner Kunkel
> > >> >
> > >>
> > >
> >
>
RE: Pig 0.3.0 and Hadoop 0.20.0
Posted by "Naber, Chad" <CN...@edmunds.com>.
If you are not using Hadoop on Demand (hod), you will need to comment out the hod properties in pig's conf dir.
-----Original Message-----
From: Turner Kunkel [mailto:thkunkel@gmail.com]
Sent: Wednesday, August 19, 2009 11:50 AM
To: pig-user@hadoop.apache.org
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....
-Turner
On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan <sm...@yahoo-inc.com>wrote:
> You need a special hadoop20.jar to build the pig sources. Check out PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> Sent: Wed 8/19/2009 9:53 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > >> >
> > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from
> the
> > >> start?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > --
> > >> >
> > >> > -Turner Kunkel
> > >> >
> > >>
> > >
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....
-Turner
On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan <sm...@yahoo-inc.com>wrote:
> You need a special hadoop20.jar to build the pig sources. Check out PIG-660
>
> Santhosh
>
>
> -----Original Message-----
> From: Turner Kunkel [mailto:thkunkel@gmail.com]
> Sent: Wed 8/19/2009 9:53 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
>
> Hm, still nothing. Maybe I have to build it differently? I will play
> around with the environment settings, but any more input is appreciated.
>
> -Turner
>
> On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >wrote:
>
> > Don't point it at mapred-site.xml directly, but the conf directory,
> > instead.
> > (it is smart enough to pick up the new config files when it's running on
> > 20).
> >
> > I tried setting the path directly to mapred-site.xml like you have it,
> > and indeed, that doesn't work. Changing to the directory should fix
> > your issue.
> >
> > -D
> >
> > On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> > > I am running off of unpacked tar.gz files of Hadoop and Pig. Was
> working
> > > with 0.18.0 and had it working fine.
> > > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> > Pig
> > > is looking for hadoop-site.xml instead of mapred-site.xml.
> > > My environment looks like this now:
> > >
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > >
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > > PIGDIR=/usr/local/pig-0.3.0
> > > HADOOP_HOME=/usr/local/hadoop
> > > HADOOPDIR=/usr/local/hadoop/conf
> > > PIG_HADOOP_VERSION=20
> > >
> > > Pig still isn't connecting correctly.
> > >
> > > -Turner
> > >
> > > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> > >wrote:
> > >
> > >> Nipun and Turner,
> > >> What are you setting PIG_CLASSPATH to?
> > >>
> > >> My environment works if I set it to
> > >> /path/to/pig.jar:path/to/mapred-site.xml
> > >>
> > >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> > >> breakage -- I haven't quite decided if that's a bug or not.)
> > >>
> > >> For completeness, a full set of environment variables that make it
> > >> work for me. I am running the Cloudera CDH2 testing package for
> > >> debian/ubuntu (from http://archive.cloudera.com) and a patched
> version
> > >> of Pig from trunk.
> > >>
> > >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> > >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> > >> PIG_HADOOP_VERSION=20
> > >> PIG_CONF_DIR=./conf/
> > >>
> > >>
> >
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> > >>
> > >>
> > >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com>
> > wrote:
> > >> > Hello,
> > >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> > longer
> > >> > connects to the HDFS.
> > >> > I tried rebuilding and applying patch PIG660.
> > >> > I have a script that I run that exports these environment tables
> > >> correctly:
> > >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > >> > Pig still connects to file:/// instead of HDFS.
> > >> > I rebuilt the Pig package so that there are no patches applied at
> the
> > >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> > >> >
> > >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from
> the
> > >> start?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > --
> > >> >
> > >> > -Turner Kunkel
> > >> >
> > >>
> > >
> >
>
RE: Pig 0.3.0 and Hadoop 0.20.0
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
You need a special hadoop20.jar to build the pig sources. Check out PIG-660
Santhosh
-----Original Message-----
From: Turner Kunkel [mailto:thkunkel@gmail.com]
Sent: Wed 8/19/2009 9:53 AM
To: pig-user@hadoop.apache.org
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0
Hm, still nothing. Maybe I have to build it differently? I will play
around with the environment settings, but any more input is appreciated.
-Turner
On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
Hm, still nothing. Maybe I have to build it differently? I will play
around with the environment settings, but any more input is appreciated.
-Turner
On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> Don't point it at mapred-site.xml directly, but the conf directory,
> instead.
> (it is smart enough to pick up the new config files when it's running on
> 20).
>
> I tried setting the path directly to mapred-site.xml like you have it,
> and indeed, that doesn't work. Changing to the directory should fix
> your issue.
>
> -D
>
> On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com> wrote:
> > I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> > with 0.18.0 and had it working fine.
> > With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since
> Pig
> > is looking for hadoop-site.xml instead of mapred-site.xml.
> > My environment looks like this now:
> >
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> > PIGDIR=/usr/local/pig-0.3.0
> > HADOOP_HOME=/usr/local/hadoop
> > HADOOPDIR=/usr/local/hadoop/conf
> > PIG_HADOOP_VERSION=20
> >
> > Pig still isn't connecting correctly.
> >
> > -Turner
> >
> > On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dvryaboy@cloudera.com
> >wrote:
> >
> >> Nipun and Turner,
> >> What are you setting PIG_CLASSPATH to?
> >>
> >> My environment works if I set it to
> >> /path/to/pig.jar:path/to/mapred-site.xml
> >>
> >> (leaving off the path to mapred-site.xml or pig.jar both lead to
> >> breakage -- I haven't quite decided if that's a bug or not.)
> >>
> >> For completeness, a full set of environment variables that make it
> >> work for me. I am running the Cloudera CDH2 testing package for
> >> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> >> of Pig from trunk.
> >>
> >> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> >> PIGDIR=/home/dvryaboy/src/pig/trunk/
> >> PIG_HADOOP_VERSION=20
> >> PIG_CONF_DIR=./conf/
> >>
> >>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
> >>
> >>
> >> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com>
> wrote:
> >> > Hello,
> >> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no
> longer
> >> > connects to the HDFS.
> >> > I tried rebuilding and applying patch PIG660.
> >> > I have a script that I run that exports these environment tables
> >> correctly:
> >> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> >> > Pig still connects to file:/// instead of HDFS.
> >> > I rebuilt the Pig package so that there are no patches applied at the
> >> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >> >
> >> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> >> start?
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> >
> >> > -Turner Kunkel
> >> >
> >>
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Dmitriy Ryaboy <dv...@cloudera.com>.
Don't point it at mapred-site.xml directly, but the conf directory, instead.
(it is smart enough to pick up the new config files when it's running on 20).
I tried setting the path directly to mapred-site.xml like you have it,
and indeed, that doesn't work. Changing to the directory should fix
your issue.
-D
On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<th...@gmail.com> wrote:
> I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
> with 0.18.0 and had it working fine.
> With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since Pig
> is looking for hadoop-site.xml instead of mapred-site.xml.
> My environment looks like this now:
>
> JAVA_HOME=/usr/lib/jvm/java-6-sun
> PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
> PIGDIR=/usr/local/pig-0.3.0
> HADOOP_HOME=/usr/local/hadoop
> HADOOPDIR=/usr/local/hadoop/conf
> PIG_HADOOP_VERSION=20
>
> Pig still isn't connecting correctly.
>
> -Turner
>
> On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
>
>> Nipun and Turner,
>> What are you setting PIG_CLASSPATH to?
>>
>> My environment works if I set it to
>> /path/to/pig.jar:path/to/mapred-site.xml
>>
>> (leaving off the path to mapred-site.xml or pig.jar both lead to
>> breakage -- I haven't quite decided if that's a bug or not.)
>>
>> For completeness, a full set of environment variables that make it
>> work for me. I am running the Cloudera CDH2 testing package for
>> debian/ubuntu (from http://archive.cloudera.com) and a patched version
>> of Pig from trunk.
>>
>> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
>> PIGDIR=/home/dvryaboy/src/pig/trunk/
>> PIG_HADOOP_VERSION=20
>> PIG_CONF_DIR=./conf/
>>
>> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
>>
>>
>> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com> wrote:
>> > Hello,
>> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
>> > connects to the HDFS.
>> > I tried rebuilding and applying patch PIG660.
>> > I have a script that I run that exports these environment tables
>> correctly:
>> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
>> > Pig still connects to file:/// instead of HDFS.
>> > I rebuilt the Pig package so that there are no patches applied at the
>> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
>> >
>> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
>> start?
>> >
>> > Thanks!
>> >
>> > --
>> >
>> > -Turner Kunkel
>> >
>>
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Turner Kunkel <th...@gmail.com>.
I am running off of unpacked tar.gz files of Hadoop and Pig. Was working
with 0.18.0 and had it working fine.
With 0.20.0 I imagine you're correct with changing PIG_CLASSPATH since Pig
is looking for hadoop-site.xml instead of mapred-site.xml.
My environment looks like this now:
JAVA_HOME=/usr/lib/jvm/java-6-sun
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/conf/mapred-site.xml
PIGDIR=/usr/local/pig-0.3.0
HADOOP_HOME=/usr/local/hadoop
HADOOPDIR=/usr/local/hadoop/conf
PIG_HADOOP_VERSION=20
Pig still isn't connecting correctly.
-Turner
On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> Nipun and Turner,
> What are you setting PIG_CLASSPATH to?
>
> My environment works if I set it to
> /path/to/pig.jar:path/to/mapred-site.xml
>
> (leaving off the path to mapred-site.xml or pig.jar both lead to
> breakage -- I haven't quite decided if that's a bug or not.)
>
> For completeness, a full set of environment variables that make it
> work for me. I am running the Cloudera CDH2 testing package for
> debian/ubuntu (from http://archive.cloudera.com) and a patched version
> of Pig from trunk.
>
> JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
> PIGDIR=/home/dvryaboy/src/pig/trunk/
> PIG_HADOOP_VERSION=20
> PIG_CONF_DIR=./conf/
>
> PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
>
>
> On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com> wrote:
> > Hello,
> > I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
> > connects to the HDFS.
> > I tried rebuilding and applying patch PIG660.
> > I have a script that I run that exports these environment tables
> correctly:
> > JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> > Pig still connects to file:/// instead of HDFS.
> > I rebuilt the Pig package so that there are no patches applied at the
> > moment, as if I had just unpacked PIG-0.3.0 for the first time.
> >
> > So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the
> start?
> >
> > Thanks!
> >
> > --
> >
> > -Turner Kunkel
> >
>
Re: Pig 0.3.0 and Hadoop 0.20.0
Posted by Dmitriy Ryaboy <dv...@cloudera.com>.
Nipun and Turner,
What are you setting PIG_CLASSPATH to?
My environment works if I set it to /path/to/pig.jar:path/to/mapred-site.xml
(leaving off the path to mapred-site.xml or pig.jar both lead to
breakage -- I haven't quite decided if that's a bug or not.)
For completeness, a full set of environment variables that make it
work for me. I am running the Cloudera CDH2 testing package for
debian/ubuntu (from http://archive.cloudera.com) and a patched version
of Pig from trunk.
JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
PIGDIR=/home/dvryaboy/src/pig/trunk/
PIG_HADOOP_VERSION=20
PIG_CONF_DIR=./conf/
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk/pig.jar
On Tue, Aug 18, 2009 at 10:37 AM, Turner Kunkel<th...@gmail.com> wrote:
> Hello,
> I upgraded to Hadoop 0.20.0 (it's stable and working) and Pig no longer
> connects to the HDFS.
> I tried rebuilding and applying patch PIG660.
> I have a script that I run that exports these environment tables correctly:
> JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
> Pig still connects to file:/// instead of HDFS.
> I rebuilt the Pig package so that there are no patches applied at the
> moment, as if I had just unpacked PIG-0.3.0 for the first time.
>
> So, what do I do to get Pig 0.3.0 to work with Hadoop 0.20.0 from the start?
>
> Thanks!
>
> --
>
> -Turner Kunkel
>