You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Alex Parvulescu <al...@gmail.com> on 2010/02/12 15:51:11 UTC
pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Hello,
I seem to have broken my Pig install, and I don't know where to look.
If I use directly the script (grunt) everything works ok, but every time I
try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
org.apache.pig.Main script1-hadoop.pig'
I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR
Obviously I have the PIGDIR var set:
> echo $PIGDIR
> /home/alex/hadoop/pig
This is something that I did, as I have used 0.5 and 0.6 and a patched
version of 0.6 in parallel, but I can't figure out where to look. Any
version I try to start now, gives the same error.
any help would be greatly appreciated!
alex
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Still not sure how that's happening, but glad it's working for you now :-).
-D
On Wed, Feb 24, 2010 at 6:15 AM, Alex Parvulescu
<al...@gmail.com>wrote:
> Hello and sorry for the absence
>
> I went through a hard drive crash, and a 2 day recovery process.
>
> I have incredible news... Pig is working again!!!!!!!!!!
>
> it felt good to finally say it out loud
>
> I've found the problem. Check this out:
>
> --------- This is my test script -----------
>
> -- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
> test_lite.pig
>
> A = load 't.csv' using PigStorage('\t') as (id: long);
> store A into 'output/pig' using PigStorage('\t');
>
> --------------------
>
> I just saved the run instruction in this script to I don't forget it.
> Notice the '$PIGDIR' and '$HADOOPDIR'
>
> It apprears this is the source if the evil
> ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> replacing this with
> -- $whatever
> will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
> whatever
>
> Now that it's working again, I'll be back with more issues in no time :)
>
> Thanks a million to everybody, especially Dmitriy!
>
> alex
>
>
> On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan <
> ashutosh.chauhan@gmail.com> wrote:
>
> > Alex,
> >
> > I just downloaded the latest Pig 0.6 Release Candidate from
> > http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>
> <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz
> >and
> > followed the instructions in "Trying the Release" section as outlined in
> > http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>
> <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt
> >and
> > without taking any extra step anywhere in between I am able to get Pig
> > running. You may want to do the same thing. If it works that would be
> great
> > else we will be sure that its some environment problem in your setup and
> we
> > can take it from there.
> >
> > Ashutosh
> >
> > On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
> >
> > > Probably unrelated, but I always run from the root directory, not from
> > > bin/,
> > > when working with an svn version. Also, never had dev vs core issues.
> Can
> > > you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> > > pig.jar (not core, not dev) on $PIG_CLASSPATH?
> > >
> > > How about avoiding the pig script and just running
> > >
> > > java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
> > > <<yourscript>> ?
> > >
> > > I grepped through all the code, including the codegenned stuff, and
> none
> > of
> > > it contains the string PIGDIR (except bin/pig of course).
> > >
> > > At this point, assuming none of the above helps (which I have little
> hope
> > > it
> > > will), the best I can say is set up your eclipse environment and step
> > > through pig's main with the debugger.
> > >
> > > -D
> > >
> > > On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> > > <al...@gmail.com>wrote:
> > >
> > > > Hello
> > > >
> > > > This is getting tiresome :)
> > > >
> > > > Today, I tried with a fresh download of pig 0.5 and I get the exact
> > same
> > > > error. Something somewhere just broke down, and it took my machine
> with
> > > it.
> > > > At this point I have no way to run any kind of pig version.
> > > > Needless to say, this is a big problem for me :)
> > > >
> > > > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> > > error.
> > > > I even dropped the sym links.
> > > >
> > > > Back to our debugging session:
> > > >
> > > > > svn co
> > > >
> > >
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > > > A pig-branch-0.6-take2/test/bin/test-patch.sh
> > > > Checked out external at revision 910901.
> > > >
> > > > Checked out revision 910900.
> > > >
> > > > > cd pig-branch-0.6-take2/
> > > > > ant
> > > > BUILD SUCCESSFUL
> > > > Total time: 26 seconds
> > > >
> > > > > cd bin/
> > > >
> > > > next is to check if all vars are there
> > > > > printenv | grep PIG
> > > > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> > > > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> > > >
> > > > > printenv | grep HADOOP
> > > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > > >
> > > > > pig
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > jline/ConsoleReaderInputStream
> > > > Caused by: java.lang.ClassNotFoundException:
> > > jline.ConsoleReaderInputStream
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > > > Could not find the main class: org.apache.pig.Main. Program will
> exit.
> > > >
> > > > I've sent some email to the list about this problem before. A quick
> fix
> > > is
> > > > to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > > > > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
> > > >
> > > > > pig
> > > > 2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging
> > error
> > > > messages to:
> > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> > > > 2010-02-17 11:48:53,175 [main] INFO
> > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> > > Connecting
> > > > to hadoop file system at: file:///
> > > > grunt>
> > > >
> > > > Now at least it will start.
> > > >
> > > > Getting to the initial issue:
> > > >
> > > > the '-f' option:
> > > > > pig -f /home/alex/hadoop/test/test_lite.pig
> > > > 2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging
> > error
> > > > messages to:
> > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> > > > 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR
> 2999:
> > > > Unexpected internal error. Undefined parameter : PIGDIR
> > > > Details at logfile:
> > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> > > >
> > > > the '- secretDebugCmd' option
> > > >
> > > > pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
> > > > dry run:
> > > > /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> > > > -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
> > > > -Dpig.log.file=pig.log
> > > > -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
> > > > -Dpig.root.logger=INFO,console,DRFA -classpath
> > > >
> > > >
> > >
> >
> /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
> > > > org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig
> > > >
> > > > I did not change the scripts, and I get the same error.
> > > >
> > > > This is really killing me :)
> > > >
> > > > thanks,
> > > > alex
> > > >
> > > >
> > > >
> > > > On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hm, nothing jumps out. Definitely the error you are getting
> indicates
> > > > that
> > > > > somehow the preprocessor is trying to substitute a variable called
> > > PIGDIR
> > > > > in
> > > > > your script. Which is odd. Does the same thing happen if you try
> > > bin/pig
> > > > -f
> > > > > test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's
> > > > secret),
> > > > > and sending along the output. Why are you using pig-core instead of
> > > > pig.jar
> > > > > that ant should be generating for you when you build? Where did you
> > get
> > > > it,
> > > > > how did you build it, and what's the output of cksum on it?
> > > > > The fact that now your 0.5 is broken too makes me think that maybe
> > your
> > > > > symlinks are messed up. Surely a change to one jar shouldn't be
> > > affecting
> > > > a
> > > > > totally unrelated jar in a directory the first jar doesn't know
> > about.
> > > > >
> > > > > Sorry about the barrage of questions, I am just a bit dumbfounded
> > about
> > > > how
> > > > > this could even begin to start happening. Any preprocessor experts
> > > > around?
> > > > >
> > > > > -D
> > > > >
> > > > >
> > > > > On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
> > > > > <al...@gmail.com>wrote:
> > > > >
> > > > > > Hello
> > > > > >
> > > > > > And thanks again for all your help
> > > > > >
> > > > > > I have a symbolic link /home/alex/hadoop/pig which points to
> > > > > > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and
> build
> > of
> > > > the
> > > > > > 0.6 branch.
> > > > > > I'm running the script from /home/alex/hadoop/test.
> > > > > >
> > > > > > I didn't touch pig.properties, also I'm running with the default
> > pig
> > > > > > script.
> > > > > >
> > > > > > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> > > > > > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> > > > > > work(Exception
> > > > > > in thread "main" java.lang.NoClassDefFoundError:
> > > org/apache/pig/Main).
> > > > > >
> > > > > > I'm running with a fresh 0.6 build and still no luck.
> > > > > >
> > > > > > The info you reqested:
> > > > > >
> > > > > > alex@alex-desktop:~/hadoop/test$ pwd
> > > > > > /home/alex/hadoop/test
> > > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> > > > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> > > > > > PIGDIR=/home/alex/hadoop/pig
> > > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> > > > > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > > > > >
> > > > > > alex@alex-desktop:~/hadoop/test$ cat
> > > > > > /home/alex/hadoop/pig/conf/pig.properties
> > > > > > # Pig configuration file. All values can be overwritten by
> command
> > > line
> > > > > > arguments.
> > > > > > # see bin/pig -help
> > > > > >
> > > > > > # log4jconf log4j configuration file
> > > > > > # log4jconf=./conf/log4j.properties
> > > > > >
> > > > > > # brief logging (no timestamps)
> > > > > > brief=false
> > > > > >
> > > > > > # clustername, name of the hadoop jobtracker. If no port is
> defined
> > > > port
> > > > > > 50020 will be used.
> > > > > > #cluster
> > > > > >
> > > > > > #debug level, INFO is default
> > > > > > debug=INFO
> > > > > >
> > > > > > # a file that contains pig script
> > > > > > #file=
> > > > > >
> > > > > > # load jarfile, colon separated
> > > > > > #jar=
> > > > > >
> > > > > > #verbose print all log messages to screen (default to print only
> > INFO
> > > > and
> > > > > > above to screen)
> > > > > > verbose=false
> > > > > >
> > > > > > #exectype local|mapreduce, mapreduce is default
> > > > > > #exectype=mapreduce
> > > > > > # hod realted properties
> > > > > > #ssh.gateway
> > > > > > #hod.expect.root
> > > > > > #hod.expect.uselatest
> > > > > > #hod.command
> > > > > > #hod.config.dir
> > > > > > #hod.param
> > > > > >
> > > > > >
> > > > > > #Do not spill temp files smaller than this size (bytes)
> > > > > > pig.spill.size.threshold=5000000
> > > > > > #EXPERIMENT: Activate garbage collection when spilling a file
> > bigger
> > > > than
> > > > > > this size (bytes)
> > > > > > #This should help reduce the number of files being spilled.
> > > > > > pig.spill.gc.activation.size=40000000
> > > > > >
> > > > > >
> > > > > > ######################
> > > > > > # Everything below this line is Yahoo specific. Note that I've
> > made
> > > > > > # (almost) no changes to the lines above to make merging in from
> > > Apache
> > > > > > # easier. Any values I don't want from above I override below.
> > > > > > #
> > > > > > # This file is configured for use with HOD on the production
> > > clusters.
> > > > > If
> > > > > > you
> > > > > > # want to run pig with a static cluster you will need to remove
> > > > > everything
> > > > > > # below this line and set the cluster value (above) to the
> > > > > > # hostname and port of your job tracker.
> > > > > >
> > > > > > exectype=mapreduce
> > > > > >
> > > > > > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> > > > > > hod.server=local
> > > > > >
> > > > > > cluster.domain=inktomisearch.com
> > > > > >
> > > > > > log.file=
> > > > > >
> > > > > > yinst.cluster=kryptonite
> > > > > >
> > > > > >
> > > > > >
> > > > > > And now boom!
> > > > > >
> > > > > > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
> > > org.apache.pig.Main
> > > > > > test_lite.pig
> > > > > >
> > > > > >
> > > > > > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main -
> Logging
> > > > error
> > > > > > messages to: /home/alex/hadoop/test/pig_1266319637282.log
> > > > > > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
> > > 2999:
> > > > > > Unexpected internal error. Undefined parameter : PIGDIR
> > > > > > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
> > > > > >
> > > > > > The log file:
> > > > > > alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> > > > > > Error before Pig is launched
> > > > > > ----------------------------
> > > > > > ERROR 2999: Unexpected internal error. Undefined parameter :
> PIGDIR
> > > > > >
> > > > > > java.lang.RuntimeException: Undefined parameter : PIGDIR
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> > > > > > at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> > > > > > at org.apache.pig.Main.main(Main.java:366)
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ================================================================================
> > > > > >
> > > > > >
> > > > > > The script: 'A = load '/home/alex/hadoop/test/t.csv' using
> > > > > PigStorage('\t')
> > > > > > as (id: long); '
> > > > > > The file: '1' .
> > > > > > Simple enough :)
> > > > > >
> > > > > > I'm all out of ideas. It seems that even 0.5 is broken now. I
> can't
> > > > start
> > > > > > anything as scripts. If I go with manual processing in grunt
> (line
> > by
> > > > > line)
> > > > > > it's all good on any version.
> > > > > >
> > > > > > thanks,
> > > > > > alex
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <
> > dvryaboy@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Pig starts, the error you are getting is from inside pig.Main .
> > > > > > > I think something is getting messed up in your environment
> > because
> > > > you
> > > > > > > are juggling too many different versions of pig (granted, I
> have
> > 3
> > > or
> > > > > > > 4 in various stages of development on my laptop most of the
> time,
> > > and
> > > > > > > haven't had your problems. But then neither of them has a
> hacked
> > > > > > > bin/pig , with the exception of the cloudera one...). I've
> never
> > > > tried
> > > > > > > running multiple ant tests, either. There's a short "sanity"
> > > version
> > > > > > > of tests, ant test-commit, that runs in under 10 minutes. You
> > might
> > > > > > > want to try that if you are not doing things like changing join
> > > > > > > implementations or messing with the optimizer.
> > > > > > >
> > > > > > > Let's do this: send a full trace of what you are doing and what
> > > your
> > > > > > > environment looks like. Something like
> > > > > > >
> > > > > > > pwd
> > > > > > > printenv | grep JAVA
> > > > > > > printenv | grep PIG
> > > > > > > printenv | grep HADOOP
> > > > > > > cat conf/pig.properties
> > > > > > > java -cp ........
> > > > > > > <boom!>
> > > > > > >
> > > > > > > -D
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > > > > > > <al...@gmail.com> wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > sorry for the delay, but I wanted to build from the source
> > again
> > > > just
> > > > > > to
> > > > > > > > make sure.
> > > > > > > >
> > > > > > > > The script is like this:
> > > > > > > >
> > > > > > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using
> > > PigStorage('\t')
> > > > as
> > > > > > > (id:
> > > > > > > > long, hid: long, locale: chararray, r1: int, r2: int, r3:
> int,
> > > r4:
> > > > > > int);
> > > > > > > '
> > > > > > > >
> > > > > > > > That's it. My guess is that Pig doesn't even start. Do you
> > think
> > > I
> > > > > need
> > > > > > > to
> > > > > > > > change something in the properties file? I'm not sure anymore
> > and
> > > I
> > > > > > don't
> > > > > > > > have a lot of luck going through the sources.
> > > > > > > >
> > > > > > > > And another thing, I tried running 'ant test' for both
> > 0.6-branch
> > > > and
> > > > > > > trunk
> > > > > > > > at the same time (because they take a very long time) and
> both
> > > test
> > > > > > > scripts
> > > > > > > > failed. I've switched to running them one after the other and
> > > they
> > > > > are
> > > > > > > fine.
> > > > > > > > Do you think that is ok?
> > > > > > > >
> > > > > > > > thanks for your time,
> > > > > > > > alex
> > > > > > > >
> > > > > > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
> > > > dvryaboy@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> what does your script1-hadoop.pig look like?
> > > > > > > >>
> > > > > > > >> The error you are getting happens when the pig preprocessor
> > > can't
> > > > > > > >> substitute Pig variables (the stuff you specify with -param
> > and
> > > > > > > >> %default, etc). Do you have $PIGDIR in your script
> somewhere?
> > > > > > > >>
> > > > > > > >> -D
> > > > > > > >>
> > > > > > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > > > > > > >> <al...@gmail.com> wrote:
> > > > > > > >> > Hello,
> > > > > > > >> >
> > > > > > > >> > I seem to have broken my Pig install, and I don't know
> where
> > > to
> > > > > > look.
> > > > > > > >> >
> > > > > > > >> > If I use directly the script (grunt) everything works ok,
> > but
> > > > > every
> > > > > > > time
> > > > > > > >> I
> > > > > > > >> > try to run a pig script: 'java -cp
> > > > > $PIGDIR/pig.jar:$HADOOPSITEPATH
> > > > > > > >> > org.apache.pig.Main script1-hadoop.pig'
> > > > > > > >> >
> > > > > > > >> > I get this nice error: [main] ERROR org.apache.pig.Main -
> > > ERROR
> > > > > > 2999:
> > > > > > > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > > > > > > >> >
> > > > > > > >> > Obviously I have the PIGDIR var set:
> > > > > > > >> >> echo $PIGDIR
> > > > > > > >> >> /home/alex/hadoop/pig
> > > > > > > >> >
> > > > > > > >> > This is something that I did, as I have used 0.5 and 0.6
> and
> > a
> > > > > > patched
> > > > > > > >> > version of 0.6 in parallel, but I can't figure out where
> to
> > > > look.
> > > > > > Any
> > > > > > > >> > version I try to start now, gives the same error.
> > > > > > > >> >
> > > > > > > >> > any help would be greatly appreciated!
> > > > > > > >> >
> > > > > > > >> > alex
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Alex Parvulescu <al...@gmail.com>.
Hello and sorry for the absence
I went through a hard drive crash, and a 2 day recovery process.
I have incredible news... Pig is working again!!!!!!!!!!
it felt good to finally say it out loud
I've found the problem. Check this out:
--------- This is my test script -----------
-- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
test_lite.pig
A = load 't.csv' using PigStorage('\t') as (id: long);
store A into 'output/pig' using PigStorage('\t');
--------------------
I just saved the run instruction in this script to I don't forget it.
Notice the '$PIGDIR' and '$HADOOPDIR'
It apprears this is the source if the evil
ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
replacing this with
-- $whatever
will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
whatever
Now that it's working again, I'll be back with more issues in no time :)
Thanks a million to everybody, especially Dmitriy!
alex
On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan <
ashutosh.chauhan@gmail.com> wrote:
> Alex,
>
> I just downloaded the latest Pig 0.6 Release Candidate from
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>and
> followed the instructions in "Trying the Release" section as outlined in
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>and
> without taking any extra step anywhere in between I am able to get Pig
> running. You may want to do the same thing. If it works that would be great
> else we will be sure that its some environment problem in your setup and we
> can take it from there.
>
> Ashutosh
>
> On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > Probably unrelated, but I always run from the root directory, not from
> > bin/,
> > when working with an svn version. Also, never had dev vs core issues. Can
> > you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> > pig.jar (not core, not dev) on $PIG_CLASSPATH?
> >
> > How about avoiding the pig script and just running
> >
> > java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
> > <<yourscript>> ?
> >
> > I grepped through all the code, including the codegenned stuff, and none
> of
> > it contains the string PIGDIR (except bin/pig of course).
> >
> > At this point, assuming none of the above helps (which I have little hope
> > it
> > will), the best I can say is set up your eclipse environment and step
> > through pig's main with the debugger.
> >
> > -D
> >
> > On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> > <al...@gmail.com>wrote:
> >
> > > Hello
> > >
> > > This is getting tiresome :)
> > >
> > > Today, I tried with a fresh download of pig 0.5 and I get the exact
> same
> > > error. Something somewhere just broke down, and it took my machine with
> > it.
> > > At this point I have no way to run any kind of pig version.
> > > Needless to say, this is a big problem for me :)
> > >
> > > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> > error.
> > > I even dropped the sym links.
> > >
> > > Back to our debugging session:
> > >
> > > > svn co
> > >
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > > A pig-branch-0.6-take2/test/bin/test-patch.sh
> > > Checked out external at revision 910901.
> > >
> > > Checked out revision 910900.
> > >
> > > > cd pig-branch-0.6-take2/
> > > > ant
> > > BUILD SUCCESSFUL
> > > Total time: 26 seconds
> > >
> > > > cd bin/
> > >
> > > next is to check if all vars are there
> > > > printenv | grep PIG
> > > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> > > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> > >
> > > > printenv | grep HADOOP
> > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > >
> > > > pig
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > jline/ConsoleReaderInputStream
> > > Caused by: java.lang.ClassNotFoundException:
> > jline.ConsoleReaderInputStream
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > > Could not find the main class: org.apache.pig.Main. Program will exit.
> > >
> > > I've sent some email to the list about this problem before. A quick fix
> > is
> > > to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > > > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
> > >
> > > > pig
> > > 2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging
> error
> > > messages to:
> > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> > > 2010-02-17 11:48:53,175 [main] INFO
> > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> > Connecting
> > > to hadoop file system at: file:///
> > > grunt>
> > >
> > > Now at least it will start.
> > >
> > > Getting to the initial issue:
> > >
> > > the '-f' option:
> > > > pig -f /home/alex/hadoop/test/test_lite.pig
> > > 2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging
> error
> > > messages to:
> > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> > > 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
> > > Unexpected internal error. Undefined parameter : PIGDIR
> > > Details at logfile:
> > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> > >
> > > the '- secretDebugCmd' option
> > >
> > > pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
> > > dry run:
> > > /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> > > -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
> > > -Dpig.log.file=pig.log
> > > -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
> > > -Dpig.root.logger=INFO,console,DRFA -classpath
> > >
> > >
> >
> /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
> > > org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig
> > >
> > > I did not change the scripts, and I get the same error.
> > >
> > > This is really killing me :)
> > >
> > > thanks,
> > > alex
> > >
> > >
> > >
> > > On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > > wrote:
> > >
> > > > Hm, nothing jumps out. Definitely the error you are getting indicates
> > > that
> > > > somehow the preprocessor is trying to substitute a variable called
> > PIGDIR
> > > > in
> > > > your script. Which is odd. Does the same thing happen if you try
> > bin/pig
> > > -f
> > > > test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's
> > > secret),
> > > > and sending along the output. Why are you using pig-core instead of
> > > pig.jar
> > > > that ant should be generating for you when you build? Where did you
> get
> > > it,
> > > > how did you build it, and what's the output of cksum on it?
> > > > The fact that now your 0.5 is broken too makes me think that maybe
> your
> > > > symlinks are messed up. Surely a change to one jar shouldn't be
> > affecting
> > > a
> > > > totally unrelated jar in a directory the first jar doesn't know
> about.
> > > >
> > > > Sorry about the barrage of questions, I am just a bit dumbfounded
> about
> > > how
> > > > this could even begin to start happening. Any preprocessor experts
> > > around?
> > > >
> > > > -D
> > > >
> > > >
> > > > On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
> > > > <al...@gmail.com>wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > And thanks again for all your help
> > > > >
> > > > > I have a symbolic link /home/alex/hadoop/pig which points to
> > > > > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build
> of
> > > the
> > > > > 0.6 branch.
> > > > > I'm running the script from /home/alex/hadoop/test.
> > > > >
> > > > > I didn't touch pig.properties, also I'm running with the default
> pig
> > > > > script.
> > > > >
> > > > > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> > > > > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> > > > > work(Exception
> > > > > in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/pig/Main).
> > > > >
> > > > > I'm running with a fresh 0.6 build and still no luck.
> > > > >
> > > > > The info you reqested:
> > > > >
> > > > > alex@alex-desktop:~/hadoop/test$ pwd
> > > > > /home/alex/hadoop/test
> > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> > > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> > > > > PIGDIR=/home/alex/hadoop/pig
> > > > > alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> > > > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > > > >
> > > > > alex@alex-desktop:~/hadoop/test$ cat
> > > > > /home/alex/hadoop/pig/conf/pig.properties
> > > > > # Pig configuration file. All values can be overwritten by command
> > line
> > > > > arguments.
> > > > > # see bin/pig -help
> > > > >
> > > > > # log4jconf log4j configuration file
> > > > > # log4jconf=./conf/log4j.properties
> > > > >
> > > > > # brief logging (no timestamps)
> > > > > brief=false
> > > > >
> > > > > # clustername, name of the hadoop jobtracker. If no port is defined
> > > port
> > > > > 50020 will be used.
> > > > > #cluster
> > > > >
> > > > > #debug level, INFO is default
> > > > > debug=INFO
> > > > >
> > > > > # a file that contains pig script
> > > > > #file=
> > > > >
> > > > > # load jarfile, colon separated
> > > > > #jar=
> > > > >
> > > > > #verbose print all log messages to screen (default to print only
> INFO
> > > and
> > > > > above to screen)
> > > > > verbose=false
> > > > >
> > > > > #exectype local|mapreduce, mapreduce is default
> > > > > #exectype=mapreduce
> > > > > # hod realted properties
> > > > > #ssh.gateway
> > > > > #hod.expect.root
> > > > > #hod.expect.uselatest
> > > > > #hod.command
> > > > > #hod.config.dir
> > > > > #hod.param
> > > > >
> > > > >
> > > > > #Do not spill temp files smaller than this size (bytes)
> > > > > pig.spill.size.threshold=5000000
> > > > > #EXPERIMENT: Activate garbage collection when spilling a file
> bigger
> > > than
> > > > > this size (bytes)
> > > > > #This should help reduce the number of files being spilled.
> > > > > pig.spill.gc.activation.size=40000000
> > > > >
> > > > >
> > > > > ######################
> > > > > # Everything below this line is Yahoo specific. Note that I've
> made
> > > > > # (almost) no changes to the lines above to make merging in from
> > Apache
> > > > > # easier. Any values I don't want from above I override below.
> > > > > #
> > > > > # This file is configured for use with HOD on the production
> > clusters.
> > > > If
> > > > > you
> > > > > # want to run pig with a static cluster you will need to remove
> > > > everything
> > > > > # below this line and set the cluster value (above) to the
> > > > > # hostname and port of your job tracker.
> > > > >
> > > > > exectype=mapreduce
> > > > >
> > > > > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> > > > > hod.server=local
> > > > >
> > > > > cluster.domain=inktomisearch.com
> > > > >
> > > > > log.file=
> > > > >
> > > > > yinst.cluster=kryptonite
> > > > >
> > > > >
> > > > >
> > > > > And now boom!
> > > > >
> > > > > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
> > org.apache.pig.Main
> > > > > test_lite.pig
> > > > >
> > > > >
> > > > > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
> > > error
> > > > > messages to: /home/alex/hadoop/test/pig_1266319637282.log
> > > > > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
> > 2999:
> > > > > Unexpected internal error. Undefined parameter : PIGDIR
> > > > > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
> > > > >
> > > > > The log file:
> > > > > alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> > > > > Error before Pig is launched
> > > > > ----------------------------
> > > > > ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> > > > >
> > > > > java.lang.RuntimeException: Undefined parameter : PIGDIR
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> > > > > at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> > > > > at org.apache.pig.Main.main(Main.java:366)
> > > > >
> > > > >
> > > >
> > >
> >
> ================================================================================
> > > > >
> > > > >
> > > > > The script: 'A = load '/home/alex/hadoop/test/t.csv' using
> > > > PigStorage('\t')
> > > > > as (id: long); '
> > > > > The file: '1' .
> > > > > Simple enough :)
> > > > >
> > > > > I'm all out of ideas. It seems that even 0.5 is broken now. I can't
> > > start
> > > > > anything as scripts. If I go with manual processing in grunt (line
> by
> > > > line)
> > > > > it's all good on any version.
> > > > >
> > > > > thanks,
> > > > > alex
> > > > >
> > > > >
> > > > > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <
> dvryaboy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Pig starts, the error you are getting is from inside pig.Main .
> > > > > > I think something is getting messed up in your environment
> because
> > > you
> > > > > > are juggling too many different versions of pig (granted, I have
> 3
> > or
> > > > > > 4 in various stages of development on my laptop most of the time,
> > and
> > > > > > haven't had your problems. But then neither of them has a hacked
> > > > > > bin/pig , with the exception of the cloudera one...). I've never
> > > tried
> > > > > > running multiple ant tests, either. There's a short "sanity"
> > version
> > > > > > of tests, ant test-commit, that runs in under 10 minutes. You
> might
> > > > > > want to try that if you are not doing things like changing join
> > > > > > implementations or messing with the optimizer.
> > > > > >
> > > > > > Let's do this: send a full trace of what you are doing and what
> > your
> > > > > > environment looks like. Something like
> > > > > >
> > > > > > pwd
> > > > > > printenv | grep JAVA
> > > > > > printenv | grep PIG
> > > > > > printenv | grep HADOOP
> > > > > > cat conf/pig.properties
> > > > > > java -cp ........
> > > > > > <boom!>
> > > > > >
> > > > > > -D
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > > > > > <al...@gmail.com> wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > sorry for the delay, but I wanted to build from the source
> again
> > > just
> > > > > to
> > > > > > > make sure.
> > > > > > >
> > > > > > > The script is like this:
> > > > > > >
> > > > > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using
> > PigStorage('\t')
> > > as
> > > > > > (id:
> > > > > > > long, hid: long, locale: chararray, r1: int, r2: int, r3: int,
> > r4:
> > > > > int);
> > > > > > '
> > > > > > >
> > > > > > > That's it. My guess is that Pig doesn't even start. Do you
> think
> > I
> > > > need
> > > > > > to
> > > > > > > change something in the properties file? I'm not sure anymore
> and
> > I
> > > > > don't
> > > > > > > have a lot of luck going through the sources.
> > > > > > >
> > > > > > > And another thing, I tried running 'ant test' for both
> 0.6-branch
> > > and
> > > > > > trunk
> > > > > > > at the same time (because they take a very long time) and both
> > test
> > > > > > scripts
> > > > > > > failed. I've switched to running them one after the other and
> > they
> > > > are
> > > > > > fine.
> > > > > > > Do you think that is ok?
> > > > > > >
> > > > > > > thanks for your time,
> > > > > > > alex
> > > > > > >
> > > > > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
> > > dvryaboy@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> what does your script1-hadoop.pig look like?
> > > > > > >>
> > > > > > >> The error you are getting happens when the pig preprocessor
> > can't
> > > > > > >> substitute Pig variables (the stuff you specify with -param
> and
> > > > > > >> %default, etc). Do you have $PIGDIR in your script somewhere?
> > > > > > >>
> > > > > > >> -D
> > > > > > >>
> > > > > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > > > > > >> <al...@gmail.com> wrote:
> > > > > > >> > Hello,
> > > > > > >> >
> > > > > > >> > I seem to have broken my Pig install, and I don't know where
> > to
> > > > > look.
> > > > > > >> >
> > > > > > >> > If I use directly the script (grunt) everything works ok,
> but
> > > > every
> > > > > > time
> > > > > > >> I
> > > > > > >> > try to run a pig script: 'java -cp
> > > > $PIGDIR/pig.jar:$HADOOPSITEPATH
> > > > > > >> > org.apache.pig.Main script1-hadoop.pig'
> > > > > > >> >
> > > > > > >> > I get this nice error: [main] ERROR org.apache.pig.Main -
> > ERROR
> > > > > 2999:
> > > > > > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > > > > > >> >
> > > > > > >> > Obviously I have the PIGDIR var set:
> > > > > > >> >> echo $PIGDIR
> > > > > > >> >> /home/alex/hadoop/pig
> > > > > > >> >
> > > > > > >> > This is something that I did, as I have used 0.5 and 0.6 and
> a
> > > > > patched
> > > > > > >> > version of 0.6 in parallel, but I can't figure out where to
> > > look.
> > > > > Any
> > > > > > >> > version I try to start now, gives the same error.
> > > > > > >> >
> > > > > > >> > any help would be greatly appreciated!
> > > > > > >> >
> > > > > > >> > alex
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Ashutosh Chauhan <as...@gmail.com>.
Alex,
I just downloaded the latest Pig 0.6 Release Candidate from
http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz and
followed the instructions in "Trying the Release" section as outlined in
http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt and
without taking any extra step anywhere in between I am able to get Pig
running. You may want to do the same thing. If it works that would be great
else we will be sure that its some environment problem in your setup and we
can take it from there.
Ashutosh
On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Probably unrelated, but I always run from the root directory, not from
> bin/,
> when working with an svn version. Also, never had dev vs core issues. Can
> you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
> pig.jar (not core, not dev) on $PIG_CLASSPATH?
>
> How about avoiding the pig script and just running
>
> java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
> <<yourscript>> ?
>
> I grepped through all the code, including the codegenned stuff, and none of
> it contains the string PIGDIR (except bin/pig of course).
>
> At this point, assuming none of the above helps (which I have little hope
> it
> will), the best I can say is set up your eclipse environment and step
> through pig's main with the debugger.
>
> -D
>
> On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
> <al...@gmail.com>wrote:
>
> > Hello
> >
> > This is getting tiresome :)
> >
> > Today, I tried with a fresh download of pig 0.5 and I get the exact same
> > error. Something somewhere just broke down, and it took my machine with
> it.
> > At this point I have no way to run any kind of pig version.
> > Needless to say, this is a big problem for me :)
> >
> > Another thing I tried is to do a fresh build of the 0.6 branch. Same
> error.
> > I even dropped the sym links.
> >
> > Back to our debugging session:
> >
> > > svn co
> >
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> > A pig-branch-0.6-take2/test/bin/test-patch.sh
> > Checked out external at revision 910901.
> >
> > Checked out revision 910900.
> >
> > > cd pig-branch-0.6-take2/
> > > ant
> > BUILD SUCCESSFUL
> > Total time: 26 seconds
> >
> > > cd bin/
> >
> > next is to check if all vars are there
> > > printenv | grep PIG
> > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> >
> > > printenv | grep HADOOP
> > HADOOP_HOME=/home/alex/hadoop/hadoop
> > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> >
> > > pig
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > jline/ConsoleReaderInputStream
> > Caused by: java.lang.ClassNotFoundException:
> jline.ConsoleReaderInputStream
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > Could not find the main class: org.apache.pig.Main. Program will exit.
> >
> > I've sent some email to the list about this problem before. A quick fix
> is
> > to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
> >
> > > pig
> > 2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
> > messages to:
> > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> > 2010-02-17 11:48:53,175 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> > to hadoop file system at: file:///
> > grunt>
> >
> > Now at least it will start.
> >
> > Getting to the initial issue:
> >
> > the '-f' option:
> > > pig -f /home/alex/hadoop/test/test_lite.pig
> > 2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
> > messages to:
> > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> > 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
> > Unexpected internal error. Undefined parameter : PIGDIR
> > Details at logfile:
> > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> >
> > the '- secretDebugCmd' option
> >
> > pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
> > dry run:
> > /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> > -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
> > -Dpig.log.file=pig.log
> > -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
> > -Dpig.root.logger=INFO,console,DRFA -classpath
> >
> >
> /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
> > org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig
> >
> > I did not change the scripts, and I get the same error.
> >
> > This is really killing me :)
> >
> > thanks,
> > alex
> >
> >
> >
> > On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> >
> > > Hm, nothing jumps out. Definitely the error you are getting indicates
> > that
> > > somehow the preprocessor is trying to substitute a variable called
> PIGDIR
> > > in
> > > your script. Which is odd. Does the same thing happen if you try
> bin/pig
> > -f
> > > test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's
> > secret),
> > > and sending along the output. Why are you using pig-core instead of
> > pig.jar
> > > that ant should be generating for you when you build? Where did you get
> > it,
> > > how did you build it, and what's the output of cksum on it?
> > > The fact that now your 0.5 is broken too makes me think that maybe your
> > > symlinks are messed up. Surely a change to one jar shouldn't be
> affecting
> > a
> > > totally unrelated jar in a directory the first jar doesn't know about.
> > >
> > > Sorry about the barrage of questions, I am just a bit dumbfounded about
> > how
> > > this could even begin to start happening. Any preprocessor experts
> > around?
> > >
> > > -D
> > >
> > >
> > > On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
> > > <al...@gmail.com>wrote:
> > >
> > > > Hello
> > > >
> > > > And thanks again for all your help
> > > >
> > > > I have a symbolic link /home/alex/hadoop/pig which points to
> > > > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of
> > the
> > > > 0.6 branch.
> > > > I'm running the script from /home/alex/hadoop/test.
> > > >
> > > > I didn't touch pig.properties, also I'm running with the default pig
> > > > script.
> > > >
> > > > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> > > > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> > > > work(Exception
> > > > in thread "main" java.lang.NoClassDefFoundError:
> org/apache/pig/Main).
> > > >
> > > > I'm running with a fresh 0.6 build and still no luck.
> > > >
> > > > The info you reqested:
> > > >
> > > > alex@alex-desktop:~/hadoop/test$ pwd
> > > > /home/alex/hadoop/test
> > > > alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> > > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > > alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> > > > PIGDIR=/home/alex/hadoop/pig
> > > > alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> > > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > > >
> > > > alex@alex-desktop:~/hadoop/test$ cat
> > > > /home/alex/hadoop/pig/conf/pig.properties
> > > > # Pig configuration file. All values can be overwritten by command
> line
> > > > arguments.
> > > > # see bin/pig -help
> > > >
> > > > # log4jconf log4j configuration file
> > > > # log4jconf=./conf/log4j.properties
> > > >
> > > > # brief logging (no timestamps)
> > > > brief=false
> > > >
> > > > # clustername, name of the hadoop jobtracker. If no port is defined
> > port
> > > > 50020 will be used.
> > > > #cluster
> > > >
> > > > #debug level, INFO is default
> > > > debug=INFO
> > > >
> > > > # a file that contains pig script
> > > > #file=
> > > >
> > > > # load jarfile, colon separated
> > > > #jar=
> > > >
> > > > #verbose print all log messages to screen (default to print only INFO
> > and
> > > > above to screen)
> > > > verbose=false
> > > >
> > > > #exectype local|mapreduce, mapreduce is default
> > > > #exectype=mapreduce
> > > > # hod realted properties
> > > > #ssh.gateway
> > > > #hod.expect.root
> > > > #hod.expect.uselatest
> > > > #hod.command
> > > > #hod.config.dir
> > > > #hod.param
> > > >
> > > >
> > > > #Do not spill temp files smaller than this size (bytes)
> > > > pig.spill.size.threshold=5000000
> > > > #EXPERIMENT: Activate garbage collection when spilling a file bigger
> > than
> > > > this size (bytes)
> > > > #This should help reduce the number of files being spilled.
> > > > pig.spill.gc.activation.size=40000000
> > > >
> > > >
> > > > ######################
> > > > # Everything below this line is Yahoo specific. Note that I've made
> > > > # (almost) no changes to the lines above to make merging in from
> Apache
> > > > # easier. Any values I don't want from above I override below.
> > > > #
> > > > # This file is configured for use with HOD on the production
> clusters.
> > > If
> > > > you
> > > > # want to run pig with a static cluster you will need to remove
> > > everything
> > > > # below this line and set the cluster value (above) to the
> > > > # hostname and port of your job tracker.
> > > >
> > > > exectype=mapreduce
> > > >
> > > > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> > > > hod.server=local
> > > >
> > > > cluster.domain=inktomisearch.com
> > > >
> > > > log.file=
> > > >
> > > > yinst.cluster=kryptonite
> > > >
> > > >
> > > >
> > > > And now boom!
> > > >
> > > > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
> org.apache.pig.Main
> > > > test_lite.pig
> > > >
> > > >
> > > > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
> > error
> > > > messages to: /home/alex/hadoop/test/pig_1266319637282.log
> > > > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
> 2999:
> > > > Unexpected internal error. Undefined parameter : PIGDIR
> > > > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
> > > >
> > > > The log file:
> > > > alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> > > > Error before Pig is launched
> > > > ----------------------------
> > > > ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> > > >
> > > > java.lang.RuntimeException: Undefined parameter : PIGDIR
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> > > > at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> > > > at org.apache.pig.Main.main(Main.java:366)
> > > >
> > > >
> > >
> >
> ================================================================================
> > > >
> > > >
> > > > The script: 'A = load '/home/alex/hadoop/test/t.csv' using
> > > PigStorage('\t')
> > > > as (id: long); '
> > > > The file: '1' .
> > > > Simple enough :)
> > > >
> > > > I'm all out of ideas. It seems that even 0.5 is broken now. I can't
> > start
> > > > anything as scripts. If I go with manual processing in grunt (line by
> > > line)
> > > > it's all good on any version.
> > > >
> > > > thanks,
> > > > alex
> > > >
> > > >
> > > > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dv...@gmail.com>
> > > > wrote:
> > > >
> > > > > Pig starts, the error you are getting is from inside pig.Main .
> > > > > I think something is getting messed up in your environment because
> > you
> > > > > are juggling too many different versions of pig (granted, I have 3
> or
> > > > > 4 in various stages of development on my laptop most of the time,
> and
> > > > > haven't had your problems. But then neither of them has a hacked
> > > > > bin/pig , with the exception of the cloudera one...). I've never
> > tried
> > > > > running multiple ant tests, either. There's a short "sanity"
> version
> > > > > of tests, ant test-commit, that runs in under 10 minutes. You might
> > > > > want to try that if you are not doing things like changing join
> > > > > implementations or messing with the optimizer.
> > > > >
> > > > > Let's do this: send a full trace of what you are doing and what
> your
> > > > > environment looks like. Something like
> > > > >
> > > > > pwd
> > > > > printenv | grep JAVA
> > > > > printenv | grep PIG
> > > > > printenv | grep HADOOP
> > > > > cat conf/pig.properties
> > > > > java -cp ........
> > > > > <boom!>
> > > > >
> > > > > -D
> > > > >
> > > > >
> > > > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > > > > <al...@gmail.com> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > sorry for the delay, but I wanted to build from the source again
> > just
> > > > to
> > > > > > make sure.
> > > > > >
> > > > > > The script is like this:
> > > > > >
> > > > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using
> PigStorage('\t')
> > as
> > > > > (id:
> > > > > > long, hid: long, locale: chararray, r1: int, r2: int, r3: int,
> r4:
> > > > int);
> > > > > '
> > > > > >
> > > > > > That's it. My guess is that Pig doesn't even start. Do you think
> I
> > > need
> > > > > to
> > > > > > change something in the properties file? I'm not sure anymore and
> I
> > > > don't
> > > > > > have a lot of luck going through the sources.
> > > > > >
> > > > > > And another thing, I tried running 'ant test' for both 0.6-branch
> > and
> > > > > trunk
> > > > > > at the same time (because they take a very long time) and both
> test
> > > > > scripts
> > > > > > failed. I've switched to running them one after the other and
> they
> > > are
> > > > > fine.
> > > > > > Do you think that is ok?
> > > > > >
> > > > > > thanks for your time,
> > > > > > alex
> > > > > >
> > > > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
> > dvryaboy@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> what does your script1-hadoop.pig look like?
> > > > > >>
> > > > > >> The error you are getting happens when the pig preprocessor
> can't
> > > > > >> substitute Pig variables (the stuff you specify with -param and
> > > > > >> %default, etc). Do you have $PIGDIR in your script somewhere?
> > > > > >>
> > > > > >> -D
> > > > > >>
> > > > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > > > > >> <al...@gmail.com> wrote:
> > > > > >> > Hello,
> > > > > >> >
> > > > > >> > I seem to have broken my Pig install, and I don't know where
> to
> > > > look.
> > > > > >> >
> > > > > >> > If I use directly the script (grunt) everything works ok, but
> > > every
> > > > > time
> > > > > >> I
> > > > > >> > try to run a pig script: 'java -cp
> > > $PIGDIR/pig.jar:$HADOOPSITEPATH
> > > > > >> > org.apache.pig.Main script1-hadoop.pig'
> > > > > >> >
> > > > > >> > I get this nice error: [main] ERROR org.apache.pig.Main -
> ERROR
> > > > 2999:
> > > > > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > > > > >> >
> > > > > >> > Obviously I have the PIGDIR var set:
> > > > > >> >> echo $PIGDIR
> > > > > >> >> /home/alex/hadoop/pig
> > > > > >> >
> > > > > >> > This is something that I did, as I have used 0.5 and 0.6 and a
> > > > patched
> > > > > >> > version of 0.6 in parallel, but I can't figure out where to
> > look.
> > > > Any
> > > > > >> > version I try to start now, gives the same error.
> > > > > >> >
> > > > > >> > any help would be greatly appreciated!
> > > > > >> >
> > > > > >> > alex
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Probably unrelated, but I always run from the root directory, not from bin/,
when working with an svn version. Also, never had dev vs core issues. Can
you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
pig.jar (not core, not dev) on $PIG_CLASSPATH?
How about avoiding the pig script and just running
java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
<<yourscript>> ?
I grepped through all the code, including the codegenned stuff, and none of
it contains the string PIGDIR (except bin/pig of course).
At this point, assuming none of the above helps (which I have little hope it
will), the best I can say is set up your eclipse environment and step
through pig's main with the debugger.
-D
On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
<al...@gmail.com>wrote:
> Hello
>
> This is getting tiresome :)
>
> Today, I tried with a fresh download of pig 0.5 and I get the exact same
> error. Something somewhere just broke down, and it took my machine with it.
> At this point I have no way to run any kind of pig version.
> Needless to say, this is a big problem for me :)
>
> Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
> I even dropped the sym links.
>
> Back to our debugging session:
>
> > svn co
> http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
> A pig-branch-0.6-take2/test/bin/test-patch.sh
> Checked out external at revision 910901.
>
> Checked out revision 910900.
>
> > cd pig-branch-0.6-take2/
> > ant
> BUILD SUCCESSFUL
> Total time: 26 seconds
>
> > cd bin/
>
> next is to check if all vars are there
> > printenv | grep PIG
> PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
> PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
>
> > printenv | grep HADOOP
> HADOOP_HOME=/home/alex/hadoop/hadoop
> HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> HADOOPDIR=/home/alex/hadoop/hadoop/conf
>
> > pig
> Exception in thread "main" java.lang.NoClassDefFoundError:
> jline/ConsoleReaderInputStream
> Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> Could not find the main class: org.apache.pig.Main. Program will exit.
>
> I've sent some email to the list about this problem before. A quick fix is
> to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
>
> > pig
> 2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
> messages to:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
> 2010-02-17 11:48:53,175 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt>
>
> Now at least it will start.
>
> Getting to the initial issue:
>
> the '-f' option:
> > pig -f /home/alex/hadoop/test/test_lite.pig
> 2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
> messages to:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
> 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. Undefined parameter : PIGDIR
> Details at logfile:
> /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
>
> the '- secretDebugCmd' option
>
> pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
> dry run:
> /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
> -Dpig.log.file=pig.log
> -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
> -Dpig.root.logger=INFO,console,DRFA -classpath
>
> /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
> org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig
>
> I did not change the scripts, and I get the same error.
>
> This is really killing me :)
>
> thanks,
> alex
>
>
>
> On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
>
> > Hm, nothing jumps out. Definitely the error you are getting indicates
> that
> > somehow the preprocessor is trying to substitute a variable called PIGDIR
> > in
> > your script. Which is odd. Does the same thing happen if you try bin/pig
> -f
> > test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's
> secret),
> > and sending along the output. Why are you using pig-core instead of
> pig.jar
> > that ant should be generating for you when you build? Where did you get
> it,
> > how did you build it, and what's the output of cksum on it?
> > The fact that now your 0.5 is broken too makes me think that maybe your
> > symlinks are messed up. Surely a change to one jar shouldn't be affecting
> a
> > totally unrelated jar in a directory the first jar doesn't know about.
> >
> > Sorry about the barrage of questions, I am just a bit dumbfounded about
> how
> > this could even begin to start happening. Any preprocessor experts
> around?
> >
> > -D
> >
> >
> > On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
> > <al...@gmail.com>wrote:
> >
> > > Hello
> > >
> > > And thanks again for all your help
> > >
> > > I have a symbolic link /home/alex/hadoop/pig which points to
> > > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of
> the
> > > 0.6 branch.
> > > I'm running the script from /home/alex/hadoop/test.
> > >
> > > I didn't touch pig.properties, also I'm running with the default pig
> > > script.
> > >
> > > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> > > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> > > work(Exception
> > > in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).
> > >
> > > I'm running with a fresh 0.6 build and still no luck.
> > >
> > > The info you reqested:
> > >
> > > alex@alex-desktop:~/hadoop/test$ pwd
> > > /home/alex/hadoop/test
> > > alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> > > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > > alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> > > PIGDIR=/home/alex/hadoop/pig
> > > alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> > > HADOOP_HOME=/home/alex/hadoop/hadoop
> > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> > >
> > > alex@alex-desktop:~/hadoop/test$ cat
> > > /home/alex/hadoop/pig/conf/pig.properties
> > > # Pig configuration file. All values can be overwritten by command line
> > > arguments.
> > > # see bin/pig -help
> > >
> > > # log4jconf log4j configuration file
> > > # log4jconf=./conf/log4j.properties
> > >
> > > # brief logging (no timestamps)
> > > brief=false
> > >
> > > # clustername, name of the hadoop jobtracker. If no port is defined
> port
> > > 50020 will be used.
> > > #cluster
> > >
> > > #debug level, INFO is default
> > > debug=INFO
> > >
> > > # a file that contains pig script
> > > #file=
> > >
> > > # load jarfile, colon separated
> > > #jar=
> > >
> > > #verbose print all log messages to screen (default to print only INFO
> and
> > > above to screen)
> > > verbose=false
> > >
> > > #exectype local|mapreduce, mapreduce is default
> > > #exectype=mapreduce
> > > # hod realted properties
> > > #ssh.gateway
> > > #hod.expect.root
> > > #hod.expect.uselatest
> > > #hod.command
> > > #hod.config.dir
> > > #hod.param
> > >
> > >
> > > #Do not spill temp files smaller than this size (bytes)
> > > pig.spill.size.threshold=5000000
> > > #EXPERIMENT: Activate garbage collection when spilling a file bigger
> than
> > > this size (bytes)
> > > #This should help reduce the number of files being spilled.
> > > pig.spill.gc.activation.size=40000000
> > >
> > >
> > > ######################
> > > # Everything below this line is Yahoo specific. Note that I've made
> > > # (almost) no changes to the lines above to make merging in from Apache
> > > # easier. Any values I don't want from above I override below.
> > > #
> > > # This file is configured for use with HOD on the production clusters.
> > If
> > > you
> > > # want to run pig with a static cluster you will need to remove
> > everything
> > > # below this line and set the cluster value (above) to the
> > > # hostname and port of your job tracker.
> > >
> > > exectype=mapreduce
> > >
> > > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> > > hod.server=local
> > >
> > > cluster.domain=inktomisearch.com
> > >
> > > log.file=
> > >
> > > yinst.cluster=kryptonite
> > >
> > >
> > >
> > > And now boom!
> > >
> > > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
> > > test_lite.pig
> > >
> > >
> > > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
> error
> > > messages to: /home/alex/hadoop/test/pig_1266319637282.log
> > > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
> > > Unexpected internal error. Undefined parameter : PIGDIR
> > > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
> > >
> > > The log file:
> > > alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> > > Error before Pig is launched
> > > ----------------------------
> > > ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> > >
> > > java.lang.RuntimeException: Undefined parameter : PIGDIR
> > > at
> > >
> > >
> >
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> > > at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> > > at org.apache.pig.Main.main(Main.java:366)
> > >
> > >
> >
> ================================================================================
> > >
> > >
> > > The script: 'A = load '/home/alex/hadoop/test/t.csv' using
> > PigStorage('\t')
> > > as (id: long); '
> > > The file: '1' .
> > > Simple enough :)
> > >
> > > I'm all out of ideas. It seems that even 0.5 is broken now. I can't
> start
> > > anything as scripts. If I go with manual processing in grunt (line by
> > line)
> > > it's all good on any version.
> > >
> > > thanks,
> > > alex
> > >
> > >
> > > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dv...@gmail.com>
> > > wrote:
> > >
> > > > Pig starts, the error you are getting is from inside pig.Main .
> > > > I think something is getting messed up in your environment because
> you
> > > > are juggling too many different versions of pig (granted, I have 3 or
> > > > 4 in various stages of development on my laptop most of the time, and
> > > > haven't had your problems. But then neither of them has a hacked
> > > > bin/pig , with the exception of the cloudera one...). I've never
> tried
> > > > running multiple ant tests, either. There's a short "sanity" version
> > > > of tests, ant test-commit, that runs in under 10 minutes. You might
> > > > want to try that if you are not doing things like changing join
> > > > implementations or messing with the optimizer.
> > > >
> > > > Let's do this: send a full trace of what you are doing and what your
> > > > environment looks like. Something like
> > > >
> > > > pwd
> > > > printenv | grep JAVA
> > > > printenv | grep PIG
> > > > printenv | grep HADOOP
> > > > cat conf/pig.properties
> > > > java -cp ........
> > > > <boom!>
> > > >
> > > > -D
> > > >
> > > >
> > > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > > > <al...@gmail.com> wrote:
> > > > > Hello,
> > > > >
> > > > > sorry for the delay, but I wanted to build from the source again
> just
> > > to
> > > > > make sure.
> > > > >
> > > > > The script is like this:
> > > > >
> > > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t')
> as
> > > > (id:
> > > > > long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
> > > int);
> > > > '
> > > > >
> > > > > That's it. My guess is that Pig doesn't even start. Do you think I
> > need
> > > > to
> > > > > change something in the properties file? I'm not sure anymore and I
> > > don't
> > > > > have a lot of luck going through the sources.
> > > > >
> > > > > And another thing, I tried running 'ant test' for both 0.6-branch
> and
> > > > trunk
> > > > > at the same time (because they take a very long time) and both test
> > > > scripts
> > > > > failed. I've switched to running them one after the other and they
> > are
> > > > fine.
> > > > > Do you think that is ok?
> > > > >
> > > > > thanks for your time,
> > > > > alex
> > > > >
> > > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
> dvryaboy@gmail.com>
> > > > wrote:
> > > > >
> > > > >> what does your script1-hadoop.pig look like?
> > > > >>
> > > > >> The error you are getting happens when the pig preprocessor can't
> > > > >> substitute Pig variables (the stuff you specify with -param and
> > > > >> %default, etc). Do you have $PIGDIR in your script somewhere?
> > > > >>
> > > > >> -D
> > > > >>
> > > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > > > >> <al...@gmail.com> wrote:
> > > > >> > Hello,
> > > > >> >
> > > > >> > I seem to have broken my Pig install, and I don't know where to
> > > look.
> > > > >> >
> > > > >> > If I use directly the script (grunt) everything works ok, but
> > every
> > > > time
> > > > >> I
> > > > >> > try to run a pig script: 'java -cp
> > $PIGDIR/pig.jar:$HADOOPSITEPATH
> > > > >> > org.apache.pig.Main script1-hadoop.pig'
> > > > >> >
> > > > >> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
> > > 2999:
> > > > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > > > >> >
> > > > >> > Obviously I have the PIGDIR var set:
> > > > >> >> echo $PIGDIR
> > > > >> >> /home/alex/hadoop/pig
> > > > >> >
> > > > >> > This is something that I did, as I have used 0.5 and 0.6 and a
> > > patched
> > > > >> > version of 0.6 in parallel, but I can't figure out where to
> look.
> > > Any
> > > > >> > version I try to start now, gives the same error.
> > > > >> >
> > > > >> > any help would be greatly appreciated!
> > > > >> >
> > > > >> > alex
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Alex Parvulescu <al...@gmail.com>.
Hello
This is getting tiresome :)
Today, I tried with a fresh download of pig 0.5 and I get the exact same
error. Something somewhere just broke down, and it took my machine with it.
At this point I have no way to run any kind of pig version.
Needless to say, this is a big problem for me :)
Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
I even dropped the sym links.
Back to our debugging session:
> svn co http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
Fetching external item into 'pig-branch-0.6-take2/test/bin'
A pig-branch-0.6-take2/test/bin/test-patch.sh
Checked out external at revision 910901.
Checked out revision 910900.
> cd pig-branch-0.6-take2/
> ant
BUILD SUCCESSFUL
Total time: 26 seconds
> cd bin/
next is to check if all vars are there
> printenv | grep PIG
PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
> printenv | grep HADOOP
HADOOP_HOME=/home/alex/hadoop/hadoop
HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
HADOOPDIR=/home/alex/hadoop/hadoop/conf
> pig
Exception in thread "main" java.lang.NoClassDefFoundError:
jline/ConsoleReaderInputStream
Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: org.apache.pig.Main. Program will exit.
I've sent some email to the list about this problem before. A quick fix is
to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
> cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
> pig
2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
messages to:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
2010-02-17 11:48:53,175 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt>
Now at least it will start.
Getting to the initial issue:
the '-f' option:
> pig -f /home/alex/hadoop/test/test_lite.pig
2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
messages to:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR
Details at logfile:
/home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
the '- secretDebugCmd' option
pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
dry run:
/usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
-Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
-Dpig.log.file=pig.log
-Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
-Dpig.root.logger=INFO,console,DRFA -classpath
/home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig
I did not change the scripts, and I get the same error.
This is really killing me :)
thanks,
alex
On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Hm, nothing jumps out. Definitely the error you are getting indicates that
> somehow the preprocessor is trying to substitute a variable called PIGDIR
> in
> your script. Which is odd. Does the same thing happen if you try bin/pig -f
> test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
> and sending along the output. Why are you using pig-core instead of pig.jar
> that ant should be generating for you when you build? Where did you get it,
> how did you build it, and what's the output of cksum on it?
> The fact that now your 0.5 is broken too makes me think that maybe your
> symlinks are messed up. Surely a change to one jar shouldn't be affecting a
> totally unrelated jar in a directory the first jar doesn't know about.
>
> Sorry about the barrage of questions, I am just a bit dumbfounded about how
> this could even begin to start happening. Any preprocessor experts around?
>
> -D
>
>
> On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
> <al...@gmail.com>wrote:
>
> > Hello
> >
> > And thanks again for all your help
> >
> > I have a symbolic link /home/alex/hadoop/pig which points to
> > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
> > 0.6 branch.
> > I'm running the script from /home/alex/hadoop/test.
> >
> > I didn't touch pig.properties, also I'm running with the default pig
> > script.
> >
> > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> > work(Exception
> > in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).
> >
> > I'm running with a fresh 0.6 build and still no luck.
> >
> > The info you reqested:
> >
> > alex@alex-desktop:~/hadoop/test$ pwd
> > /home/alex/hadoop/test
> > alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> > JAVA_HOME=/usr/lib/jvm/java-6-sun
> > alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> > PIGDIR=/home/alex/hadoop/pig
> > alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> > HADOOP_HOME=/home/alex/hadoop/hadoop
> > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> > HADOOPDIR=/home/alex/hadoop/hadoop/conf
> >
> > alex@alex-desktop:~/hadoop/test$ cat
> > /home/alex/hadoop/pig/conf/pig.properties
> > # Pig configuration file. All values can be overwritten by command line
> > arguments.
> > # see bin/pig -help
> >
> > # log4jconf log4j configuration file
> > # log4jconf=./conf/log4j.properties
> >
> > # brief logging (no timestamps)
> > brief=false
> >
> > # clustername, name of the hadoop jobtracker. If no port is defined port
> > 50020 will be used.
> > #cluster
> >
> > #debug level, INFO is default
> > debug=INFO
> >
> > # a file that contains pig script
> > #file=
> >
> > # load jarfile, colon separated
> > #jar=
> >
> > #verbose print all log messages to screen (default to print only INFO and
> > above to screen)
> > verbose=false
> >
> > #exectype local|mapreduce, mapreduce is default
> > #exectype=mapreduce
> > # hod realted properties
> > #ssh.gateway
> > #hod.expect.root
> > #hod.expect.uselatest
> > #hod.command
> > #hod.config.dir
> > #hod.param
> >
> >
> > #Do not spill temp files smaller than this size (bytes)
> > pig.spill.size.threshold=5000000
> > #EXPERIMENT: Activate garbage collection when spilling a file bigger than
> > this size (bytes)
> > #This should help reduce the number of files being spilled.
> > pig.spill.gc.activation.size=40000000
> >
> >
> > ######################
> > # Everything below this line is Yahoo specific. Note that I've made
> > # (almost) no changes to the lines above to make merging in from Apache
> > # easier. Any values I don't want from above I override below.
> > #
> > # This file is configured for use with HOD on the production clusters.
> If
> > you
> > # want to run pig with a static cluster you will need to remove
> everything
> > # below this line and set the cluster value (above) to the
> > # hostname and port of your job tracker.
> >
> > exectype=mapreduce
> >
> > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> > hod.server=local
> >
> > cluster.domain=inktomisearch.com
> >
> > log.file=
> >
> > yinst.cluster=kryptonite
> >
> >
> >
> > And now boom!
> >
> > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
> > test_lite.pig
> >
> >
> > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
> > messages to: /home/alex/hadoop/test/pig_1266319637282.log
> > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
> > Unexpected internal error. Undefined parameter : PIGDIR
> > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
> >
> > The log file:
> > alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> > Error before Pig is launched
> > ----------------------------
> > ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
> >
> > java.lang.RuntimeException: Undefined parameter : PIGDIR
> > at
> >
> >
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> > at
> >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> > at
> >
> >
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> > at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> > at org.apache.pig.Main.main(Main.java:366)
> >
> >
> ================================================================================
> >
> >
> > The script: 'A = load '/home/alex/hadoop/test/t.csv' using
> PigStorage('\t')
> > as (id: long); '
> > The file: '1' .
> > Simple enough :)
> >
> > I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
> > anything as scripts. If I go with manual processing in grunt (line by
> line)
> > it's all good on any version.
> >
> > thanks,
> > alex
> >
> >
> > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> >
> > > Pig starts, the error you are getting is from inside pig.Main .
> > > I think something is getting messed up in your environment because you
> > > are juggling too many different versions of pig (granted, I have 3 or
> > > 4 in various stages of development on my laptop most of the time, and
> > > haven't had your problems. But then neither of them has a hacked
> > > bin/pig , with the exception of the cloudera one...). I've never tried
> > > running multiple ant tests, either. There's a short "sanity" version
> > > of tests, ant test-commit, that runs in under 10 minutes. You might
> > > want to try that if you are not doing things like changing join
> > > implementations or messing with the optimizer.
> > >
> > > Let's do this: send a full trace of what you are doing and what your
> > > environment looks like. Something like
> > >
> > > pwd
> > > printenv | grep JAVA
> > > printenv | grep PIG
> > > printenv | grep HADOOP
> > > cat conf/pig.properties
> > > java -cp ........
> > > <boom!>
> > >
> > > -D
> > >
> > >
> > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > > <al...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > sorry for the delay, but I wanted to build from the source again just
> > to
> > > > make sure.
> > > >
> > > > The script is like this:
> > > >
> > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as
> > > (id:
> > > > long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
> > int);
> > > '
> > > >
> > > > That's it. My guess is that Pig doesn't even start. Do you think I
> need
> > > to
> > > > change something in the properties file? I'm not sure anymore and I
> > don't
> > > > have a lot of luck going through the sources.
> > > >
> > > > And another thing, I tried running 'ant test' for both 0.6-branch and
> > > trunk
> > > > at the same time (because they take a very long time) and both test
> > > scripts
> > > > failed. I've switched to running them one after the other and they
> are
> > > fine.
> > > > Do you think that is ok?
> > > >
> > > > thanks for your time,
> > > > alex
> > > >
> > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > > wrote:
> > > >
> > > >> what does your script1-hadoop.pig look like?
> > > >>
> > > >> The error you are getting happens when the pig preprocessor can't
> > > >> substitute Pig variables (the stuff you specify with -param and
> > > >> %default, etc). Do you have $PIGDIR in your script somewhere?
> > > >>
> > > >> -D
> > > >>
> > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > > >> <al...@gmail.com> wrote:
> > > >> > Hello,
> > > >> >
> > > >> > I seem to have broken my Pig install, and I don't know where to
> > look.
> > > >> >
> > > >> > If I use directly the script (grunt) everything works ok, but
> every
> > > time
> > > >> I
> > > >> > try to run a pig script: 'java -cp
> $PIGDIR/pig.jar:$HADOOPSITEPATH
> > > >> > org.apache.pig.Main script1-hadoop.pig'
> > > >> >
> > > >> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
> > 2999:
> > > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > > >> >
> > > >> > Obviously I have the PIGDIR var set:
> > > >> >> echo $PIGDIR
> > > >> >> /home/alex/hadoop/pig
> > > >> >
> > > >> > This is something that I did, as I have used 0.5 and 0.6 and a
> > patched
> > > >> > version of 0.6 in parallel, but I can't figure out where to look.
> > Any
> > > >> > version I try to start now, gives the same error.
> > > >> >
> > > >> > any help would be greatly appreciated!
> > > >> >
> > > >> > alex
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Hm, nothing jumps out. Definitely the error you are getting indicates that
somehow the preprocessor is trying to substitute a variable called PIGDIR in
your script. Which is odd. Does the same thing happen if you try bin/pig -f
test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
and sending along the output. Why are you using pig-core instead of pig.jar
that ant should be generating for you when you build? Where did you get it,
how did you build it, and what's the output of cksum on it?
The fact that now your 0.5 is broken too makes me think that maybe your
symlinks are messed up. Surely a change to one jar shouldn't be affecting a
totally unrelated jar in a directory the first jar doesn't know about.
Sorry about the barrage of questions, I am just a bit dumbfounded about how
this could even begin to start happening. Any preprocessor experts around?
-D
On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
<al...@gmail.com>wrote:
> Hello
>
> And thanks again for all your help
>
> I have a symbolic link /home/alex/hadoop/pig which points to
> /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
> 0.6 branch.
> I'm running the script from /home/alex/hadoop/test.
>
> I didn't touch pig.properties, also I'm running with the default pig
> script.
>
> The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
> to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
> work(Exception
> in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).
>
> I'm running with a fresh 0.6 build and still no luck.
>
> The info you reqested:
>
> alex@alex-desktop:~/hadoop/test$ pwd
> /home/alex/hadoop/test
> alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
> JAVA_HOME=/usr/lib/jvm/java-6-sun
> alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
> PIGDIR=/home/alex/hadoop/pig
> alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
> HADOOP_HOME=/home/alex/hadoop/hadoop
> HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
> HADOOPDIR=/home/alex/hadoop/hadoop/conf
>
> alex@alex-desktop:~/hadoop/test$ cat
> /home/alex/hadoop/pig/conf/pig.properties
> # Pig configuration file. All values can be overwritten by command line
> arguments.
> # see bin/pig -help
>
> # log4jconf log4j configuration file
> # log4jconf=./conf/log4j.properties
>
> # brief logging (no timestamps)
> brief=false
>
> # clustername, name of the hadoop jobtracker. If no port is defined port
> 50020 will be used.
> #cluster
>
> #debug level, INFO is default
> debug=INFO
>
> # a file that contains pig script
> #file=
>
> # load jarfile, colon separated
> #jar=
>
> #verbose print all log messages to screen (default to print only INFO and
> above to screen)
> verbose=false
>
> #exectype local|mapreduce, mapreduce is default
> #exectype=mapreduce
> # hod realted properties
> #ssh.gateway
> #hod.expect.root
> #hod.expect.uselatest
> #hod.command
> #hod.config.dir
> #hod.param
>
>
> #Do not spill temp files smaller than this size (bytes)
> pig.spill.size.threshold=5000000
> #EXPERIMENT: Activate garbage collection when spilling a file bigger than
> this size (bytes)
> #This should help reduce the number of files being spilled.
> pig.spill.gc.activation.size=40000000
>
>
> ######################
> # Everything below this line is Yahoo specific. Note that I've made
> # (almost) no changes to the lines above to make merging in from Apache
> # easier. Any values I don't want from above I override below.
> #
> # This file is configured for use with HOD on the production clusters. If
> you
> # want to run pig with a static cluster you will need to remove everything
> # below this line and set the cluster value (above) to the
> # hostname and port of your job tracker.
>
> exectype=mapreduce
>
> hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
> hod.server=local
>
> cluster.domain=inktomisearch.com
>
> log.file=
>
> yinst.cluster=kryptonite
>
>
>
> And now boom!
>
> java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
> test_lite.pig
>
>
> 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
> messages to: /home/alex/hadoop/test/pig_1266319637282.log
> 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. Undefined parameter : PIGDIR
> Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
>
> The log file:
> alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
>
> java.lang.RuntimeException: Undefined parameter : PIGDIR
> at
>
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
> at
>
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> at
>
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
> at org.apache.pig.Main.main(Main.java:366)
>
> ================================================================================
>
>
> The script: 'A = load '/home/alex/hadoop/test/t.csv' using PigStorage('\t')
> as (id: long); '
> The file: '1' .
> Simple enough :)
>
> I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
> anything as scripts. If I go with manual processing in grunt (line by line)
> it's all good on any version.
>
> thanks,
> alex
>
>
> On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
>
> > Pig starts, the error you are getting is from inside pig.Main .
> > I think something is getting messed up in your environment because you
> > are juggling too many different versions of pig (granted, I have 3 or
> > 4 in various stages of development on my laptop most of the time, and
> > haven't had your problems. But then neither of them has a hacked
> > bin/pig , with the exception of the cloudera one...). I've never tried
> > running multiple ant tests, either. There's a short "sanity" version
> > of tests, ant test-commit, that runs in under 10 minutes. You might
> > want to try that if you are not doing things like changing join
> > implementations or messing with the optimizer.
> >
> > Let's do this: send a full trace of what you are doing and what your
> > environment looks like. Something like
> >
> > pwd
> > printenv | grep JAVA
> > printenv | grep PIG
> > printenv | grep HADOOP
> > cat conf/pig.properties
> > java -cp ........
> > <boom!>
> >
> > -D
> >
> >
> > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> > <al...@gmail.com> wrote:
> > > Hello,
> > >
> > > sorry for the delay, but I wanted to build from the source again just
> to
> > > make sure.
> > >
> > > The script is like this:
> > >
> > > 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as
> > (id:
> > > long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
> int);
> > '
> > >
> > > That's it. My guess is that Pig doesn't even start. Do you think I need
> > to
> > > change something in the properties file? I'm not sure anymore and I
> don't
> > > have a lot of luck going through the sources.
> > >
> > > And another thing, I tried running 'ant test' for both 0.6-branch and
> > trunk
> > > at the same time (because they take a very long time) and both test
> > scripts
> > > failed. I've switched to running them one after the other and they are
> > fine.
> > > Do you think that is ok?
> > >
> > > thanks for your time,
> > > alex
> > >
> > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> > >
> > >> what does your script1-hadoop.pig look like?
> > >>
> > >> The error you are getting happens when the pig preprocessor can't
> > >> substitute Pig variables (the stuff you specify with -param and
> > >> %default, etc). Do you have $PIGDIR in your script somewhere?
> > >>
> > >> -D
> > >>
> > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> > >> <al...@gmail.com> wrote:
> > >> > Hello,
> > >> >
> > >> > I seem to have broken my Pig install, and I don't know where to
> look.
> > >> >
> > >> > If I use directly the script (grunt) everything works ok, but every
> > time
> > >> I
> > >> > try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> > >> > org.apache.pig.Main script1-hadoop.pig'
> > >> >
> > >> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
> 2999:
> > >> > Unexpected internal error. Undefined parameter : PIGDIR
> > >> >
> > >> > Obviously I have the PIGDIR var set:
> > >> >> echo $PIGDIR
> > >> >> /home/alex/hadoop/pig
> > >> >
> > >> > This is something that I did, as I have used 0.5 and 0.6 and a
> patched
> > >> > version of 0.6 in parallel, but I can't figure out where to look.
> Any
> > >> > version I try to start now, gives the same error.
> > >> >
> > >> > any help would be greatly appreciated!
> > >> >
> > >> > alex
> > >> >
> > >>
> > >
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Alex Parvulescu <al...@gmail.com>.
Hello
And thanks again for all your help
I have a symbolic link /home/alex/hadoop/pig which points to
/home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
0.6 branch.
I'm running the script from /home/alex/hadoop/test.
I didn't touch pig.properties, also I'm running with the default pig script.
The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not work(Exception
in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).
I'm running with a fresh 0.6 build and still no luck.
The info you reqested:
alex@alex-desktop:~/hadoop/test$ pwd
/home/alex/hadoop/test
alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
JAVA_HOME=/usr/lib/jvm/java-6-sun
alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
PIGDIR=/home/alex/hadoop/pig
alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
HADOOP_HOME=/home/alex/hadoop/hadoop
HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
HADOOPDIR=/home/alex/hadoop/hadoop/conf
alex@alex-desktop:~/hadoop/test$ cat
/home/alex/hadoop/pig/conf/pig.properties
# Pig configuration file. All values can be overwritten by command line
arguments.
# see bin/pig -help
# log4jconf log4j configuration file
# log4jconf=./conf/log4j.properties
# brief logging (no timestamps)
brief=false
# clustername, name of the hadoop jobtracker. If no port is defined port
50020 will be used.
#cluster
#debug level, INFO is default
debug=INFO
# a file that contains pig script
#file=
# load jarfile, colon separated
#jar=
#verbose print all log messages to screen (default to print only INFO and
above to screen)
verbose=false
#exectype local|mapreduce, mapreduce is default
#exectype=mapreduce
# hod realted properties
#ssh.gateway
#hod.expect.root
#hod.expect.uselatest
#hod.command
#hod.config.dir
#hod.param
#Do not spill temp files smaller than this size (bytes)
pig.spill.size.threshold=5000000
#EXPERIMENT: Activate garbage collection when spilling a file bigger than
this size (bytes)
#This should help reduce the number of files being spilled.
pig.spill.gc.activation.size=40000000
######################
# Everything below this line is Yahoo specific. Note that I've made
# (almost) no changes to the lines above to make merging in from Apache
# easier. Any values I don't want from above I override below.
#
# This file is configured for use with HOD on the production clusters. If
you
# want to run pig with a static cluster you will need to remove everything
# below this line and set the cluster value (above) to the
# hostname and port of your job tracker.
exectype=mapreduce
hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
hod.server=local
cluster.domain=inktomisearch.com
log.file=
yinst.cluster=kryptonite
And now boom!
java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
test_lite.pig
2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
messages to: /home/alex/hadoop/test/pig_1266319637282.log
2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR
Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log
The log file:
alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
java.lang.RuntimeException: Undefined parameter : PIGDIR
at
org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
at org.apache.pig.Main.main(Main.java:366)
================================================================================
The script: 'A = load '/home/alex/hadoop/test/t.csv' using PigStorage('\t')
as (id: long); '
The file: '1' .
Simple enough :)
I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
anything as scripts. If I go with manual processing in grunt (line by line)
it's all good on any version.
thanks,
alex
On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Pig starts, the error you are getting is from inside pig.Main .
> I think something is getting messed up in your environment because you
> are juggling too many different versions of pig (granted, I have 3 or
> 4 in various stages of development on my laptop most of the time, and
> haven't had your problems. But then neither of them has a hacked
> bin/pig , with the exception of the cloudera one...). I've never tried
> running multiple ant tests, either. There's a short "sanity" version
> of tests, ant test-commit, that runs in under 10 minutes. You might
> want to try that if you are not doing things like changing join
> implementations or messing with the optimizer.
>
> Let's do this: send a full trace of what you are doing and what your
> environment looks like. Something like
>
> pwd
> printenv | grep JAVA
> printenv | grep PIG
> printenv | grep HADOOP
> cat conf/pig.properties
> java -cp ........
> <boom!>
>
> -D
>
>
> On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
> <al...@gmail.com> wrote:
> > Hello,
> >
> > sorry for the delay, but I wanted to build from the source again just to
> > make sure.
> >
> > The script is like this:
> >
> > 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as
> (id:
> > long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int);
> '
> >
> > That's it. My guess is that Pig doesn't even start. Do you think I need
> to
> > change something in the properties file? I'm not sure anymore and I don't
> > have a lot of luck going through the sources.
> >
> > And another thing, I tried running 'ant test' for both 0.6-branch and
> trunk
> > at the same time (because they take a very long time) and both test
> scripts
> > failed. I've switched to running them one after the other and they are
> fine.
> > Do you think that is ok?
> >
> > thanks for your time,
> > alex
> >
> > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
> >
> >> what does your script1-hadoop.pig look like?
> >>
> >> The error you are getting happens when the pig preprocessor can't
> >> substitute Pig variables (the stuff you specify with -param and
> >> %default, etc). Do you have $PIGDIR in your script somewhere?
> >>
> >> -D
> >>
> >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> >> <al...@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > I seem to have broken my Pig install, and I don't know where to look.
> >> >
> >> > If I use directly the script (grunt) everything works ok, but every
> time
> >> I
> >> > try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> >> > org.apache.pig.Main script1-hadoop.pig'
> >> >
> >> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
> >> > Unexpected internal error. Undefined parameter : PIGDIR
> >> >
> >> > Obviously I have the PIGDIR var set:
> >> >> echo $PIGDIR
> >> >> /home/alex/hadoop/pig
> >> >
> >> > This is something that I did, as I have used 0.5 and 0.6 and a patched
> >> > version of 0.6 in parallel, but I can't figure out where to look. Any
> >> > version I try to start now, gives the same error.
> >> >
> >> > any help would be greatly appreciated!
> >> >
> >> > alex
> >> >
> >>
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Pig starts, the error you are getting is from inside pig.Main .
I think something is getting messed up in your environment because you
are juggling too many different versions of pig (granted, I have 3 or
4 in various stages of development on my laptop most of the time, and
haven't had your problems. But then neither of them has a hacked
bin/pig , with the exception of the cloudera one...). I've never tried
running multiple ant tests, either. There's a short "sanity" version
of tests, ant test-commit, that runs in under 10 minutes. You might
want to try that if you are not doing things like changing join
implementations or messing with the optimizer.
Let's do this: send a full trace of what you are doing and what your
environment looks like. Something like
pwd
printenv | grep JAVA
printenv | grep PIG
printenv | grep HADOOP
cat conf/pig.properties
java -cp ........
<boom!>
-D
On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
<al...@gmail.com> wrote:
> Hello,
>
> sorry for the delay, but I wanted to build from the source again just to
> make sure.
>
> The script is like this:
>
> 'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
> long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '
>
> That's it. My guess is that Pig doesn't even start. Do you think I need to
> change something in the properties file? I'm not sure anymore and I don't
> have a lot of luck going through the sources.
>
> And another thing, I tried running 'ant test' for both 0.6-branch and trunk
> at the same time (because they take a very long time) and both test scripts
> failed. I've switched to running them one after the other and they are fine.
> Do you think that is ok?
>
> thanks for your time,
> alex
>
> On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> what does your script1-hadoop.pig look like?
>>
>> The error you are getting happens when the pig preprocessor can't
>> substitute Pig variables (the stuff you specify with -param and
>> %default, etc). Do you have $PIGDIR in your script somewhere?
>>
>> -D
>>
>> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
>> <al...@gmail.com> wrote:
>> > Hello,
>> >
>> > I seem to have broken my Pig install, and I don't know where to look.
>> >
>> > If I use directly the script (grunt) everything works ok, but every time
>> I
>> > try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
>> > org.apache.pig.Main script1-hadoop.pig'
>> >
>> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
>> > Unexpected internal error. Undefined parameter : PIGDIR
>> >
>> > Obviously I have the PIGDIR var set:
>> >> echo $PIGDIR
>> >> /home/alex/hadoop/pig
>> >
>> > This is something that I did, as I have used 0.5 and 0.6 and a patched
>> > version of 0.6 in parallel, but I can't figure out where to look. Any
>> > version I try to start now, gives the same error.
>> >
>> > any help would be greatly appreciated!
>> >
>> > alex
>> >
>>
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Alex Parvulescu <al...@gmail.com>.
Hello,
sorry for the delay, but I wanted to build from the source again just to
make sure.
The script is like this:
'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '
That's it. My guess is that Pig doesn't even start. Do you think I need to
change something in the properties file? I'm not sure anymore and I don't
have a lot of luck going through the sources.
And another thing, I tried running 'ant test' for both 0.6-branch and trunk
at the same time (because they take a very long time) and both test scripts
failed. I've switched to running them one after the other and they are fine.
Do you think that is ok?
thanks for your time,
alex
On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> what does your script1-hadoop.pig look like?
>
> The error you are getting happens when the pig preprocessor can't
> substitute Pig variables (the stuff you specify with -param and
> %default, etc). Do you have $PIGDIR in your script somewhere?
>
> -D
>
> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
> <al...@gmail.com> wrote:
> > Hello,
> >
> > I seem to have broken my Pig install, and I don't know where to look.
> >
> > If I use directly the script (grunt) everything works ok, but every time
> I
> > try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> > org.apache.pig.Main script1-hadoop.pig'
> >
> > I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
> > Unexpected internal error. Undefined parameter : PIGDIR
> >
> > Obviously I have the PIGDIR var set:
> >> echo $PIGDIR
> >> /home/alex/hadoop/pig
> >
> > This is something that I did, as I have used 0.5 and 0.6 and a patched
> > version of 0.6 in parallel, but I can't figure out where to look. Any
> > version I try to start now, gives the same error.
> >
> > any help would be greatly appreciated!
> >
> > alex
> >
>
Re: pig start error: ERROR 2999: Unexpected internal error. Undefined
parameter : PIGDIR
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
what does your script1-hadoop.pig look like?
The error you are getting happens when the pig preprocessor can't
substitute Pig variables (the stuff you specify with -param and
%default, etc). Do you have $PIGDIR in your script somewhere?
-D
On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
<al...@gmail.com> wrote:
> Hello,
>
> I seem to have broken my Pig install, and I don't know where to look.
>
> If I use directly the script (grunt) everything works ok, but every time I
> try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
> org.apache.pig.Main script1-hadoop.pig'
>
> I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. Undefined parameter : PIGDIR
>
> Obviously I have the PIGDIR var set:
>> echo $PIGDIR
>> /home/alex/hadoop/pig
>
> This is something that I did, as I have used 0.5 and 0.6 and a patched
> version of 0.6 in parallel, but I can't figure out where to look. Any
> version I try to start now, gives the same error.
>
> any help would be greatly appreciated!
>
> alex
>