You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Iman E <ha...@yahoo.com> on 2009/12/11 19:57:27 UTC
Re: DataGenerator Location
Hi all,
Thanks for sharing PigMix with us.
I do have a related question to this thread. The generate_data.sh references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar. This jar is never generated for me. I generated one out of the test class files and hope that it was it.
My other question, I tried to run the script but I keep getting this error:
Exception in thread "main" java.io.IOException: Error opening job jar: -libjars
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:114)
at java.util.jar.JarFile.<init>(JarFile.java:133)
at java.util.jar.JarFile.<init>(JarFile.java:70)
at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
I made sure that the sdsuLibJKD12.jar is in place and I was able to compile the tests using it, but still the script will not work! Any suggestions.
Thanks
________________________________
From: Ashutosh Chauhan <as...@gmail.com>
To: pig-user@hadoop.apache.org
Sent: Mon, November 2, 2009 12:01:00 PM
Subject: Re: DataGenerator Location
>
>
> I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
> cannot find any such classpath. Where can I expect to find this
> DataGenerator class?
>
As Alan wrote earlier DataGenerator is in patch
https://issues.apache.org/jira/browse/PIG-200 You need to download the patch
and build it.
>
> Finally, that Wiki page refers to specifying the path to $zipfjar:
> sdsuLibJKD12.jar . Where can I find this jar file?
>
http://www.eli.sdsu.edu/java-SDSU/
Hope it helps,
Ashutosh
Re: DataGenerator Location
Posted by Iman E <ha...@yahoo.com>.
Thanks a lot Rob. It worked. It turned to be that I had a problem with my test.jar.
________________________________
From: Rob Stewart <ro...@googlemail.com>
To: pig-user@hadoop.apache.org
Sent: Fri, December 11, 2009 7:25:00 PM
Subject: Re: DataGenerator Location
OK, it seems you have an issue with your classpaths. It may be an idea to
make a file called "generateData". Then in this, contain the following
lines:
-------------------------
export pigjar=$PIGHOME/pig-0.4.0-core.jar
export zipfjar=[LOCATION_TO]/sdsuLibJKD14.jar
export datagenjar=[LOCATION_TO]/test.jar #(that you have created
including the patch)
export conf_file=$HADOOP_HOME/conf/hadoop-site.xml
export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar
$HADOOP_HOME/bin/hadoop jar $datagenjar
org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -rows 100
-f generated_data.dat s:8:50:z:0
----------------------
Providing "echo $HADOOP_CLASSPATH" looks OK, there should be a file
generated_data.dat in the same directory as the generateData file, once the
generateData file has been executed.
Let us know how you get on.
Rob Stewart
2009/12/11 Iman E <ha...@yahoo.com>
> Thankyou so much, Rob for your reply.
>
> Moving the -libjars fixed the error, but I still get this error:
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.pig.test.utils.datagen.DataGenerator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>
> The datagenjar (which is the test.jar that I have generated) has the
> org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix
> the error.
> Thanks
> Iman
>
>
>
>
> ________________________________
> From: Rob Stewart <ro...@googlemail.com>
> To: pig-user@hadoop.apache.org
> Sent: Fri, December 11, 2009 2:13:17 PM
> Subject: Re: DataGenerator Location
>
> This is a good point... the DataGenerator on the Wiki page needs updating.
>
> In Hadoop 0.20 the -libjars comes *after* the jar file.
>
> It should say:
> hadoop jar $datagenjar
> org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
> -libjars
> $zipfjar -conf $conf_file [options] colspec....
>
>
> Iman - Could you try doing this and let us know?
>
> Thanks,
>
> Rob Stewart
>
>
> 2009/12/11 Iman E <ha...@yahoo.com>
>
> > Hi all,
> > Thanks for sharing PigMix with us.
> > I do have a related question to this thread. The generate_data.sh
> > references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
> > This jar is never generated for me. I generated one out of the test class
> > files and hope that it was it.
> > My other question, I tried to run the script but I keep getting this
> error:
> > Exception in thread "main" java.io.IOException: Error opening job jar:
> > -libjars
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> > Caused by: java.util..zip.ZipException: error in opening zip file
> > at java.util.zip.ZipFile.open(Native Method)
> > at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> > at java.util.jar.JarFile.<init>(JarFile.java:133)
> > at java.util.jar.JarFile.<init>(JarFile.java:70)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
> >
> > I made sure that the sdsuLibJKD12.jar is in place and I was able to
> compile
> > the tests using it, but still the script will not work! Any suggestions.
> > Thanks
> >
> >
> >
> >
> > ________________________________
> > From: Ashutosh Chauhan <as...@gmail.com>
> > To: pig-user@hadoop.apache..org
> > Sent: Mon, November 2, 2009 12:01:00 PM
> > Subject: Re: DataGenerator Location
> >
> > >
> > >
> > > I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
> > > cannot find any such classpath. Where can I expect to find this
> > > DataGenerator class?
> > >
> >
> > As Alan wrote earlier DataGenerator is in patch
> > https://issues.apache.org/jira/browse/PIG-200 You need to download the
> > patch
> > and build it.
> >
> > >
> > > Finally, that Wiki page refers to specifying the path to $zipfjar:
> > > sdsuLibJKD12.jar . Where can I find this jar file?
> > >
> >
> > http://www.eli.sdsu.edu/java-SDSU/
> >
> > Hope it helps,
> > Ashutosh
> >
> >
> >
> >
>
>
>
>
>
Re: DataGenerator Location
Posted by Rob Stewart <ro...@googlemail.com>.
OK, it seems you have an issue with your classpaths. It may be an idea to
make a file called "generateData". Then in this, contain the following
lines:
-------------------------
export pigjar=$PIGHOME/pig-0.4.0-core.jar
export zipfjar=[LOCATION_TO]/sdsuLibJKD14.jar
export datagenjar=[LOCATION_TO]/test.jar #(that you have created
including the patch)
export conf_file=$HADOOP_HOME/conf/hadoop-site.xml
export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar
$HADOOP_HOME/bin/hadoop jar $datagenjar
org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -rows 100
-f generated_data.dat s:8:50:z:0
----------------------
Providing "echo $HADOOP_CLASSPATH" looks OK, there should be a file
generated_data.dat in the same directory as the generateData file, once the
generateData file has been executed.
Let us know how you get on.
Rob Stewart
2009/12/11 Iman E <ha...@yahoo.com>
> Thankyou so much, Rob for your reply.
>
> Moving the -libjars fixed the error, but I still get this error:
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.pig.test.utils.datagen.DataGenerator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>
> The datagenjar (which is the test.jar that I have generated) has the
> org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix
> the error.
> Thanks
> Iman
>
>
>
>
> ________________________________
> From: Rob Stewart <ro...@googlemail.com>
> To: pig-user@hadoop.apache.org
> Sent: Fri, December 11, 2009 2:13:17 PM
> Subject: Re: DataGenerator Location
>
> This is a good point.. the DataGenerator on the Wiki page needs updating.
>
> In Hadoop 0.20 the -libjars comes *after* the jar file.
>
> It should say:
> hadoop jar $datagenjar
> org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
> -libjars
> $zipfjar -conf $conf_file [options] colspec...
>
>
> Iman - Could you try doing this and let us know?
>
> Thanks,
>
> Rob Stewart
>
>
> 2009/12/11 Iman E <ha...@yahoo.com>
>
> > Hi all,
> > Thanks for sharing PigMix with us.
> > I do have a related question to this thread. The generate_data.sh
> > references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
> > This jar is never generated for me. I generated one out of the test class
> > files and hope that it was it.
> > My other question, I tried to run the script but I keep getting this
> error:
> > Exception in thread "main" java.io.IOException: Error opening job jar:
> > -libjars
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> > Caused by: java.util.zip.ZipException: error in opening zip file
> > at java.util.zip.ZipFile.open(Native Method)
> > at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> > at java.util.jar.JarFile.<init>(JarFile.java:133)
> > at java.util.jar.JarFile.<init>(JarFile.java:70)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
> >
> > I made sure that the sdsuLibJKD12.jar is in place and I was able to
> compile
> > the tests using it, but still the script will not work! Any suggestions.
> > Thanks
> >
> >
> >
> >
> > ________________________________
> > From: Ashutosh Chauhan <as...@gmail.com>
> > To: pig-user@hadoop.apache.org
> > Sent: Mon, November 2, 2009 12:01:00 PM
> > Subject: Re: DataGenerator Location
> >
> > >
> > >
> > > I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
> > > cannot find any such classpath. Where can I expect to find this
> > > DataGenerator class?
> > >
> >
> > As Alan wrote earlier DataGenerator is in patch
> > https://issues.apache.org/jira/browse/PIG-200 You need to download the
> > patch
> > and build it.
> >
> > >
> > > Finally, that Wiki page refers to specifying the path to $zipfjar:
> > > sdsuLibJKD12.jar . Where can I find this jar file?
> > >
> >
> > http://www.eli.sdsu.edu/java-SDSU/
> >
> > Hope it helps,
> > Ashutosh
> >
> >
> >
> >
>
>
>
>
>
Re: DataGenerator Location
Posted by Iman E <ha...@yahoo.com>.
Thankyou so much, Rob for your reply.
Moving the -libjars fixed the error, but I still get this error:
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.pig.test.utils.datagen.DataGenerator
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
The datagenjar (which is the test.jar that I have generated) has the org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix the error.
Thanks
Iman
________________________________
From: Rob Stewart <ro...@googlemail.com>
To: pig-user@hadoop.apache.org
Sent: Fri, December 11, 2009 2:13:17 PM
Subject: Re: DataGenerator Location
This is a good point.. the DataGenerator on the Wiki page needs updating.
In Hadoop 0.20 the -libjars comes *after* the jar file.
It should say:
hadoop jar $datagenjar
org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
-libjars
$zipfjar -conf $conf_file [options] colspec...
Iman - Could you try doing this and let us know?
Thanks,
Rob Stewart
2009/12/11 Iman E <ha...@yahoo.com>
> Hi all,
> Thanks for sharing PigMix with us.
> I do have a related question to this thread. The generate_data.sh
> references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
> This jar is never generated for me. I generated one out of the test class
> files and hope that it was it.
> My other question, I tried to run the script but I keep getting this error:
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:133)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
> I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
> the tests using it, but still the script will not work! Any suggestions.
> Thanks
>
>
>
>
> ________________________________
> From: Ashutosh Chauhan <as...@gmail.com>
> To: pig-user@hadoop.apache.org
> Sent: Mon, November 2, 2009 12:01:00 PM
> Subject: Re: DataGenerator Location
>
> >
> >
> > I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
> > cannot find any such classpath. Where can I expect to find this
> > DataGenerator class?
> >
>
> As Alan wrote earlier DataGenerator is in patch
> https://issues.apache.org/jira/browse/PIG-200 You need to download the
> patch
> and build it.
>
> >
> > Finally, that Wiki page refers to specifying the path to $zipfjar:
> > sdsuLibJKD12.jar . Where can I find this jar file?
> >
>
> http://www.eli.sdsu.edu/java-SDSU/
>
> Hope it helps,
> Ashutosh
>
>
>
>
Re: DataGenerator Location
Posted by Rob Stewart <ro...@googlemail.com>.
This is a good point.. the DataGenerator on the Wiki page needs updating.
In Hadoop 0.20 the -libjars comes *after* the jar file.
It should say:
hadoop jar $datagenjar
org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
-libjars
$zipfjar -conf $conf_file [options] colspec...
Iman - Could you try doing this and let us know?
Thanks,
Rob Stewart
2009/12/11 Iman E <ha...@yahoo.com>
> Hi all,
> Thanks for sharing PigMix with us.
> I do have a related question to this thread. The generate_data.sh
> references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
> This jar is never generated for me. I generated one out of the test class
> files and hope that it was it.
> My other question, I tried to run the script but I keep getting this error:
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:133)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
> I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
> the tests using it, but still the script will not work! Any suggestions.
> Thanks
>
>
>
>
> ________________________________
> From: Ashutosh Chauhan <as...@gmail.com>
> To: pig-user@hadoop.apache.org
> Sent: Mon, November 2, 2009 12:01:00 PM
> Subject: Re: DataGenerator Location
>
> >
> >
> > I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
> > cannot find any such classpath. Where can I expect to find this
> > DataGenerator class?
> >
>
> As Alan wrote earlier DataGenerator is in patch
> https://issues.apache.org/jira/browse/PIG-200 You need to download the
> patch
> and build it.
>
> >
> > Finally, that Wiki page refers to specifying the path to $zipfjar:
> > sdsuLibJKD12.jar . Where can I find this jar file?
> >
>
> http://www.eli.sdsu.edu/java-SDSU/
>
> Hope it helps,
> Ashutosh
>
>
>
>