You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "Kaluskar, Sanjay" <sk...@informatica.com> on 2010/08/04 12:13:07 UTC
RE: UDF with dependency on external jars & native code
The register isn't working after I made some changes to mapred-site.xml.
Right now I am executing PIG script from the command-line as follows:
java -cp infapig.jar:${HADOOP_HOME}/conf org.apache.pig.Main
script.pig
In my script I have a register infapig.jar, which contains everything
needed by my UDF. But, I get the following error:
2010-08-04 20:44:44,933 [main] ERROR
org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate
exception from backed error: Error: java.lang.ClassNotFoundException:
com.informatica.products.pigudf.PigudfException
PigudfException is an exception defined in one of the jars on the
classpath of infapig.jar.
I have the following in mapred-site.xml:
<property>
<name>mapred.cache.archives</name>
<value>hdfs://inarch03.informatica.com:54310/infadoop/installer-9.0.2-SN
APSHOT.zip#infadoop</value>
</property>
<property>
<name>mapred.create.symlink</name>
<value>yes</value>
</property>
<property>
<name>mapred.job.classpath.files</name>
<value>infadoop/infapig.jar</value>
</property>
I have also tried adding the following to mapred-site.xml:
<property>
<name>pig.additional.jars</name>
<value>infadoop/infapig.jar</value>
</property>
However, the following executions work:
java -cp infapig.jar:${HADOOP_HOME}/conf org.apache.pig.Main -x
local script.pig
java -cp infapig.jar org.apache.pig.Main script.pig
-----Original Message-----
From: Thejas M Nair [mailto:tejas@yahoo-inc.com]
Sent: Thursday, July 29, 2010 11:16 PM
To: pig-user@hadoop.apache.org; Kaluskar, Sanjay
Subject: Re: UDF with dependency on external jars & native code
You can use the MR distributed cache to push the native libs - see -
http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Distri
bute
dCache
"The DistributedCache can also be used to distribute both jars and
native libraries for use in the map and/or reduce tasks. The child-jvm
always has its current working directory added to the java.library.path
and LD_LIBRARY_PATH. And hence the cached libraries can be loaded via
System.loadLibrary or System.load . More details on how to load
shared
libraries through distributed cache are documented at
native_libraries.htm"
So using -Dmapred.cache.files=<dfs path to file>, in your pig
commandline should work.
Please let us know if this worked for you.
For the jars, you can also use a commandline option -
-Dpig.additional.jars="jar1:jar2.."
(thanks to Pradeep for suggesting this solution)
Thanks,
Thejas
On 7/26/10 9:38 AM, "Kaluskar, Sanjay" <sk...@informatica.com>
wrote:
> I am new to PIG and running into a fairly basic problem. I have a UDF
> which depends on some other 3rd party jars & libraries. I can call the
> UDF from my PIG script either from grunt or by running "java -cp ...
> org.apache.pig.Main <script>" in local mode, when I have the jars on
> the classpath and the libraries on LD_LIBRARY_PATH. But, in mapreduce
> mode I get errors from Hadoop because it doesn't find the classes &
libraries.
>
> I saw another thread on this forum, which had a workaround for the
jar.
> I can explicitly call register on the dependency, and that seems to
> fix the problem. But, there doesn't seem to be a way of specifying the
> native libraries to PIG such that the map/reduce jobs are set up to
> access them.
>
> I am using PIG 0.5.0. Any help is appreciated!
>
> Thanks,
> -sanjay
>
RE: UDF with dependency on external jars & native code
Posted by "Kaluskar, Sanjay" <sk...@informatica.com>.
I read the tutorial once more & decided to try out the simplest possible
config that might work, which is copying the necessary native libs using
dist cache (they are in an archive called installer-9.0.2-SNAPSHOT.zip),
and hoping that PIG will take care of copying the jars that are
registered. I also need to set some env variables to get my native code
to work.
So, now I have the following in the mapred-site.xml:
<property>
<name>mapred.cache.archives</name>
<value>hdfs://inarch03.informatica.com:54310/infadoop/installer-9.0.2-SN
APSHOT.zip#infadoop</value>
</property>
<property>
<name>mapred.create.symlink</name>
<value>yes</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value> -Xmx512M -Djava.library.path=infadoop/infa-resources </value>
</property>
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=infadoop/infa-resources,INFA_RESOURCES=infadoop/i
nfa-resources,IMF_CPP_RESOURCE_PATH=infadoop/infa-resources</value>
</property>
With this change the job setup fails, and I see the following error in
the userlogs:
Exception in thread "main" java.lang.NoClassDefFoundError:
Caused by: java.lang.ClassNotFoundException:
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: . Program will exit.
-----Original Message-----
From: Thejas M Nair [mailto:tejas@yahoo-inc.com
<ma...@yahoo-inc.com> ]
Sent: Thursday, August 05, 2010 12:16 AM
To: pig-user@hadoop.apache.org; Kaluskar, Sanjay
Subject: Re: UDF with dependency on external jars & native code
On 8/4/10 3:13 AM, "Kaluskar, Sanjay" <sk...@informatica.com> wrote:
> The register isn't working after I made some changes to
mapred-site.xml.
> Right now I am executing PIG script from the command-line as follows:
>
Do you know what change in mapred-site.xml caused it to stop working ?
Is it after adding mapred.cache.archives ?
> PigudfException is an exception defined in one of the jars on the
> classpath of infapig.jar.
Is PigudfException also packaged within the jar ?
-Thejas
> -----Original Message-----
> From: Thejas M Nair [mailto:tejas@yahoo-inc.com
<ma...@yahoo-inc.com> ]
> Sent: Thursday, July 29, 2010 11:16 PM
> To: pig-user@hadoop.apache.org; Kaluskar, Sanjay
> Subject: Re: UDF with dependency on external jars & native code
>
> You can use the MR distributed cache to push the native libs - see -
> http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Dist
<http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Dist>
> ri
> bute
> dCache
>
> "The DistributedCache can also be used to distribute both jars and
> native libraries for use in the map and/or reduce tasks. The
> child-jvm always has its current working directory added to the
> java.library.path and LD_LIBRARY_PATH. And hence the cached libraries
can be loaded via
> System.loadLibrary or System.load . More details on how to load
> shared
> libraries through distributed cache are documented at
> native_libraries.htm"
>
> So using -Dmapred.cache.files=<dfs path to file>, in your pig
> commandline should work.
>
> Please let us know if this worked for you.
>
> For the jars, you can also use a commandline option -
> -Dpig.additional.jars="jar1:jar2.."
>
> (thanks to Pradeep for suggesting this solution)
>
> Thanks,
> Thejas
>
> On 7/26/10 9:38 AM, "Kaluskar, Sanjay" <sk...@informatica.com>
> wrote:
>
>> I am new to PIG and running into a fairly basic problem. I have a UDF
>> which depends on some other 3rd party jars & libraries. I can call
>> the UDF from my PIG script either from grunt or by running "java -cp
...
>> org.apache.pig.Main <script>" in local mode, when I have the jars on
>> the classpath and the libraries on LD_LIBRARY_PATH. But, in mapreduce
>> mode I get errors from Hadoop because it doesn't find the classes &
> libraries.
>>
>> I saw another thread on this forum, which had a workaround for the
> jar.
>> I can explicitly call register on the dependency, and that seems to
>> fix the problem. But, there doesn't seem to be a way of specifying
>> the native libraries to PIG such that the map/reduce jobs are set up
>> to access them.
>>
>> I am using PIG 0.5.0. Any help is appreciated!
>>
>> Thanks,
>> -sanjay
>>
>
>
>
>
>
Re: UDF with dependency on external jars & native code
Posted by Thejas M Nair <te...@yahoo-inc.com>.
On 8/4/10 3:13 AM, "Kaluskar, Sanjay" <sk...@informatica.com> wrote:
> The register isn't working after I made some changes to mapred-site.xml.
> Right now I am executing PIG script from the command-line as follows:
>
Do you know what change in mapred-site.xml caused it to stop working ? Is it
after adding mapred.cache.archives ?
> PigudfException is an exception defined in one of the jars on the
> classpath of infapig.jar.
Is PigudfException also packaged within the jar ?
-Thejas
> -----Original Message-----
> From: Thejas M Nair [mailto:tejas@yahoo-inc.com]
> Sent: Thursday, July 29, 2010 11:16 PM
> To: pig-user@hadoop.apache.org; Kaluskar, Sanjay
> Subject: Re: UDF with dependency on external jars & native code
>
> You can use the MR distributed cache to push the native libs - see -
> http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Distri
> bute
> dCache
>
> "The DistributedCache can also be used to distribute both jars and
> native libraries for use in the map and/or reduce tasks. The child-jvm
> always has its current working directory added to the java.library.path
> and LD_LIBRARY_PATH. And hence the cached libraries can be loaded via
> System.loadLibrary or System.load . More details on how to load
> shared
> libraries through distributed cache are documented at
> native_libraries.htm"
>
> So using -Dmapred.cache.files=<dfs path to file>, in your pig
> commandline should work.
>
> Please let us know if this worked for you.
>
> For the jars, you can also use a commandline option -
> -Dpig.additional.jars="jar1:jar2.."
>
> (thanks to Pradeep for suggesting this solution)
>
> Thanks,
> Thejas
>
> On 7/26/10 9:38 AM, "Kaluskar, Sanjay" <sk...@informatica.com>
> wrote:
>
>> I am new to PIG and running into a fairly basic problem. I have a UDF
>> which depends on some other 3rd party jars & libraries. I can call the
>> UDF from my PIG script either from grunt or by running "java -cp ...
>> org.apache.pig.Main <script>" in local mode, when I have the jars on
>> the classpath and the libraries on LD_LIBRARY_PATH. But, in mapreduce
>> mode I get errors from Hadoop because it doesn't find the classes &
> libraries.
>>
>> I saw another thread on this forum, which had a workaround for the
> jar.
>> I can explicitly call register on the dependency, and that seems to
>> fix the problem. But, there doesn't seem to be a way of specifying the
>> native libraries to PIG such that the map/reduce jobs are set up to
>> access them.
>>
>> I am using PIG 0.5.0. Any help is appreciated!
>>
>> Thanks,
>> -sanjay
>>
>
>
>
>
>