Re: Import libraries in Jython UDFs

[Russell Jurney <ru...@gmail.com>]

Fixed by https://issues.apache.org/jira/browse/PIG-2665 in Pig 0.11


On Wed, Jul 25, 2012 at 10:16 PM, Russell Jurney
<ru...@gmail.com>wrote:

> Does Jython.jar have the base libs?
>
> I'm running this in local mode, btw.
>
> Russell Jurney
> twitter.com/rjurney
> russell.jurney@gmail.com
> datasyndrome.com
>
> On Jul 25, 2012, at 5:11 PM, Aniket Mokashi <an...@gmail.com> wrote:
>
> > Hi Russell,
> >
> > Check if your hadoop has MAPREDUCE-967. If not, pig's job.jar will be
> > unjared on the tasks. With that, jython's static method's do not put /Lib
> > in python sys path. You can use the instrumentation code i shared in your
> > python udf file and find the sys.path on task nodes to confirm this. If
> > /Lib is on sys.path, all imports will work.
> >
> > One workaround is to put jython jar on all nodes of the cluster under
> > hadoop lib.
> >
> > ~Aniket
> >
> > On Tue, Jul 24, 2012 at 5:21 PM, Russell Jurney <
> russell.jurney@gmail.com>wrote:
> >
> >> Thanks, but that isn't my issue. I am unable to import any packages.
> >> Trying to get the path right...
> >>
> >> Russell Jurney http://datasyndrome.com
> >>
> >> On Jul 24, 2012, at 10:00 AM, Chun Yang <
> cyang@contractor.salesforce.com>
> >> wrote:
> >>
> >>> Hi Russell,
> >>>
> >>> Are you able to import other modules beside email? If not, maybe this
> is
> >>> related to your problem:
> https://issues.apache.org/jira/browse/PIG-2665
> >>>
> >>> -Chun
> >>>
> >>> On 7/23/12 11:26 PM, "Russell Jurney" <ru...@gmail.com>
> wrote:
> >>>
> >>>> ls /me/jython2.5.2/Lib/
> >>>>
> >>>> tons of class files...
> >>>> email/
> >>>>
> >>>>
> >>>> This is in local mode, atm. I add this directory to my java classpath,
> >>>> check.
> >>>>
> >>>> On Mon, Jul 23, 2012 at 11:10 PM, Aniket Mokashi <aniket486@gmail.com
> >>> wrote:
> >>>>
> >>>>> jar tf jython.jar | grep email
> >>>>>
> >>>>> Having jar in PIG_CLASSPATH would work if you have
> >>>>> https://issues.apache.org/jira/browse/MAPREDUCE-967.
> >>>>>
> >>>>> You can use following to debug the sys.path on tasknodes-
> >>>>>
> >>>>> from java.lang import System
> >>>>> print "python.home "
> >>>>> print System.getProperties().getProperty("python.home")
> >>>>> print "java.class.path "
> >>>>> print System.getProperties().getProperty("java.class.path")
> >>>>> print "install.root "
> >>>>> print System.getProperties().getProperty("install.root")
> >>>>> print "python.home "
> >>>>> print System.getProperties().getProperty("python.home")
> >>>>>
> >>>>> ~Aniket
> >>>>>
> >>>>> On Mon, Jul 23, 2012 at 6:33 PM, Russell Jurney <
> >> russell.jurney@gmail.com
> >>>>>> wrote:
> >>>>>
> >>>>>> No, how do I find which jar the email package is in?
> >>>>>>
> >>>>>> On Mon, Jul 23, 2012 at 6:02 PM, Norbert Burger <
> >>>>> norbert.burger@gmail.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> Have you registered the JAR in your Pig script (for local mode) and
> >>>>>>> also added it to PIG_CLASSPATH (for remote mode, to get it into the
> >>>>>>> distributed cache)?
> >>>>>>>
> >>>>>>> Norbert
> >>>>>>>
> >>>>>>> On Mon, Jul 23, 2012 at 8:33 PM, Russell Jurney
> >>>>>>> <ru...@gmail.com> wrote:
> >>>>>>>> The email package is a part of Jython, I believe:
> >>>>>>>> http://www.jython.org/docs/library/email.html
> >>>>>>>>
> >>>>>>>> However, when I 'import email' in udfs.py, I get this error:
> >>>>>>>>
> >>>>>>>> 2012-07-23 17:32:51,027 [main] ERROR
> >>>>> org.apache.pig.tools.grunt.Grunt -
> >>>>>>>> ERROR 1121: Python Error. Traceback (most recent call last):
> >>>>>>>> File "/Users/rjurney/Collecting-Data/src/pig/udfs.py", line 1, in
> >>>>>>> <module>
> >>>>>>>>   import email
> >>>>>>>> ImportError: No module named email
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> How do I import and use built-in packages in Jython?
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
> >>>>>>> datasyndrome.com
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
> >>>>>> datasyndrome.com
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> "...:::Aniket:::... Quetzalco@tl"
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
> >> datasyndrome.com
> >>>
> >>
> >
> >
> >
> > --
> > "...:::Aniket:::... Quetzalco@tl"
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com