You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Bill Janssen <ja...@parc.com> on 2011/03/04 06:50:40 UTC
Using JCC / PyLucene with JEPP?
New topic.
I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
with Tomcat.
Now, am I going to have to do some trickery to get a VM? Or will
getVMEnv() just work with a previously initialized JVM?
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.
See
http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201101.mbox/%3calpine.OSX.2.01.1101111021510.2366@yuzu.local%3e
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Andi Vajda <va...@apache.org>.
On Fri, 4 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>>>> If JEPP execs python, then that Python VM runs in a subprocess of the
>>>
>>> It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
>>> So the Python VM is running in the same address space as Java.
>>
>> Ok, so how is JEPP doing this from Tomcat without patching the Tomcat
>> startup code so that the PythonVM is initialized in the main thread ?
>
> I don't know that it is. What I'm interested in doing is starting a
> Python interpreter in Java, then occasionally calling into it to exec
> some Python code.
I see. Look at the PythonVM.instantiate() method. It's the one that does the
python equivalent of "from module import blah; blah()".
You could add another method there that does an exec or eval instead.
Andi..
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> >> If JEPP execs python, then that Python VM runs in a subprocess of the
> >
> > It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
> > So the Python VM is running in the same address space as Java.
>
> Ok, so how is JEPP doing this from Tomcat without patching the Tomcat
> startup code so that the PythonVM is initialized in the main thread ?
I don't know that it is. What I'm interested in doing is starting a
Python interpreter in Java, then occasionally calling into it to exec
some Python code. It would be great if it could also eval a Python
expression and return the unicode() of the result. I'd like to see if
I can add that functionality to PythonVM. Then I'll worry about Tomcat,
where I imagine I'll have to do just what you've already done.
> If what you say is true then how is that different from what Roman is
> doing with JCC and Python's subprocessing module ?
I don't know that it is. I use the subprocess module a lot in UpLib, too.
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Andi Vajda <va...@apache.org>.
On Mar 4, 2011, at 11:34, Bill Janssen <ja...@parc.com> wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>>
>> On Fri, 4 Mar 2011, Bill Janssen wrote:
>>
>>> Andi Vajda <va...@apache.org> wrote:
>>>
>>>>> Or if I need to run the same Python program multiple times?
>>>>
>>>> There is no notion of python program in this context (no
>>>> __main__). You import a module, instantiate a class from it and invoke
>>>> its methods. You can import and instantiate as many classes and
>>>> modules as you like, of course.
>>>
>>> JEPP has an interface to Python's "exec". This avoids the overhead of
>>> writing a Java class which is then subclassed in Python which is then
>>> instantiated and brought out to Java again so that you can call one of
>>> its methods. That would be a handy thing to have in PythonVM. I
>>> suppose I could add it, after I get PythonVM working (this loadLibrary
>>> thing).
>>>
>>> Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
>>> some way to pass the Java VM "into" the JEPP instantiation of Python, in
>>> such a way that that initVM() could find it.
>>
>> If JEPP execs python, then that Python VM runs in a subprocess of the
>
> It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
> So the Python VM is running in the same address space as Java.
Ok, so how is JEPP doing this from Tomcat without patching the Tomcat startup code so that the PythonVM is initialized in the main thread ?
If what you say is true then how is that different from what Roman is doing with JCC and Python's subprocessing module ?
Andi..
>
>> Java process. If you then call JCC's initVM() from that subprocess, I
>> do not know how the JNI api would behave in that situation. Will it
>> "find" the parent process' JVM, I don't know. I doubt that JNI has all
>> the magic required for transparent IPC to the parent process built-in
>> out of the box.
>
> Right. If I was going to do that, I'd just use java.lang.Runtime.exec().
>
> Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
>
> On Fri, 4 Mar 2011, Bill Janssen wrote:
>
> > Andi Vajda <va...@apache.org> wrote:
> >
> >>> Or if I need to run the same Python program multiple times?
> >>
> >> There is no notion of python program in this context (no
> >> __main__). You import a module, instantiate a class from it and invoke
> >> its methods. You can import and instantiate as many classes and
> >> modules as you like, of course.
> >
> > JEPP has an interface to Python's "exec". This avoids the overhead of
> > writing a Java class which is then subclassed in Python which is then
> > instantiated and brought out to Java again so that you can call one of
> > its methods. That would be a handy thing to have in PythonVM. I
> > suppose I could add it, after I get PythonVM working (this loadLibrary
> > thing).
> >
> > Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
> > some way to pass the Java VM "into" the JEPP instantiation of Python, in
> > such a way that that initVM() could find it.
>
> If JEPP execs python, then that Python VM runs in a subprocess of the
It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
So the Python VM is running in the same address space as Java.
> Java process. If you then call JCC's initVM() from that subprocess, I
> do not know how the JNI api would behave in that situation. Will it
> "find" the parent process' JVM, I don't know. I doubt that JNI has all
> the magic required for transparent IPC to the parent process built-in
> out of the box.
Right. If I was going to do that, I'd just use java.lang.Runtime.exec().
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Andi Vajda <va...@apache.org>.
On Fri, 4 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>>> Or if I need to run the same Python program multiple times?
>>
>> There is no notion of python program in this context (no
>> __main__). You import a module, instantiate a class from it and invoke
>> its methods. You can import and instantiate as many classes and
>> modules as you like, of course.
>
> JEPP has an interface to Python's "exec". This avoids the overhead of
> writing a Java class which is then subclassed in Python which is then
> instantiated and brought out to Java again so that you can call one of
> its methods. That would be a handy thing to have in PythonVM. I
> suppose I could add it, after I get PythonVM working (this loadLibrary
> thing).
>
> Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
> some way to pass the Java VM "into" the JEPP instantiation of Python, in
> such a way that that initVM() could find it.
If JEPP execs python, then that Python VM runs in a subprocess of the Java
process. If you then call JCC's initVM() from that subprocess, I do not know
how the JNI api would behave in that situation. Will it "find" the parent
process' JVM, I don't know. I doubt that JNI has all the magic required for
transparent IPC to the parent process built-in out of the box.
This is somewhat similar in setup to what Roman is doing, using Python's
subprocessing module insteaf. I had done that too, originally, but gave up
because of inextricable deadlocks.
Andi..
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> > Or if I need to run the same Python program multiple times?
>
> There is no notion of python program in this context (no
> __main__). You import a module, instantiate a class from it and invoke
> its methods. You can import and instantiate as many classes and
> modules as you like, of course.
JEPP has an interface to Python's "exec". This avoids the overhead of
writing a Java class which is then subclassed in Python which is then
instantiated and brought out to Java again so that you can call one of
its methods. That would be a handy thing to have in PythonVM. I
suppose I could add it, after I get PythonVM working (this loadLibrary
thing).
Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
some way to pass the Java VM "into" the JEPP instantiation of Python, in
such a way that that initVM() could find it.
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Andi Vajda <va...@apache.org>.
On Mar 4, 2011, at 9:40, Bill Janssen <ja...@parc.com> wrote:
> The number of patches to Tomcat make me uneasy. I was hoping to bundle
> all this into a .war file containing a servlet which would use it with
> an unmodified Tomcat.
>
> Does that seem possible?
No. The PythonVM must be initialized in the main thread and you have no control over that in Tomcat out of the box, apparently.
> Also, what if I want to run a number of different Python programs? Can
> I instantiate PythonVM more than once?
No. The PythonVM is loaded in process as a shared library and there can only be one per process. In other words, only one PythonVM can be instantiated from that library in a given process. See docs about Py_Initialize().
Andi..
> Or if I need to run the same
> Python program multiple times?
There is no notion of python program in this context (no __main__). You import a module, instantiate a class from it and invoke its methods. You can import and instantiate as many classes and modules as you like, of course.
Andi..
>
> Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
The number of patches to Tomcat make me uneasy. I was hoping to bundle
all this into a .war file containing a servlet which would use it with
an unmodified Tomcat.
Does that seem possible?
Also, what if I want to run a number of different Python programs? Can
I instantiate PythonVM more than once? Or if I need to run the same
Python program multiple times?
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Roman Chyla <ro...@gmail.com>.
Yes, and I can say it is working extremely well so far - we have done
and are doing some extensive benchmarking and tests. I also use
multiprocessing inside (python2.6) and I hope I would be able to
publish the source code soon, it could be re-usable. If you are
interested before that happens, please send me an email.
Best,
Roman
On Fri, Mar 4, 2011 at 7:27 AM, Andi Vajda <va...@apache.org> wrote:
>
> On Mar 3, 2011, at 21:50, Bill Janssen <ja...@parc.com> wrote:
>
>> New topic.
>>
>> I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
>> Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
>> with Tomcat.
>>
>> Now, am I going to have to do some trickery to get a VM? Or will
>> getVMEnv() just work with a previously initialized JVM?
>
> Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.
> I sent a long, detailed answer. I believe it was to Roman Chyla. A quick look in this mailing list archives should help you locate that thread and get answers to the above questions.
>
> Andi..
>
>>
>> Bill
>
>
Re: Using JCC / PyLucene with JEPP?
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> Not so long ago on this list someone asked about this, using python
> from java via jcc, something I've been doing with tomcat for a couple
> of years now. I sent a long, detailed answer. I believe it was to
> Roman Chyla. A quick look in this mailing list archives should help
> you locate that thread and get answers to the above questions.
Ah, right. I'd forgotten that you've added Java-side support to JCC.
Maybe I can do this without JEPP.
Bill
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Mon, 7 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>>> So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
>>> it's there. So I tried this simple program:
>>>
>>> import org.apache.jcc.PythonVM;
>>>
>>> public class test {
>>>
>>> public static void main (String[] argv) {
>>> PythonVM.start("/usr/bin/python",
>>> new String[] { "-c", "import time; print time.localtime()"});
>>> }
>>> }
>>
>> It seems that you think that this is going to run /usr/bin/python.
>> Not really.
>
> Right. I finally broke down and read the code. I see there's no way to
> do a "simple" test case.
>
> What the "instantiate" code does is to load a callable from a specified
> module, then call it, passing no arguments. That callable must return
> an instance of a Python class which explicitly extends a Java class, if
> the value is to be returned to Java. Otherwise, a value of "null" is
> returned. The callable may of course have side-effects in the Python
> world.
>
> It would be nice to be able to pass arguments to the callable.
>
> And there doesn't seem to be any way to modify Python's sys.path before
> one attempts to instantiate one's own Python class, either. Adding a
> PythonVM.exec() method would be a help there.
Patches welcome !
Andi..
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Mon, 7 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>>> So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
>>> it's there. So I tried this simple program:
>>>
>>> import org.apache.jcc.PythonVM;
>>>
>>> public class test {
>>>
>>> public static void main (String[] argv) {
>>> PythonVM.start("/usr/bin/python",
>>> new String[] { "-c", "import time; print time.localtime()"});
>>> }
>>> }
>>
>> It seems that you think that this is going to run /usr/bin/python.
>> Not really.
>
> Right. I finally broke down and read the code. I see there's no way to
> do a "simple" test case.
>
> What the "instantiate" code does is to load a callable from a specified
> module, then call it, passing no arguments. That callable must return
> an instance of a Python class which explicitly extends a Java class, if
> the value is to be returned to Java. Otherwise, a value of "null" is
> returned. The callable may of course have side-effects in the Python
> world.
>
> It would be nice to be able to pass arguments to the callable.
>
> And there doesn't seem to be any way to modify Python's sys.path before
> one attempts to instantiate one's own Python class, either. Adding a
> PythonVM.exec() method would be a help there.
>
>> To fix that, to help the dynamic linker to load that library, you need
>> to change the LFLAGS for 'darwin' in JCC's setup.py to also list the
>> python framework by adding '-framework', 'Python' to that list.
>> After rebuilding JCC, be sure then that otool lists libpython.dylib to
>> verify that it worked.
>
> Is there any reason not to make that change globally? Is there a
> downside to automatically building against the Python framework on OS X?
Yes, if one uses a virtual env or some other non-standard Python framework
location, it's better to let the python executable provide the correct
library than having to ensure that it matches in setup.py.
I believe, at first, it was that way, there was a -framework Python entry.
I then removed it because it was in the way, if I remember correctly.
Andi..
Re: using org.apache.jcc.PythonVM
Posted by Roman Chyla <ro...@gmail.com>.
On Mon, Mar 7, 2011 at 6:59 PM, Bill Janssen <ja...@parc.com> wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>> > So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
>> > it's there. So I tried this simple program:
>> >
>> > import org.apache.jcc.PythonVM;
>> >
>> > public class test {
>> >
>> > public static void main (String[] argv) {
>> > PythonVM.start("/usr/bin/python",
>> > new String[] { "-c", "import time; print time.localtime()"});
>> > }
>> > }
>>
>> It seems that you think that this is going to run /usr/bin/python.
>> Not really.
>
> Right. I finally broke down and read the code. I see there's no way to
> do a "simple" test case.
>
> What the "instantiate" code does is to load a callable from a specified
> module, then call it, passing no arguments. That callable must return
> an instance of a Python class which explicitly extends a Java class, if
> the value is to be returned to Java. Otherwise, a value of "null" is
> returned. The callable may of course have side-effects in the Python
> world.
>
> It would be nice to be able to pass arguments to the callable.
>
> And there doesn't seem to be any way to modify Python's sys.path before
> one attempts to instantiate one's own Python class, either. Adding a
> PythonVM.exec() method would be a help there.
I didn't read the whole tread, so with possibly giving misguided comments....
you may want to check if you are setting correctly those things:
-Djava.library.path=
if running non-standard python...
PYTHONHOME and LD_LIBRARY_PATH
I have:
export LD_LIBRARY_PATH=/opt/rchyla/python26/lib/:/opt/rchyla/python26/lib/python2.6/lib-dynload/
and when building on Mac, you can do
export JCC_FLAGS="-framework JavaVM, -framework Python"
roman
>
>> To fix that, to help the dynamic linker to load that library, you need
>> to change the LFLAGS for 'darwin' in JCC's setup.py to also list the
>> python framework by adding '-framework', 'Python' to that list.
>> After rebuilding JCC, be sure then that otool lists libpython.dylib to
>> verify that it worked.
>
> Is there any reason not to make that change globally? Is there a
> downside to automatically building against the Python framework on OS X?
>
> Bill
>
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> > So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
> > it's there. So I tried this simple program:
> >
> > import org.apache.jcc.PythonVM;
> >
> > public class test {
> >
> > public static void main (String[] argv) {
> > PythonVM.start("/usr/bin/python",
> > new String[] { "-c", "import time; print time.localtime()"});
> > }
> > }
>
> It seems that you think that this is going to run /usr/bin/python.
> Not really.
Right. I finally broke down and read the code. I see there's no way to
do a "simple" test case.
What the "instantiate" code does is to load a callable from a specified
module, then call it, passing no arguments. That callable must return
an instance of a Python class which explicitly extends a Java class, if
the value is to be returned to Java. Otherwise, a value of "null" is
returned. The callable may of course have side-effects in the Python
world.
It would be nice to be able to pass arguments to the callable.
And there doesn't seem to be any way to modify Python's sys.path before
one attempts to instantiate one's own Python class, either. Adding a
PythonVM.exec() method would be a help there.
> To fix that, to help the dynamic linker to load that library, you need
> to change the LFLAGS for 'darwin' in JCC's setup.py to also list the
> python framework by adding '-framework', 'Python' to that list.
> After rebuilding JCC, be sure then that otool lists libpython.dylib to
> verify that it worked.
Is there any reason not to make that change globally? Is there a
downside to automatically building against the Python framework on OS X?
Bill
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Fri, 4 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>> Something's off. libjcc.so is not shown in your list.
>> You need to solve that mystery before embedding can proceed.
>
> So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
> it's there. So I tried this simple program:
>
> import org.apache.jcc.PythonVM;
>
> public class test {
>
> public static void main (String[] argv) {
> PythonVM.start("/usr/bin/python",
> new String[] { "-c", "import time; print time.localtime()"});
> }
> }
It seems that you think that this is going to run /usr/bin/python.
Not really. The dynamic linker is going to load whatever libpython.dylib JCC
was linked with, provided it knows to do that (see below), then JCC is
going to call
Py_SetProgramName("/usr/bin/python") and
PySys_SetArgv({"/usr/bin/python", "-c", "import time; print time.localtime()")
This is done in jcc.cpp, in the _PythonVM_init() function.
No python code is going to be executed until you actually call instantiate()
with a python module and class name. These argv things are just stuffed into
sys.argv, not interpreted or anything.
> Here's what happens:
See below...
> % javac -classpath /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/jcc/classes:. test.java
> % java -Djava.library.path=/Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg -classpath /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/jcc/classes:. test
> Exception in thread "main" java.lang.UnsatisfiedLinkError: /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib: Symbol not found: __Py_NoneStruct Referenced from: /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib Expected in: dynamic lookup
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1824)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1748)
> at java.lang.Runtime.loadLibrary0(Runtime.java:822)
> at java.lang.System.loadLibrary(System.java:993)
> at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
> at test.main(test.java:6)
> % otool -L /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib
> /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib:
> @rpath/libjcc.dylib (compatibility version 2.6.0, current version 2.6.0)
> /System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM (compatibility version 1.0.0, current version 1.0.0)
> /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
> /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.5)
Notice how libpython.dylib (or whatever its correct name is) is not listed
here. This means that JCC was built to expect the main process to provide
libpython.dylib to the dynamic linker which works in the usual - starting in
a python process - case.
To fix that, to help the dynamic linker to load that library, you need to
change the LFLAGS for 'darwin' in JCC's setup.py to also list the python
framework by adding '-framework', 'Python' to that list.
After rebuilding JCC, be sure then that otool lists libpython.dylib to
verify that it worked.
Andi..
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> Something's off. libjcc.so is not shown in your list.
> You need to solve that mystery before embedding can proceed.
So, I went to my Mac, and looked for libjcc.dylib. Sure enough,
it's there. So I tried this simple program:
import org.apache.jcc.PythonVM;
public class test {
public static void main (String[] argv) {
PythonVM.start("/usr/bin/python",
new String[] { "-c", "import time; print time.localtime()"});
}
}
Here's what happens:
% javac -classpath /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/jcc/classes:. test.java
% java -Djava.library.path=/Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg -classpath /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/jcc/classes:. test
Exception in thread "main" java.lang.UnsatisfiedLinkError: /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib: Symbol not found: __Py_NoneStruct Referenced from: /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib Expected in: dynamic lookup
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1824)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1748)
at java.lang.Runtime.loadLibrary0(Runtime.java:822)
at java.lang.System.loadLibrary(System.java:993)
at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
at test.main(test.java:6)
% otool -L /Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib
/Library/Python/2.5/site-packages/JCC-2.6-py2.5-macosx-10.5-i386.egg/libjcc.dylib:
@rpath/libjcc.dylib (compatibility version 2.6.0, current version 2.6.0)
/System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.5)
%
Bill
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Bill Janssen <ja...@parc.com> wrote:
> Andi Vajda <va...@apache.org> wrote:
>
> > > I did patch setuptools, and as you can see below, the config.py says
> > > "Shared=True", so I believe I have shared mode enabled. I'm certainly
> > > using jcc with the "--shared" switch with no complaints.
> >
> > Something's off. libjcc.so is not shown in your list.
> > You need to solve that mystery before embedding can proceed.
> >
> > Andi..
>
> Looks like the patch failed, and the check doesn't notice that failure.
> I removed setuptools, then re-installed it, then went to build JCC
> again. It told me I needed the patch, so I tried to apply the patch per
> instructions:
>
> % sudo patch -d /usr/lib/python2.6/dist-packages -Nup0 < /tmp/pylucene/branches/branch_3x/jcc/jcc/patches/patch.43.0.6c11
> patching file setuptools/extension.py
> patching file setuptools/command/build_ext.py
> Hunk #1 FAILED at 85.
> Hunk #2 succeeded at 177 (offset 7 lines).
> Hunk #3 succeeded at 259 (offset 7 lines).
> 1 out of 3 hunks FAILED -- saving rejects to file setuptools/command/build_ext.py.rej
> %
>
> The EGG-INFO for "setuptools" says:
>
> Metadata-Version: 1.0
> Name: setuptools
> Version: 0.6c11
> Summary: xxxx
> Home-page: xxx
> Author: xxx
> Author-email: xxx
> License: xxx
> Description: xxx
>
> build_ext.py:get_ext_filename() is:
>
> def get_ext_filename(self, fullname):
> filename = _build_ext.get_ext_filename(self,fullname)
> if fullname not in self.ext_map:
> return filename
> ext = self.ext_map[fullname]
> if isinstance(ext,Library):
> fn, ext = os.path.splitext(filename)
> return self.shlib_compiler.library_filename(fn,libtype)
> elif use_stubs and ext._links_to_dynamic:
> d,fn = os.path.split(filename)
> return os.path.join(d,'dl-'+fn)
> else:
> return filename
>
> I believe the patched version should look like this:
>
> def get_ext_filename(self, fullname):
> filename = _build_ext.get_ext_filename(self,fullname)
> if fullname not in self.ext_map:
> return filename
> ext = self.ext_map[fullname]
> if isinstance(ext,Library):
> if ext.force_shared and not use_stubs:
> _libtype = 'shared'
> else:
> _libtype = libtype
> fn, ext = os.path.splitext(filename)
> return self.shlib_compiler.library_filename(fn,_libtype)
> elif use_stubs and ext._links_to_dynamic:
> d,fn = os.path.split(filename)
> return os.path.join(d,'dl-'+fn)
> else:
> return filename
After fixing get_ext_filename(), I now have libjcc.so on my Linux box.
Bill
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> > I did patch setuptools, and as you can see below, the config.py says
> > "Shared=True", so I believe I have shared mode enabled. I'm certainly
> > using jcc with the "--shared" switch with no complaints.
>
> Something's off. libjcc.so is not shown in your list.
> You need to solve that mystery before embedding can proceed.
>
> Andi..
Looks like the patch failed, and the check doesn't notice that failure.
I removed setuptools, then re-installed it, then went to build JCC
again. It told me I needed the patch, so I tried to apply the patch per
instructions:
% sudo patch -d /usr/lib/python2.6/dist-packages -Nup0 < /tmp/pylucene/branches/branch_3x/jcc/jcc/patches/patch.43.0.6c11
patching file setuptools/extension.py
patching file setuptools/command/build_ext.py
Hunk #1 FAILED at 85.
Hunk #2 succeeded at 177 (offset 7 lines).
Hunk #3 succeeded at 259 (offset 7 lines).
1 out of 3 hunks FAILED -- saving rejects to file setuptools/command/build_ext.py.rej
%
The EGG-INFO for "setuptools" says:
Metadata-Version: 1.0
Name: setuptools
Version: 0.6c11
Summary: xxxx
Home-page: xxx
Author: xxx
Author-email: xxx
License: xxx
Description: xxx
build_ext.py:get_ext_filename() is:
def get_ext_filename(self, fullname):
filename = _build_ext.get_ext_filename(self,fullname)
if fullname not in self.ext_map:
return filename
ext = self.ext_map[fullname]
if isinstance(ext,Library):
fn, ext = os.path.splitext(filename)
return self.shlib_compiler.library_filename(fn,libtype)
elif use_stubs and ext._links_to_dynamic:
d,fn = os.path.split(filename)
return os.path.join(d,'dl-'+fn)
else:
return filename
I believe the patched version should look like this:
def get_ext_filename(self, fullname):
filename = _build_ext.get_ext_filename(self,fullname)
if fullname not in self.ext_map:
return filename
ext = self.ext_map[fullname]
if isinstance(ext,Library):
if ext.force_shared and not use_stubs:
_libtype = 'shared'
else:
_libtype = libtype
fn, ext = os.path.splitext(filename)
return self.shlib_compiler.library_filename(fn,_libtype)
elif use_stubs and ext._links_to_dynamic:
d,fn = os.path.split(filename)
return os.path.join(d,'dl-'+fn)
else:
return filename
Right?
Bill
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Fri, 4 Mar 2011, Bill Janssen wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>> On Mar 4, 2011, at 10:56, Bill Janssen <ja...@parc.com> wrote:
>>
>>> Andi Vajda <va...@apache.org> wrote:
>>>
>>>> There are two shared libraries for jcc: _jcc.so, the python extension,
>>>> and libjcc.so, the shared mode runtime shared library. That's the one
>>>> that gets loaded by PythonVM.java. Its directory must be on java's
>>>> java.library.path.
>>>
>>> No such beast gets built or installed:
>>
>> If you didn't patch setuptools on linux as prompted by the jcc build
>> then you don't have shared mode enabled and no libjcc.so. Shared mode
>> is required for embedding.
>
> I did patch setuptools, and as you can see below, the config.py says
> "Shared=True", so I believe I have shared mode enabled. I'm certainly
> using jcc with the "--shared" switch with no complaints.
Something's off. libjcc.so is not shown in your list.
You need to solve that mystery before embedding can proceed.
Andi..
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> On Mar 4, 2011, at 10:56, Bill Janssen <ja...@parc.com> wrote:
>
> > Andi Vajda <va...@apache.org> wrote:
> >
> >> There are two shared libraries for jcc: _jcc.so, the python extension,
> >> and libjcc.so, the shared mode runtime shared library. That's the one
> >> that gets loaded by PythonVM.java. Its directory must be on java's
> >> java.library.path.
> >
> > No such beast gets built or installed:
>
> If you didn't patch setuptools on linux as prompted by the jcc build
> then you don't have shared mode enabled and no libjcc.so. Shared mode
> is required for embedding.
I did patch setuptools, and as you can see below, the config.py says
"Shared=True", so I believe I have shared mode enabled. I'm certainly
using jcc with the "--shared" switch with no complaints.
Bill
> > Here's what's in config.py:
> >
> > INCLUDES=['/usr/lib/jvm/java-6-openjdk/include', '/usr/lib/jvm/java-6-openjdk/include/linux']
> > CFLAGS=['-fno-strict-aliasing', '-Wno-write-strings']
> > DEBUG_CFLAGS=['-O0', '-g', '-DDEBUG']
> > LFLAGS=['-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64', '-ljava', '-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server', '-ljvm', '-Wl,-rpath=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server']
> > IMPLIB_LFLAGS=[]
> > SHARED=True
> > VERSION="2.7"
> >
> > There *is* a libjcc.a.
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Mar 4, 2011, at 10:56, Bill Janssen <ja...@parc.com> wrote:
> Andi Vajda <va...@apache.org> wrote:
>
>> There are two shared libraries for jcc: _jcc.so, the python extension,
>> and libjcc.so, the shared mode runtime shared library. That's the one
>> that gets loaded by PythonVM.java. Its directory must be on java's
>> java.library.path.
>
> No such beast gets built or installed:
If you didn't patch setuptools on linux as prompted by the jcc build then you don't have shared mode enabled and no libjcc.so. Shared mode is required for embedding.
Andi..
>
> % find /usr/local/lib/python2.6 -name \*.so
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.so
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.so
> /usr/local/lib/python2.6/dist-packages/lucene-3.0.2-py2.6-linux-x86_64.egg/lucene/_lucene.so
> % find /tmp/pylucene/branches/branch_3x/ -name \*.so
> /tmp/pylucene/branches/branch_3x/jcc/build/lib.linux-x86_64-2.6/jcc/_jcc.so
> %
>
> Here's what's in config.py:
>
> INCLUDES=['/usr/lib/jvm/java-6-openjdk/include', '/usr/lib/jvm/java-6-openjdk/include/linux']
> CFLAGS=['-fno-strict-aliasing', '-Wno-write-strings']
> DEBUG_CFLAGS=['-O0', '-g', '-DDEBUG']
> LFLAGS=['-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64', '-ljava', '-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server', '-ljvm', '-Wl,-rpath=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server']
> IMPLIB_LFLAGS=[]
> SHARED=True
> VERSION="2.7"
>
> There *is* a libjcc.a.
>
> Bill
Re: using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:
> There are two shared libraries for jcc: _jcc.so, the python extension,
> and libjcc.so, the shared mode runtime shared library. That's the one
> that gets loaded by PythonVM.java. Its directory must be on java's
> java.library.path.
No such beast gets built or installed:
% find /usr/local/lib/python2.6 -name \*.so
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.so
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.so
/usr/local/lib/python2.6/dist-packages/lucene-3.0.2-py2.6-linux-x86_64.egg/lucene/_lucene.so
% find /tmp/pylucene/branches/branch_3x/ -name \*.so
/tmp/pylucene/branches/branch_3x/jcc/build/lib.linux-x86_64-2.6/jcc/_jcc.so
%
Here's what's in config.py:
INCLUDES=['/usr/lib/jvm/java-6-openjdk/include', '/usr/lib/jvm/java-6-openjdk/include/linux']
CFLAGS=['-fno-strict-aliasing', '-Wno-write-strings']
DEBUG_CFLAGS=['-O0', '-g', '-DDEBUG']
LFLAGS=['-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64', '-ljava', '-L/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server', '-ljvm', '-Wl,-rpath=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server']
IMPLIB_LFLAGS=[]
SHARED=True
VERSION="2.7"
There *is* a libjcc.a.
Bill
Re: using org.apache.jcc.PythonVM
Posted by Andi Vajda <va...@apache.org>.
On Fri, 4 Mar 2011, Bill Janssen wrote:
> Hmmm, having this difficulty:
>
> % find /usr/local/lib/python2.6 -name \*jcc\*
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/libjcc.a
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.pyc
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.py
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.so
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/sources/jccfuncs.h
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/sources/jcc.cpp
> /usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/classes/org/apache/jcc
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/libjcc.a
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.pyc
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.py
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.so
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/sources/jccfuncs.h
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/sources/jcc.cpp
> /usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/classes/org/apache/jcc
> % java -classpath .:/tmp/pylucene/branches/branch_3x/jcc/build/lib.linux-x86_64-2.6/jcc/classes -Djava.library.path=/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc test /u/tests/docs/testpage.txt /u/tests/docs/testpage.txt
> Exception in thread "main" java.lang.UnsatisfiedLinkError: no jcc in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681)
> at java.lang.Runtime.loadLibrary0(Runtime.java:840)
> at java.lang.System.loadLibrary(System.java:1047)
> at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
> at test.main(test.java:11)
> %
>
> The shared library is "_jcc.so", but the Java code loads "jcc". Is that
> a problem?
There are two shared libraries for jcc: _jcc.so, the python extension, and
libjcc.so, the shared mode runtime shared library. That's the one that
gets loaded by PythonVM.java. Its directory must be on java's java.library.path.
$ find _install/lib/python2.6/site-packages/JCC-2.7-py2.6-linux-x86_64.egg | grep "\.so"
_install/lib/python2.6/site-packages/JCC-2.7-py2.6-linux-x86_64.egg/libjcc.so
_install/lib/python2.6/site-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.so
Andi..
using org.apache.jcc.PythonVM
Posted by Bill Janssen <ja...@parc.com>.
Hmmm, having this difficulty:
% find /usr/local/lib/python2.6 -name \*jcc\*
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/libjcc.a
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.pyc
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.py
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/_jcc.so
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/sources/jccfuncs.h
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/sources/jcc.cpp
/usr/local/lib/python2.6/dist-packages/JCC-2.7-py2.6-linux-x86_64.egg/jcc/classes/org/apache/jcc
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/libjcc.a
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.pyc
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.py
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/_jcc.so
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/sources/jccfuncs.h
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/sources/jcc.cpp
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc/classes/org/apache/jcc
% java -classpath .:/tmp/pylucene/branches/branch_3x/jcc/build/lib.linux-x86_64-2.6/jcc/classes -Djava.library.path=/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/jcc test /u/tests/docs/testpage.txt /u/tests/docs/testpage.txt
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jcc in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681)
at java.lang.Runtime.loadLibrary0(Runtime.java:840)
at java.lang.System.loadLibrary(System.java:1047)
at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
at test.main(test.java:11)
%
The shared library is "_jcc.so", but the Java code loads "jcc". Is that
a problem?
Bill
Re: Using JCC / PyLucene with JEPP?
Posted by Andi Vajda <va...@apache.org>.
On Mar 3, 2011, at 21:50, Bill Janssen <ja...@parc.com> wrote:
> New topic.
>
> I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
> Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
> with Tomcat.
>
> Now, am I going to have to do some trickery to get a VM? Or will
> getVMEnv() just work with a previously initialized JVM?
Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.
I sent a long, detailed answer. I believe it was to Roman Chyla. A quick look in this mailing list archives should help you locate that thread and get answers to the above questions.
Andi..
>
> Bill