You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by "h.g. g.h." <hi...@gmail.com> on 2011/07/15 02:05:09 UTC
PyLucene, multiprocessing, high thread count
Hi,
I am trying to run parallel lucene queries using PyLucene and Python's
Multiprocessing package. I run it on a large server machine with thousands
of cores. The problem that I am running into is that it is a shared machine
so I have to request a fixed number of CPUs per job (< 100). Apparently,
each process that runs Lucene queries, generates a very large number of
threads and so even with 50 processes (if I request 50 cpus), the number of
threads becomes > 500. I need to limit the thread-count so it doesn't
overshoot the allotted CPU resources.
Is there a way to limit the number of threads that Lucene/Pylucene is
allowed to create?
Thanks for your help and time.
Best Regards,
hg
Re: PyLucene, multiprocessing, high thread count
Posted by Andi Vajda <va...@apache.org>.
On Mon, 18 Jul 2011, h.g. g.h. wrote:
>>> The staff here who run parallel/multithreaded java code use the cmd
>> "java
>>> -XX:ParallelGCThreads=16 JavaApp". I tried to pass the same argument as
>>> "initVM(vmargs="-XX:ParallelGCThreads=4", but it doesn't obey. Am I
>> missing
>>> something here, or misusing it, may be??
>>
>> What are your assumptions here ?
>>
>
> I think my assumption here is that -XX:ParallelGCThreads is a keyword
> argument that vmarg can convey while initializing the JVM...
Maybe it has a lower bounds ?
>> Maybe Python isn't adapted to the constraints you must run in ? Why not use
>> Java directly ?
>>
>>
>
> We're not using Java since the rest of the application is already all coded
> up in Python... We decided to venture into Lucene only very recently and
> thought of using the PyLucene interface...
> The only constraint really is that the application should run in the
> resources allocated to it, and not based on the totality of all resources
> available on the shared machine...
Ok, then using threads instead of subprocesses is your best bet.
Andi..
>
>
>
>> Andi..
>>
>>>
>>> Himanshu
>>>
>>>
>>> On Fri, Jul 15, 2011 at 9:47 AM, Christian Heimes <li...@cheimes.de>
>> wrote:
>>>
>>>> Am 15.07.2011 10:10, schrieb Andi Vajda:
>>>>> PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is
>>>> created with all its threads. This can get insane pretty quickly.
>>>>
>>>> The Java VM starts a lot of threads. On my Linux box eleven threads
>>>> additional threads are running after initVM() has been called.
>>>>
>>>>>>> import lucene, os, psutil
>>>>>>> psutil.Process(os.getpid()).get_num_threads()
>>>> 1
>>>>>>> lucene.initVM()
>>>> <jcc.JCCEnv object at 0x7f23a66f31e0>
>>>>>>> psutil.Process(os.getpid()).get_num_threads()
>>>> 12
>>>>
>>>> Christian
>>>>
>>
>
Re: PyLucene, multiprocessing, high thread count
Posted by "h.g. g.h." <hi...@gmail.com>.
On Sat, Jul 16, 2011 at 5:41 AM, Andi Vajda <va...@apache.org> wrote:
>
> On Jul 16, 2011, at 2:53, "h.g. g.h." <hi...@gmail.com> wrote:
>
> > Thanks a lot for your replies, Andi and Christian. This is exactly what I
> > saw... about 11 threads per python process that was making lucene
> queries...
> >
> > I have a follow-up question now (may be even naive):
> > Andi, about your suggestion of using multithreading and that PyLucene
> will
> > release GIL when it calls into JVM... So, when it does that, another
> python
> > thread will acquire GIL, and then call into JVM, and then another
> thread...
> > so on and so forth. Will this again not lead to too many Java threads
> > running in parallel?? Did I misunderstand what you were suggesting?
>
> It depends how long the queries are running for. It's true that only one
> Python thread can start queries at a time because it holds the GIL. If these
> queries are difficult enough and run for a long time, decent concurrency can
> still be achieved.
>
> > The staff here who run parallel/multithreaded java code use the cmd
> "java
> > -XX:ParallelGCThreads=16 JavaApp". I tried to pass the same argument as
> > "initVM(vmargs="-XX:ParallelGCThreads=4", but it doesn't obey. Am I
> missing
> > something here, or misusing it, may be??
>
> What are your assumptions here ?
>
I think my assumption here is that -XX:ParallelGCThreads is a keyword
argument that vmarg can convey while initializing the JVM...
> Maybe Python isn't adapted to the constraints you must run in ? Why not use
> Java directly ?
>
>
We're not using Java since the rest of the application is already all coded
up in Python... We decided to venture into Lucene only very recently and
thought of using the PyLucene interface...
The only constraint really is that the application should run in the
resources allocated to it, and not based on the totality of all resources
available on the shared machine...
> Andi..
>
> >
> > Himanshu
> >
> >
> > On Fri, Jul 15, 2011 at 9:47 AM, Christian Heimes <li...@cheimes.de>
> wrote:
> >
> >> Am 15.07.2011 10:10, schrieb Andi Vajda:
> >>> PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is
> >> created with all its threads. This can get insane pretty quickly.
> >>
> >> The Java VM starts a lot of threads. On my Linux box eleven threads
> >> additional threads are running after initVM() has been called.
> >>
> >>>>> import lucene, os, psutil
> >>>>> psutil.Process(os.getpid()).get_num_threads()
> >> 1
> >>>>> lucene.initVM()
> >> <jcc.JCCEnv object at 0x7f23a66f31e0>
> >>>>> psutil.Process(os.getpid()).get_num_threads()
> >> 12
> >>
> >> Christian
> >>
>
Re: PyLucene, multiprocessing, high thread count
Posted by Andi Vajda <va...@apache.org>.
On Jul 16, 2011, at 2:53, "h.g. g.h." <hi...@gmail.com> wrote:
> Thanks a lot for your replies, Andi and Christian. This is exactly what I
> saw... about 11 threads per python process that was making lucene queries...
>
> I have a follow-up question now (may be even naive):
> Andi, about your suggestion of using multithreading and that PyLucene will
> release GIL when it calls into JVM... So, when it does that, another python
> thread will acquire GIL, and then call into JVM, and then another thread...
> so on and so forth. Will this again not lead to too many Java threads
> running in parallel?? Did I misunderstand what you were suggesting?
It depends how long the queries are running for. It's true that only one Python thread can start queries at a time because it holds the GIL. If these queries are difficult enough and run for a long time, decent concurrency can still be achieved.
> The staff here who run parallel/multithreaded java code use the cmd "java
> -XX:ParallelGCThreads=16 JavaApp". I tried to pass the same argument as
> "initVM(vmargs="-XX:ParallelGCThreads=4", but it doesn't obey. Am I missing
> something here, or misusing it, may be??
What are your assumptions here ?
Maybe Python isn't adapted to the constraints you must run in ? Why not use Java directly ?
Andi..
>
> Himanshu
>
>
> On Fri, Jul 15, 2011 at 9:47 AM, Christian Heimes <li...@cheimes.de> wrote:
>
>> Am 15.07.2011 10:10, schrieb Andi Vajda:
>>> PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is
>> created with all its threads. This can get insane pretty quickly.
>>
>> The Java VM starts a lot of threads. On my Linux box eleven threads
>> additional threads are running after initVM() has been called.
>>
>>>>> import lucene, os, psutil
>>>>> psutil.Process(os.getpid()).get_num_threads()
>> 1
>>>>> lucene.initVM()
>> <jcc.JCCEnv object at 0x7f23a66f31e0>
>>>>> psutil.Process(os.getpid()).get_num_threads()
>> 12
>>
>> Christian
>>
Re: PyLucene, multiprocessing, high thread count
Posted by "h.g. g.h." <hi...@gmail.com>.
Thanks a lot for your replies, Andi and Christian. This is exactly what I
saw... about 11 threads per python process that was making lucene queries...
I have a follow-up question now (may be even naive):
Andi, about your suggestion of using multithreading and that PyLucene will
release GIL when it calls into JVM... So, when it does that, another python
thread will acquire GIL, and then call into JVM, and then another thread...
so on and so forth. Will this again not lead to too many Java threads
running in parallel?? Did I misunderstand what you were suggesting?
The staff here who run parallel/multithreaded java code use the cmd "java
-XX:ParallelGCThreads=16 JavaApp". I tried to pass the same argument as
"initVM(vmargs="-XX:ParallelGCThreads=4", but it doesn't obey. Am I missing
something here, or misusing it, may be??
Himanshu
On Fri, Jul 15, 2011 at 9:47 AM, Christian Heimes <li...@cheimes.de> wrote:
> Am 15.07.2011 10:10, schrieb Andi Vajda:
> > PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is
> created with all its threads. This can get insane pretty quickly.
>
> The Java VM starts a lot of threads. On my Linux box eleven threads
> additional threads are running after initVM() has been called.
>
> >>> import lucene, os, psutil
> >>> psutil.Process(os.getpid()).get_num_threads()
> 1
> >>> lucene.initVM()
> <jcc.JCCEnv object at 0x7f23a66f31e0>
> >>> psutil.Process(os.getpid()).get_num_threads()
> 12
>
> Christian
>
Re: PyLucene, multiprocessing, high thread count
Posted by Christian Heimes <li...@cheimes.de>.
Am 15.07.2011 10:10, schrieb Andi Vajda:
> PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is created with all its threads. This can get insane pretty quickly.
The Java VM starts a lot of threads. On my Linux box eleven threads
additional threads are running after initVM() has been called.
>>> import lucene, os, psutil
>>> psutil.Process(os.getpid()).get_num_threads()
1
>>> lucene.initVM()
<jcc.JCCEnv object at 0x7f23a66f31e0>
>>> psutil.Process(os.getpid()).get_num_threads()
12
Christian
Re: PyLucene, multiprocessing, high thread count
Posted by Andi Vajda <va...@apache.org>.
Hi,
On Jul 15, 2011, at 2:05, "h.g. g.h." <hi...@gmail.com> wrote:
> I am trying to run parallel lucene queries using PyLucene and Python's
> Multiprocessing package. I run it on a large server machine with thousands
> of cores. The problem that I am running into is that it is a shared machine
> so I have to request a fixed number of CPUs per job (< 100). Apparently,
> each process that runs Lucene queries, generates a very large number of
> threads and so even with 50 processes (if I request 50 cpus), the number of
> threads becomes > 500. I need to limit the thread-count so it doesn't
> overshoot the allotted CPU resources.
PyLucene embeds a Java VM. Thus, with each subprocess, a new JVM is created with all its threads. This can get insane pretty quickly.
Instead, you should run one process and use Java threads to parallelize your queries. Contrary to Python, Java threads run fully concurrently and PyLucene releases the Python GIL whenever it calls into the JVM. To see an example on how to use Java threads from PyLucene, please refer to the test file using threads in the test directory: http://svn.apache.org/repos/asf/lucene/pylucene/branches/pylucene_3_2/test/test_PyLuceneThread.py
> Is there a way to limit the number of threads that Lucene/Pylucene is
> allowed to create?
Not that I know of, no.
Andi..
> Thanks for your help and time.
>
> Best Regards,
> hg