You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2010/01/06 04:28:57 UTC
Threads, revisited
Greets,
Recent changes to VTable and the addition of LockFreeRegistry have made Lucy's
core OO framework thread safe. There are still a couple of TODO items:
* JSON::XS is not thread safe, and we need to replace it with our own custom
JSON parsing module.
* Lucy->error must be made into a thread local and tested as such.
Nevertheless, we are close to being able to advertise basic support for
threads.
In my opinion, we should stop there.
Here is a documentation class I would like to add, Lucy::Docs::ThreadSupport.
/** Using Lucy with threads.
*
* Lucy's primary concurrency model is processes. Threads are also
* supported, so long as those threads look like processes -- meaning that
* objects must not be shared across threads.
*
* For illustration: It is safe to start up several threads, each with its
* own search object, and have those searchers operate concurrently
* against the same index. However it is not safe (and will eventually
* result in segfaults or other severe memory errors) to start up a search
* object in a "boss" thread and then search against that object in
* several worker threads.
*/
inert class Lucy::Docs::ThreadSupport { }
The advantage of stopping at that level of support is that we can provide our
users with a rule which is clear and for practical purposes absolute:
Don't share objects across threads -- period.
There are other classes besides VTable which it might be nice to make thread
safe, like Folder -- but the instant we start down that path, we destroy the
clarity of our "don't share" rule.
There are two use cases where threads would be particularly useful, and it's
worth addressing those specifically.
First... during indexing, it is faster if you can offload analysis to worker
threads running on multiple CPUs. However, having implemented a
BackgroundMerger for KinoSearch which can operate simultaneously with an
Indexer, I am now reasonably confident that we can achieve this using multiple
Indexer objects in separate processes (or threads) writing to separate
segments.
Second... at search time, it would be useful to dedicate one CPU to each index
segment to achieve the fastest possible search response. For this, I don't
have as good an answer, but I think we will be able to max out speed on some
systems using non-portable IPC techniques. If we can achieve no-compromises
performance on a few Unixen that way, I think that's good enough for me.
Prior disussion:
http://markmail.org/thread/eyjvfc72r653t76i
https://issues.apache.org/jira/browse/LUCENE-2026
Marvin Humphrey
Re: [Lucy] Threads, revisited
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Jan 05, 2010 at 09:50:07PM -0600, Peter Karman wrote:
> Marvin Humphrey wrote on 1/5/10 9:28 PM:
> > There are still a couple of TODO items:
> >
> > * JSON::XS is not thread safe, and we need to replace it with our own
> > custom JSON parsing module.
>
> isn't that necessary anyway for non-Perl host languages?
Yes, it's just been a low priority because JSON::XS is a solid and
high-performing module.
I also don't have a burning desire to implement threads, especially since
Perl's ithreads have a lot of downsides and aren't very popular. Thread
support will be crucial for Java bindings, though -- JVM startup time is
significant enough that using processes as your concurrency model has serious
drawbacks compared to threads. And other languages have better thread support
than Perl.
The main reason I went and worked on threads now was because McCandless and I
arrived at something like consensus over in the LUCENE-2026 thread. I've
always thought it was important to have a concurrency plan. Now that plan is
firming up.
> Is there enough C in JSON::XS we could borrow as a starting point?
Possible, but if we roll our own I think we should just work from scratch off
the spec, avoiding potential IP issues.
> Or, is the MIT license compatible with the Apache license, making it
> possible to use something like http://oss.metaparadigm.com/json-c/ ?
Possible. It wouldn't necessarily be tailored to our data types, so we might
not be able to squeeze out absolute maximum performance. However, it's the
next logical intermediate step after moving away from host-specific libraries
like JSON::XS.
OTOH, I hate dependencies, because it so hard to find something suitable
that's at least as platform compatible or more so than the library it would be
supporting. (CPAN::Testers results for KinoSearch 0.30_072 so far: 32 passes,
0 fails.) And in the C world, build scripts are wildly diverse and wildly
platform incompatible.
> Just seems like there should be a standalone, Apache-friendly JSON
> reader/writer available in C.
There probably is, but the JSON spec is awesome for its simplicity. It's not
that hard to write a parser.
> >In my opinion, we should stop there.
>
> ++
Cool, looks like consensus on the way.
Marvin Humphrey
Re: [Lucy] Threads, revisited
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 1/5/10 9:28 PM:
> There are still a couple of TODO items:
>
> * JSON::XS is not thread safe, and we need to replace it with our own custom
> JSON parsing module.
isn't that necessary anyway for non-Perl host languages?
Is there enough C in JSON::XS we could borrow as a starting point?
Or, is the MIT license compatible with the Apache license, making it possible to
use something like http://oss.metaparadigm.com/json-c/ ?
Just seems like there should be a standalone, Apache-friendly JSON reader/writer
available in C.
> * Lucy->error must be made into a thread local and tested as such.
>
> Nevertheless, we are close to being able to advertise basic support for
> threads.
>
> In my opinion, we should stop there.
>
++
--
Peter Karman . http://peknet.com/ . peter@peknet.com