Re: [Lucy] Threads, revisited

[Peter Karman <pe...@peknet.com>]

Marvin Humphrey wrote on 1/5/10 9:28 PM:
>  There are still a couple of TODO items:
> 
>   * JSON::XS is not thread safe, and we need to replace it with our own custom 
>     JSON parsing module.

isn't that necessary anyway for non-Perl host languages?

Is there enough C in JSON::XS we could borrow as a starting point?
Or, is the MIT license compatible with the Apache license, making it possible to 
use something like http://oss.metaparadigm.com/json-c/ ?

Just seems like there should be a standalone, Apache-friendly JSON reader/writer 
available in C.

>   * Lucy->error must be made into a thread local and tested as such.
> 
> Nevertheless, we are close to being able to advertise basic support for
> threads.
> 
> In my opinion, we should stop there.
> 

++

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [Lucy] Threads, revisited

[Marvin Humphrey <ma...@rectangular.com>]

On Tue, Jan 05, 2010 at 09:50:07PM -0600, Peter Karman wrote:
> Marvin Humphrey wrote on 1/5/10 9:28 PM:
> > There are still a couple of TODO items:
> >
> >  * JSON::XS is not thread safe, and we need to replace it with our own 
> >  custom JSON parsing module.
> 
> isn't that necessary anyway for non-Perl host languages?

Yes, it's just been a low priority because JSON::XS is a solid and
high-performing module.

I also don't have a burning desire to implement threads, especially since
Perl's ithreads have a lot of downsides and aren't very popular.  Thread
support will be crucial for Java bindings, though -- JVM startup time is
significant enough that using processes as your concurrency model has serious
drawbacks compared to threads.  And other languages have better thread support
than Perl.

The main reason I went and worked on threads now was because McCandless and I
arrived at something like consensus over in the LUCENE-2026 thread.  I've
always thought it was important to have a concurrency plan.  Now that plan is
firming up.

> Is there enough C in JSON::XS we could borrow as a starting point?

Possible, but if we roll our own I think we should just work from scratch off
the spec, avoiding potential IP issues.

> Or, is the MIT license compatible with the Apache license, making it 
> possible to use something like http://oss.metaparadigm.com/json-c/ ?

Possible.  It wouldn't necessarily be tailored to our data types, so we might
not be able to squeeze out absolute maximum performance.  However, it's the
next logical intermediate step after moving away from host-specific libraries
like JSON::XS.

OTOH, I hate dependencies, because it so hard to find something suitable
that's at least as platform compatible or more so than the library it would be
supporting.  (CPAN::Testers results for KinoSearch 0.30_072 so far: 32 passes,
0 fails.)  And in the C world, build scripts are wildly diverse and wildly
platform incompatible.

> Just seems like there should be a standalone, Apache-friendly JSON 
> reader/writer available in C.

There probably is, but the JSON spec is awesome for its simplicity.  It's not
that hard to write a parser.

> >In my opinion, we should stop there.
> 
> ++

Cool, looks like consensus on the way.

Marvin Humphrey