Serialization of Lucene Document objects

[Erik Fäßler <er...@uni-jena.de>]

  Hi there,

I'd like to serialize some Lucene Documents I've built before. My goal 
is to send the documents over a http connection to a Solr server which 
then should add them to its index.

I thought this would work as the Document class implements Serializable 
as do the Fields. Unfortunately, the serialization fails because the 
TokenStream class is not serializable. But if you create Fields on 
TokenStreams, these are stored in the Field objects and so the 
serialization does not seem possible.

Did I miss something? Or is it just the case that you can serialize 
documents whose fields only have String values (I didn't check if this 
works indeed, but I guess so) and no documents which need a TokenStream?

Other ideas how to archive my goal? I already have a solution where I 
let Solr build the document. This is too slow, however, I'd like to 
scale to document construction, so I'd rather want to do as much as work 
before as possible (my data is rather large and of a bit complicated 
structure).

Thanks for any hints!

Regards,

     Erik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Serialization of Lucene Document objects

[Erik Fäßler <er...@uni-jena.de>]

  Hi Simon,

thanks for your answer. My comments below:
> so you mean you would want to do that analysis on the client side and
> only shoot the already tokenized values to the server?
> What exactly is too slow? Can you provide more info what the problem is?
>
> After all I think you should ask on the solr-user list instead.
The point is, I'm using some quite sophisticated NLP pipeline which 
outputs data I'd like to index. I have a component which maps this data 
structure (actually a UIMA CAS object) to lucene documents. A lot of 
things are done with the input data, including some quite custom 
adaptions of position_increments and aligning several other TokenStreams 
in terms of again position_increment and position_offset.
I cannot do such things with the native Solr XML format because I need 
several fields with the same name but different indexing / storing 
options. This is because I enrich my documents' texts with meta data 
extracted by my pipeline. So a field gets much more terms then could 
have been extracted from the text by Lucene/Solr analysis. Solr 
approximates this capability by multi-valued fields but this can't work 
the same.

I measured the timings for batches of 1000 documents sent to Solr.  I am 
sending the whole UIMA CAS in a serialized form which is a quite verbose 
XMI format.
Processing 1000 documents in Solr then takes
* approx. 11sec for deserialization
* approx. 4sec for mapping to a document
* less then 1sec for writing the documents to the index.

So most of the times gets lost by deserialization. By sending the Lucene 
documents directly I hope to reduce this overhead greatly as I'm not 
sending the verbose raw data but an already condensed form.
Second, the time for the mapping still takes some time for work which 
not necessarily has to be done on the server side. I can scale the 
clients arbitrarily so they should do most of the work.

This is why I'd like to build the Lucene documents on the client side 
and just send them to server. But now I wonder if this is possible at 
all after the serialization of lucene documents failed...

Sorry for the long read and thanks for you help :)

Erik
> Simon
>> Thanks for any hints!
>>
>> Regards,
>>
>>     Erik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Serialization of Lucene Document objects

[Simon Willnauer <si...@googlemail.com>]

On Tue, Feb 22, 2011 at 2:58 PM, Erik Fäßler <er...@uni-jena.de> wrote:
>  Hi there,
>
> I'd like to serialize some Lucene Documents I've built before. My goal is to
> send the documents over a http connection to a Solr server which then should
> add them to its index.

ok so why do you build lucene documents if you use solr? Can you use
SolrJ which does all the serialization work for you?!
>
> I thought this would work as the Document class implements Serializable as
> do the Fields. Unfortunately, the serialization fails because the
> TokenStream class is not serializable. But if you create Fields on
> TokenStreams, these are stored in the Field objects and so the serialization
> does not seem possible.
>
> Did I miss something? Or is it just the case that you can serialize
> documents whose fields only have String values (I didn't check if this works
> indeed, but I guess so) and no documents which need a TokenStream?

Again solr should do all the tokenization work for you - no need to
use a TokenSTream on the client side.
>
> Other ideas how to archive my goal? I already have a solution where I let
> Solr build the document. This is too slow, however, I'd like to scale to
> document construction, so I'd rather want to do as much as work before as
> possible (my data is rather large and of a bit complicated structure).
so you mean you would want to do that analysis on the client side and
only shoot the already tokenized values to the server?
What exactly is too slow? Can you provide more info what the problem is?

After all I think you should ask on the solr-user list instead.

Simon
>
> Thanks for any hints!
>
> Regards,
>
>    Erik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org