Returning snippets with results

["Jana, Kumar Raja" <kj...@ptc.com>]

Hi,
 
I want to get snippets along with my results. For this, I use the
Highlighting Feature to return the context of fragment size 10. 
 
Some of the documents are very large (over 30 MB) in size and the
Highlighting feature works only for stored fields. So this makes it
necessary for me to store the content of the these huge documents to get
the snippets.
 
Is there any other way to get the snippets without storing the entire
content??
 
 
Thanks,
Kumar
 

RE: Returning snippets with results

["Jana, Kumar Raja" <kj...@ptc.com>]

Hi Grant,

Thanks for the help. I have decided to store only the first MB in Solr
and return snippets for results matching within that MB. For the rest of
the results, tough luck!!!!



-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org] 
Sent: Saturday, December 06, 2008 6:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Returning snippets with results

I don't think there is, since storage (or term vectors, but that likely
won't save you any space) is the only place that Solr has the content
stored in the correct "order".  Namely, for searching, documents are
split up into an inverted index and it is really cumbersome to recreate
a document from the inverted index (and likely not even possible
depending on Analysis)

Some alternatives might be to split up your documents into smaller
chunks.

I believe there was some work/discussion on large Document highlighting
over on the Lucene mailing list.  I'd suggest looking through the
java-user@lucene.apache.org archives (via MarkMail or Nabble or one of
those) for "large document highlighting"

-Grant

On Dec 6, 2008, at 4:26 AM, Jana, Kumar Raja wrote:

> Hi,
>
> I want to get snippets along with my results. For this, I use the 
> Highlighting Feature to return the context of fragment size 10.
>
> Some of the documents are very large (over 30 MB) in size and the 
> Highlighting feature works only for stored fields. So this makes it 
> necessary for me to store the content of the these huge documents to 
> get the snippets.
>
> Is there any other way to get the snippets without storing the entire 
> content??
>
>
> Thanks,
> Kumar
>

--------------------------
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Re: Returning snippets with results

[Grant Ingersoll <gs...@apache.org>]

I don't think there is, since storage (or term vectors, but that  
likely won't save you any space) is the only place that Solr has the  
content stored in the correct "order".  Namely, for searching,  
documents are split up into an inverted index and it is really  
cumbersome to recreate a document from the inverted index (and likely  
not even possible depending on Analysis)

Some alternatives might be to split up your documents into smaller  
chunks.

I believe there was some work/discussion on large Document  
highlighting over on the Lucene mailing list.  I'd suggest looking  
through the java-user@lucene.apache.org archives (via MarkMail or  
Nabble or one of those) for "large document highlighting"

-Grant

On Dec 6, 2008, at 4:26 AM, Jana, Kumar Raja wrote:

> Hi,
>
> I want to get snippets along with my results. For this, I use the
> Highlighting Feature to return the context of fragment size 10.
>
> Some of the documents are very large (over 30 MB) in size and the
> Highlighting feature works only for stored fields. So this makes it
> necessary for me to store the content of the these huge documents to  
> get
> the snippets.
>
> Is there any other way to get the snippets without storing the entire
> content??
>
>
> Thanks,
> Kumar
>

--------------------------
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ