external fields storage

[Andrey Klochkov <ak...@griddynamics.com>]

Hi Solr users

Our index could be much smaller if we could store some of fields not in
index directly but in some kind of external storage.
All I've found until now is ExternalFileField class which shows that it's
possible to implement such a storage, but I'm quite sure that the
requirement is common and there should be some existing implementations.
Also it would be good to be able to search using these fields, to include
them in the search result sets and to update them with standard Solr update
handlers.

-- 
Andrew Klochkov

Re: external fields storage

[Mark Miller <ma...@gmail.com>]

Andrey Klochkov wrote:
> On Tue, Mar 24, 2009 at 4:43 PM, Mark Miller <ma...@gmail.com> wrote:
>
>   
>> Thats a tall order. It almost sounds as if you want to be able to not use
>> the index to store fields, but have them still fully functional as if
>> indexed. That would be quite the magic trick.
>>     
>
>
> Look here, people wanted exactly the same feature in 2004. Is it still not
> implemented?
>
> http://www.gossamer-threads.com/lists/lucene/java-user/8672
>
> --
> Andrew Klochkov
>
>   
Right - I was exaggerating your description a bit. It reads as if you 
want it to have all the same power as an indexed field. So I made a bad 
joke. If you want to be able to search the field, its index entry needs 
to be updated anyway. I don't see how you get search on external stored 
fields without having to update and keep them in the index - external 
field storage is simple to add on your own, either using that skwish 
library, or even a basic database. You can then do id to offset mapping 
like that guy is looking for - simply add the id to Lucene and do your 
content updates with the external db.

-- 
- Mark

http://www.lucidimagination.com

Re: external fields storage

[Andrey Klochkov <ak...@griddynamics.com>]

On Tue, Mar 24, 2009 at 4:43 PM, Mark Miller <ma...@gmail.com> wrote:

> Thats a tall order. It almost sounds as if you want to be able to not use
> the index to store fields, but have them still fully functional as if
> indexed. That would be quite the magic trick.


Look here, people wanted exactly the same feature in 2004. Is it still not
implemented?

http://www.gossamer-threads.com/lists/lucene/java-user/8672

--
Andrew Klochkov

Re: external fields storage

[Andrey Klochkov <ak...@griddynamics.com>]

>
>>
>> Our index could be much smaller if we could store some of fields not in
>> index directly but in some kind of external storage.
>> All I've found until now is ExternalFileField class which shows that it's
>> possible to implement such a storage, but I'm quite sure that the
>> requirement is common and there should be some existing implementations.
>> Also it would be good to be able to search using these fields, to include
>> them in the search result sets and to update them with standard Solr
>> update
>> handlers.
>>
>>
>>
> Thats a tall order. It almost sounds as if you want to be able to not use
> the index to store fields, but have them still fully functional as if
> indexed. That would be quite the magic trick.


Well there's a number of posts in different mail lists which talk about the
same requirements so I wonder is lucene/solr/smth else doesn't implement
something like this.

For example, see this post:
http://markmail.org/message/t4lv2hqtret4p62g?q=lucene+storing+fields+in+external+storage&page=1&refer=bmode2h2dwjpymba#query:lucene%20storing%20fields%20in%20external%20storage+page:1+mid:t4lv2hqtret4p62g+state:results



> You might check out http://skwish.sourceforge.net/. Its a cool little
> library that lets you store arbitrary data keyed by an auto generated id.


We already have the storage (Coherence), we just want to make it accessible
through standard Solr API and not to create an additional logic above Solr.
I mean the logic which processes result sets and add additional fields to it
by taking values from the external storage.  And in the case of that custom
post-search logic we also will have to implement some additional
filtering/ordering/etc of result sets based on values of that "external"
fields. So the question is is it possible to use Solr/Lucene features to use
external field storage for some of fields.

-- 
Andrew Klochkov

Re: external fields storage

[Mark Miller <ma...@gmail.com>]

Andrey Klochkov wrote:
> Hi Solr users
>
> Our index could be much smaller if we could store some of fields not in
> index directly but in some kind of external storage.
> All I've found until now is ExternalFileField class which shows that it's
> possible to implement such a storage, but I'm quite sure that the
> requirement is common and there should be some existing implementations.
> Also it would be good to be able to search using these fields, to include
> them in the search result sets and to update them with standard Solr update
> handlers.
>
>   
Thats a tall order. It almost sounds as if you want to be able to not 
use the index to store fields, but have them still fully functional as 
if indexed. That would be quite the magic trick.

You might check out http://skwish.sourceforge.net/. Its a cool little 
library that lets you store arbitrary data keyed by an auto generated id.

-- 
- Mark

http://www.lucidimagination.com