You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jamie Johnson <je...@gmail.com> on 2015/12/16 16:09:32 UTC
Append fields to a document
I have a use case where we only need to append some fields to a document.
To retrieve the full representation is very expensive but I can easily get
the deltas. Is it possible to just add fields to an existing Solr
document? I experimented with using overwrite=false, but that resulted in
two documents with the same uniqueKey in the index (which makes sense). Is
there a way to accomplish what I'm looking to do in Solr? My fields aren't
all stored and think it will be too expensive for me to make that change.
Any thoughts would be really appreciated.
Re: Append fields to a document
Posted by Jack Krupansky <ja...@gmail.com>.
What is the nature of your documents that reproducing them is so expensive?
Whatever it is, you should spend some time trying to reduce it to something
more manageable and performant. Generally, the primary recommendation is to
simply reindex any documents that need to be updated since atomic update
has various caveats so that it is only useful in a subset of use cases.
-- Jack Krupansky
On Wed, Dec 16, 2015 at 10:09 AM, Jamie Johnson <je...@gmail.com> wrote:
> I have a use case where we only need to append some fields to a document.
> To retrieve the full representation is very expensive but I can easily get
> the deltas. Is it possible to just add fields to an existing Solr
> document? I experimented with using overwrite=false, but that resulted in
> two documents with the same uniqueKey in the index (which makes sense). Is
> there a way to accomplish what I'm looking to do in Solr? My fields aren't
> all stored and think it will be too expensive for me to make that change.
> Any thoughts would be really appreciated.
>
Re: Append fields to a document
Posted by Erick Erickson <er...@gmail.com>.
The only way to do this currently is with Atomic Updates, which
require all fields to be stored except the destinations of copyField
directives. see:
https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
Best,
Erick
On Wed, Dec 16, 2015 at 7:09 AM, Jamie Johnson <je...@gmail.com> wrote:
> I have a use case where we only need to append some fields to a document.
> To retrieve the full representation is very expensive but I can easily get
> the deltas. Is it possible to just add fields to an existing Solr
> document? I experimented with using overwrite=false, but that resulted in
> two documents with the same uniqueKey in the index (which makes sense). Is
> there a way to accomplish what I'm looking to do in Solr? My fields aren't
> all stored and think it will be too expensive for me to make that change.
> Any thoughts would be really appreciated.
Re: Append fields to a document
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
If you enable LazyLoading and do not request them in your 'fl' list,
they should be mostly just size on disk AFAIK.
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 17 December 2015 at 08:09, Jamie Johnson <je...@gmail.com> wrote:
> The expense is in gathering the pieces to do the indexing. There isn't
> much that I can do in that regard unfortunately. I need to investigate
> storing the fields, if they aren't returned is the expense just size on
> disk or is there a memory cost as well?
> On Dec 16, 2015 7:43 PM, "Alexandre Rafalovitch" <ar...@gmail.com> wrote:
>
>> ExternalFileField might be useful in some situations.
>>
>> But also, is it possible that your Solr schema configuration is not
>> best suited for your domain? Is it - for example - possible that the
>> additional data should be in child records?
>>
>> Pure guesswork here, not enough information. But, as described, Solr
>> will not be able to fulfill your needs easily. Something will need to
>> change.
>>
>> Regards,
>> Alex.
>>
>> ----
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 16 December 2015 at 22:09, Jamie Johnson <je...@gmail.com> wrote:
>> > I have a use case where we only need to append some fields to a document.
>> > To retrieve the full representation is very expensive but I can easily
>> get
>> > the deltas. Is it possible to just add fields to an existing Solr
>> > document? I experimented with using overwrite=false, but that resulted
>> in
>> > two documents with the same uniqueKey in the index (which makes sense).
>> Is
>> > there a way to accomplish what I'm looking to do in Solr? My fields
>> aren't
>> > all stored and think it will be too expensive for me to make that change.
>> > Any thoughts would be really appreciated.
>>
Re: Append fields to a document
Posted by Jamie Johnson <je...@gmail.com>.
The expense is in gathering the pieces to do the indexing. There isn't
much that I can do in that regard unfortunately. I need to investigate
storing the fields, if they aren't returned is the expense just size on
disk or is there a memory cost as well?
On Dec 16, 2015 7:43 PM, "Alexandre Rafalovitch" <ar...@gmail.com> wrote:
> ExternalFileField might be useful in some situations.
>
> But also, is it possible that your Solr schema configuration is not
> best suited for your domain? Is it - for example - possible that the
> additional data should be in child records?
>
> Pure guesswork here, not enough information. But, as described, Solr
> will not be able to fulfill your needs easily. Something will need to
> change.
>
> Regards,
> Alex.
>
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 16 December 2015 at 22:09, Jamie Johnson <je...@gmail.com> wrote:
> > I have a use case where we only need to append some fields to a document.
> > To retrieve the full representation is very expensive but I can easily
> get
> > the deltas. Is it possible to just add fields to an existing Solr
> > document? I experimented with using overwrite=false, but that resulted
> in
> > two documents with the same uniqueKey in the index (which makes sense).
> Is
> > there a way to accomplish what I'm looking to do in Solr? My fields
> aren't
> > all stored and think it will be too expensive for me to make that change.
> > Any thoughts would be really appreciated.
>
Re: Append fields to a document
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
ExternalFileField might be useful in some situations.
But also, is it possible that your Solr schema configuration is not
best suited for your domain? Is it - for example - possible that the
additional data should be in child records?
Pure guesswork here, not enough information. But, as described, Solr
will not be able to fulfill your needs easily. Something will need to
change.
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 16 December 2015 at 22:09, Jamie Johnson <je...@gmail.com> wrote:
> I have a use case where we only need to append some fields to a document.
> To retrieve the full representation is very expensive but I can easily get
> the deltas. Is it possible to just add fields to an existing Solr
> document? I experimented with using overwrite=false, but that resulted in
> two documents with the same uniqueKey in the index (which makes sense). Is
> there a way to accomplish what I'm looking to do in Solr? My fields aren't
> all stored and think it will be too expensive for me to make that change.
> Any thoughts would be really appreciated.