Partial document update Solr

[erik rombouts <e....@gmail.com>]

Hi all,

I found the following message on the dev mailinglist -
http://www.mail-archive.com/dev@nutch.apache.org/msg10843.html - with
a "patch" to allow Solr partial document updates via SolrWriter. There was
a request from Julien to add this to JIRA but i cannot seem to find it.
Does anybody know if this ever end up in JIRA, via the original poster or
otherwise?

And as a second question, since i am investigating the possibilities, is
the Solr partial update functionality something to be used by/added to
Nutch?

Thanks in advance for any guidance :)

With best regards,
Erik Rombouts

Re: Partial document update Solr

[erik rombouts <e....@gmail.com>]

Hi Markus,

Thanks for the quick and thorough answer. I did read that all the fields
need to be stored for that yes, so i agree that in many cases this is not
really the preferred solution. I am thinking about setting up a separate
core in Solr with only a handful of fields and even less fields which would
need the update/add to multivalue functionality. But you have given me food
for thought ... i will sleep on it a little and see in the morning if there
are possibly other ways to get to what i have in mind or to invest some
time to implement and test.

Thanks again!
Erik



On Mon, Nov 11, 2013 at 11:40 PM, Markus Jelsma
<ma...@openindex.io>wrote:

> Hi Erik,
>
> That's pretty straightforward to implement. I did it for our site search
> platform to ease on transport costs, Lucene analysis and I/O in general.
> But it was quite useless when i found out i didn't read the docs that well
> and had to mark every field as stored. Solr needs every field to be stored
> so it can reinsert the document with only the updated fields.
>
> This means we would have to store a lot of data in Lucene that we don't
> have to now. Saving on transport and other things, in our case, didn't
> justify storing every field.
>
> You may want to reconsider doing partial updates. But if you still need
> them and are capable with Java, SolrJ and Nutch' SolrWriter, you can make
> it happen in a few hours.
>
> Cheers
>
> -----Original message-----
> From: erik rombouts<e....@gmail.com>
> Sent: Monday 11th November 2013 23:16
> To: dev@nutch.apache.org
> Subject: Partial document update Solr
>
> Hi all,
>
> I found the following message on the dev mailinglist -
> http://www.mail-archive.com/dev@nutch.apache.org/msg10843.html <
> http://www.mail-archive.com/dev@nutch.apache.org/msg10843.html> - with
> a "patch" to allow Solr partial document updates via SolrWriter. There was
> a request from Julien to add this to JIRA but i cannot seem to find it.
> Does anybody know if this ever end up in JIRA, via the original poster or
> otherwise?
>
> And as a second question, since i am investigating the possibilities, is
> the Solr partial update functionality something to be used by/added to
> Nutch?
>
> Thanks in advance for any guidance :)
>
> With best regards,
>
> Erik Rombouts
>
>
>

RE: Partial document update Solr

[Markus Jelsma <ma...@openindex.io>]

Hi Erik,

That's pretty straightforward to implement. I did it for our site search platform to ease on transport costs, Lucene analysis and I/O in general. But it was quite useless when i found out i didn't read the docs that well and had to mark every field as stored. Solr needs every field to be stored so it can reinsert the document with only the updated fields.

This means we would have to store a lot of data in Lucene that we don't have to now. Saving on transport and other things, in our case, didn't justify storing every field.

You may want to reconsider doing partial updates. But if you still need them and are capable with Java, SolrJ and Nutch' SolrWriter, you can make it happen in a few hours.

Cheers

-----Original message-----
From: erik rombouts<e....@gmail.com>
Sent: Monday 11th November 2013 23:16
To: dev@nutch.apache.org
Subject: Partial document update Solr

Hi all,

I found the following message on the dev mailinglist - http://www.mail-archive.com/dev@nutch.apache.org/msg10843.html <http://www.mail-archive.com/dev@nutch.apache.org/msg10843.html> - with a "patch" to allow Solr partial document updates via SolrWriter. There was a request from Julien to add this to JIRA but i cannot seem to find it. Does anybody know if this ever end up in JIRA, via the original poster or otherwise?

And as a second question, since i am investigating the possibilities, is the Solr partial update functionality something to be used by/added to Nutch?

Thanks in advance for any guidance :)

With best regards,

Erik Rombouts