You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Maria Muslea <ma...@gmail.com> on 2019/05/01 20:31:48 UTC
SolrPlugin update existing documents in newSearcher()
Hi,
I have a plugin that extends the AbstractSolrEventListener. I override the
newSearcher() method and the plan is to add some extra functionality,
namely updating existing documents by setting new values for existing
fields as well as adding new fields to the documents.
I can see that the plugin is invoked and I can get the list of documents,
but I cannot update existing fields or add new fields. I have tried various
approaches, but I cannot get it to work.
If you have any suggestions I would really appreciate it. The code that I
am currently trying is below.
Thank you,
Maria
for (DocIterator iter = docs.iterator(); iter.hasNext();) {
int doci = iter.nextDoc();
Document document = newSearcher.doc(doci);
SolrInputDocument solrInputDocument1 = new SolrInputDocument();
AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
addUpdateCommand1.clear();
solrInputDocument1.setField("id", document.get("id"));
solrInputDocument1.addField("newfield", "newvalue");
solrInputDocument1.setField("existingfield", "value");
addUpdateCommand1.solrDoc = solrInputDocument1;
getCore().getUpdateHandler().addDoc(addUpdateCommand1);
SolrQueryResponse re = new SolrQueryResponse();
SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
ModifiableSolrParams());
CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
getCore().getUpdateHandler().commit(commit);
}
Re: SolrPlugin update existing documents in newSearcher()
Posted by Maria Muslea <ma...@gmail.com>.
Thank you so much, this is very helpful.
Maria
On Thu, May 2, 2019 at 2:39 PM Jan Høydahl <ja...@cominvent.com> wrote:
> I think you cannot do that. The callback is sent AFTER a searcher is
> opened on the segment, so the index is already there.
> Normally you re-index from source if you need changes in schema or
> processing.
> If that is not possible, you must first check if ALL your fields are
> stored or docValues, if not you cannot hope to re-index from the content in
> the index in a lossless way. If you have everything stored, I'd create a
> new collection and write a script that reads all docs using cursorMark and
> indexes them into the new collection.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 2. mai 2019 kl. 19:39 skrev Maria Muslea <ma...@gmail.com>:
> >
> > Yes, I will want to also do that, but initially I need to modify the docs
> > that are already in SOLR, and I thought of doing that at startup.
> > I am able to get the documents that I would like to modify, but the
> > operations for modifying the documents don't seem to be doing anything.
> >
> > Do you see anything wrong with the way I am trying to modify the
> documents?
> >
> > Thank you for your help,
> > Maria
> >
> > On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
> >
> >> Hi
> >>
> >> I don't see your requirement clearly. Sounds like what you really need
> is
> >> an UpdateRequestProcessor where you CAN intercept docs being added and
> >> modify them as you wish.
> >> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >>> 1. mai 2019 kl. 22:31 skrev Maria Muslea <ma...@gmail.com>:
> >>>
> >>> Hi,
> >>>
> >>> I have a plugin that extends the AbstractSolrEventListener. I override
> >> the
> >>> newSearcher() method and the plan is to add some extra functionality,
> >>> namely updating existing documents by setting new values for existing
> >>> fields as well as adding new fields to the documents.
> >>>
> >>> I can see that the plugin is invoked and I can get the list of
> documents,
> >>> but I cannot update existing fields or add new fields. I have tried
> >> various
> >>> approaches, but I cannot get it to work.
> >>>
> >>> If you have any suggestions I would really appreciate it. The code
> that I
> >>> am currently trying is below.
> >>>
> >>> Thank you,
> >>> Maria
> >>>
> >>> for (DocIterator iter = docs.iterator(); iter.hasNext();) {
> >>>
> >>> int doci = iter.nextDoc();
> >>>
> >>> Document document = newSearcher.doc(doci);
> >>>
> >>>
> >>>
> >>> SolrInputDocument solrInputDocument1 = new SolrInputDocument();
> >>>
> >>> AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
> >>>
> >>> addUpdateCommand1.clear();
> >>>
> >>> solrInputDocument1.setField("id", document.get("id"));
> >>>
> >>> solrInputDocument1.addField("newfield", "newvalue");
> >>>
> >>> solrInputDocument1.setField("existingfield", "value");
> >>>
> >>> addUpdateCommand1.solrDoc = solrInputDocument1;
> >>>
> >>> getCore().getUpdateHandler().addDoc(addUpdateCommand1);
> >>>
> >>>
> >>> SolrQueryResponse re = new SolrQueryResponse();
> >>>
> >>> SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> >>> ModifiableSolrParams());
> >>>
> >>> CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
> >>>
> >>> getCore().getUpdateHandler().commit(commit);
> >>>
> >>>
> >>> }
> >>
> >>
>
>
Re: SolrPlugin update existing documents in newSearcher()
Posted by Jan Høydahl <ja...@cominvent.com>.
I think you cannot do that. The callback is sent AFTER a searcher is opened on the segment, so the index is already there.
Normally you re-index from source if you need changes in schema or processing.
If that is not possible, you must first check if ALL your fields are stored or docValues, if not you cannot hope to re-index from the content in the index in a lossless way. If you have everything stored, I'd create a new collection and write a script that reads all docs using cursorMark and indexes them into the new collection.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 2. mai 2019 kl. 19:39 skrev Maria Muslea <ma...@gmail.com>:
>
> Yes, I will want to also do that, but initially I need to modify the docs
> that are already in SOLR, and I thought of doing that at startup.
> I am able to get the documents that I would like to modify, but the
> operations for modifying the documents don't seem to be doing anything.
>
> Do you see anything wrong with the way I am trying to modify the documents?
>
> Thank you for your help,
> Maria
>
> On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi
>>
>> I don't see your requirement clearly. Sounds like what you really need is
>> an UpdateRequestProcessor where you CAN intercept docs being added and
>> modify them as you wish.
>> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 1. mai 2019 kl. 22:31 skrev Maria Muslea <ma...@gmail.com>:
>>>
>>> Hi,
>>>
>>> I have a plugin that extends the AbstractSolrEventListener. I override
>> the
>>> newSearcher() method and the plan is to add some extra functionality,
>>> namely updating existing documents by setting new values for existing
>>> fields as well as adding new fields to the documents.
>>>
>>> I can see that the plugin is invoked and I can get the list of documents,
>>> but I cannot update existing fields or add new fields. I have tried
>> various
>>> approaches, but I cannot get it to work.
>>>
>>> If you have any suggestions I would really appreciate it. The code that I
>>> am currently trying is below.
>>>
>>> Thank you,
>>> Maria
>>>
>>> for (DocIterator iter = docs.iterator(); iter.hasNext();) {
>>>
>>> int doci = iter.nextDoc();
>>>
>>> Document document = newSearcher.doc(doci);
>>>
>>>
>>>
>>> SolrInputDocument solrInputDocument1 = new SolrInputDocument();
>>>
>>> AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
>>>
>>> addUpdateCommand1.clear();
>>>
>>> solrInputDocument1.setField("id", document.get("id"));
>>>
>>> solrInputDocument1.addField("newfield", "newvalue");
>>>
>>> solrInputDocument1.setField("existingfield", "value");
>>>
>>> addUpdateCommand1.solrDoc = solrInputDocument1;
>>>
>>> getCore().getUpdateHandler().addDoc(addUpdateCommand1);
>>>
>>>
>>> SolrQueryResponse re = new SolrQueryResponse();
>>>
>>> SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
>>> ModifiableSolrParams());
>>>
>>> CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
>>>
>>> getCore().getUpdateHandler().commit(commit);
>>>
>>>
>>> }
>>
>>
Re: SolrPlugin update existing documents in newSearcher()
Posted by Maria Muslea <ma...@gmail.com>.
Yes, I will want to also do that, but initially I need to modify the docs
that are already in SOLR, and I thought of doing that at startup.
I am able to get the documents that I would like to modify, but the
operations for modifying the documents don't seem to be doing anything.
Do you see anything wrong with the way I am trying to modify the documents?
Thank you for your help,
Maria
On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi
>
> I don't see your requirement clearly. Sounds like what you really need is
> an UpdateRequestProcessor where you CAN intercept docs being added and
> modify them as you wish.
> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 1. mai 2019 kl. 22:31 skrev Maria Muslea <ma...@gmail.com>:
> >
> > Hi,
> >
> > I have a plugin that extends the AbstractSolrEventListener. I override
> the
> > newSearcher() method and the plan is to add some extra functionality,
> > namely updating existing documents by setting new values for existing
> > fields as well as adding new fields to the documents.
> >
> > I can see that the plugin is invoked and I can get the list of documents,
> > but I cannot update existing fields or add new fields. I have tried
> various
> > approaches, but I cannot get it to work.
> >
> > If you have any suggestions I would really appreciate it. The code that I
> > am currently trying is below.
> >
> > Thank you,
> > Maria
> >
> > for (DocIterator iter = docs.iterator(); iter.hasNext();) {
> >
> > int doci = iter.nextDoc();
> >
> > Document document = newSearcher.doc(doci);
> >
> >
> >
> > SolrInputDocument solrInputDocument1 = new SolrInputDocument();
> >
> > AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
> >
> > addUpdateCommand1.clear();
> >
> > solrInputDocument1.setField("id", document.get("id"));
> >
> > solrInputDocument1.addField("newfield", "newvalue");
> >
> > solrInputDocument1.setField("existingfield", "value");
> >
> > addUpdateCommand1.solrDoc = solrInputDocument1;
> >
> > getCore().getUpdateHandler().addDoc(addUpdateCommand1);
> >
> >
> > SolrQueryResponse re = new SolrQueryResponse();
> >
> > SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> > ModifiableSolrParams());
> >
> > CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
> >
> > getCore().getUpdateHandler().commit(commit);
> >
> >
> > }
>
>
Re: SolrPlugin update existing documents in newSearcher()
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi
I don't see your requirement clearly. Sounds like what you really need is an UpdateRequestProcessor where you CAN intercept docs being added and modify them as you wish. https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 1. mai 2019 kl. 22:31 skrev Maria Muslea <ma...@gmail.com>:
>
> Hi,
>
> I have a plugin that extends the AbstractSolrEventListener. I override the
> newSearcher() method and the plan is to add some extra functionality,
> namely updating existing documents by setting new values for existing
> fields as well as adding new fields to the documents.
>
> I can see that the plugin is invoked and I can get the list of documents,
> but I cannot update existing fields or add new fields. I have tried various
> approaches, but I cannot get it to work.
>
> If you have any suggestions I would really appreciate it. The code that I
> am currently trying is below.
>
> Thank you,
> Maria
>
> for (DocIterator iter = docs.iterator(); iter.hasNext();) {
>
> int doci = iter.nextDoc();
>
> Document document = newSearcher.doc(doci);
>
>
>
> SolrInputDocument solrInputDocument1 = new SolrInputDocument();
>
> AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
>
> addUpdateCommand1.clear();
>
> solrInputDocument1.setField("id", document.get("id"));
>
> solrInputDocument1.addField("newfield", "newvalue");
>
> solrInputDocument1.setField("existingfield", "value");
>
> addUpdateCommand1.solrDoc = solrInputDocument1;
>
> getCore().getUpdateHandler().addDoc(addUpdateCommand1);
>
>
> SolrQueryResponse re = new SolrQueryResponse();
>
> SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> ModifiableSolrParams());
>
> CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
>
> getCore().getUpdateHandler().commit(commit);
>
>
> }