You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dewi Wahyuni <de...@qut.edu.au> on 2012/07/09 04:15:21 UTC
PostCommit Document
Hi All,
I would like to know how to use postCommit in SOLR properly. I would like to grab the indexed document and do further processing with it. How do I capture the documents being committed to the SOLR through the arguments in the postCommit config? I'm not using SolrJ and have no intention in using Java at the moment.
If this is not possible, please let me know.
Thank you
Dewi
RE: PostCommit Document
Posted by Dewi Wahyuni <de...@qut.edu.au>.
Hi.
: >> this is an option I am exploring if the postcommit doesn't work. It's
: just in my head i thought there must be some way to get the document
: through postCommit and that I'm just missing it because I can't find the
: document that says so. Someone told me it is possible if I use SolrJ, so
: I thought it most likely would be possible with other methods.
"postCommit" hooks are not specific to a document, they are specific to a
commit. there could be lots of documents added/deleted in between
successive commits.
>> Yes I can handle multiple documents in the new app
perhaps what you are thinking of is the "UpdateProcessor" API, which let's
you inject code ino the processing change of all updates? ... i would
have suggested that in my earlier message as a way to do your
"system#2" processing, but: a) it requires you to write java; b) it would
happen *before* the document is added (which would be mor efficient then
what you are describing, but you explicitly said you didn't want system#2
to affect the document adds generated by system#1)
Yes, your statements are all correct.
Looks like I should go with the scheduled process instead.
RE: PostCommit Document
Posted by Chris Hostetter <ho...@fucit.org>.
: >> this is an option I am exploring if the postcommit doesn't work. It's
: just in my head i thought there must be some way to get the document
: through postCommit and that I'm just missing it because I can't find the
: document that says so. Someone told me it is possible if I use SolrJ, so
: I thought it most likely would be possible with other methods.
"postCommit" hooks are not specific to a document, they are specific to a
commit. there could be lots of documents added/deleted in between
successive commits.
perhaps what you are thinking of is the "UpdateProcessor" API, which let's
you inject code ino the processing change of all updates? ... i would
have suggested that in my earlier message as a way to do your
"system#2" processing, but: a) it requires you to write java; b) it would
happen *before* the document is added (which would be mor efficient then
what you are describing, but you explicitly said you didn't want system#2
to affect the document adds generated by system#1)
-Hoss
RE: PostCommit Document
Posted by Dewi Wahyuni <de...@qut.edu.au>.
Hi, thanks for the reply. My reply as below
: Basically we have an application that does indexing of documents to
: SOLR. This application is basically a third party and we didn't do much
: meddling with it. There is another application that I'm developing to
: use some of the fields data indexed (both when it is new or updated), do
: some calculation, and adds a field with the result of that process. Then
: I would like to index that document back to Solr (do an update).
:
: I know it will mean twice reindexing of the whole document and that
: sounds really inefficient, but we don't want the success of the first
: application to depend on the extra processing.
can you explain what you mean by "we don't want the success of the first
application to depend on the extra processing." ?
Im guessing you are saying that you want the documents addeded by system#1
to be in the index and searchable right away, before system#2 runs, and
even if the calclutions (and reindexing) done by system#2 fail?
>> Yes this is correct.
: I am using PHP for the application and would prefer not to use Java just
: for listener. Is there no way to do this?
Why not just include a "timestamp" field in your schema.xml, that
defaults to "NOW" (so system#1 doesn't even have to know about it) and
have system#2 periodically query for docs with a timestamp value greater
then the last time it ran? then you don't have to worry about any special
post-doc-added code, or deal with java at all.
>> this is an option I am exploring if the postcommit doesn't work. It's just in my head i thought there must be some way to get the document through postCommit and that I'm just missing it because I can't find the document that says so. Someone told me it is possible if I use SolrJ, so I thought it most likely would be possible with other methods.
If you *really* care that you get notified immediately after documents
were committed (and dont' want to just "poll", you can still configure a
postCommit listener that uses the RunExecutableListern to fire off
whatever shell process you want to tell your (external) code to run, and
then your external application can use the same polling logic to get the
list of newly added docs since the last time it ran.
>> Yes, looks like that is the option right now and with the batch previous one you've mentioned.
Thanks
RE: PostCommit Document
Posted by Chris Hostetter <ho...@fucit.org>.
: Basically we have an application that does indexing of documents to
: SOLR. This application is basically a third party and we didn't do much
: meddling with it. There is another application that I'm developing to
: use some of the fields data indexed (both when it is new or updated), do
: some calculation, and adds a field with the result of that process. Then
: I would like to index that document back to Solr (do an update).
:
: I know it will mean twice reindexing of the whole document and that
: sounds really inefficient, but we don't want the success of the first
: application to depend on the extra processing.
can you explain what you mean by "we don't want the success of the first
application to depend on the extra processing." ?
Im guessing you are saying that you want the documents addeded by system#1
to be in the index and searchable right away, before system#2 runs, and
even if the calclutions (and reindexing) done by system#2 fail?
: I am using PHP for the application and would prefer not to use Java just
: for listener. Is there no way to do this?
Why not just include a "timestamp" field in your schema.xml, that
defaults to "NOW" (so system#1 doesn't even have to know about it) and
have system#2 periodically query for docs with a timestamp value greater
then the last time it ran? then you don't have to worry about any special
post-doc-added code, or deal with java at all.
If you *really* care that you get notified immediately after documents
were committed (and dont' want to just "poll", you can still configure a
postCommit listener that uses the RunExecutableListern to fire off
whatever shell process you want to tell your (external) code to run, and
then your external application can use the same polling logic to get the
list of newly added docs since the last time it ran.
-Hoss
RE: PostCommit Document
Posted by Dewi Wahyuni <de...@qut.edu.au>.
Hi Ahmet
Basically we have an application that does indexing of documents to SOLR. This application is basically a third party and we didn't do much meddling with it. There is another application that I'm developing to use some of the fields data indexed (both when it is new or updated), do some calculation, and adds a field with the result of that process. Then I would like to index that document back to Solr (do an update).
I know it will mean twice reindexing of the whole document and that sounds really inefficient, but we don't want the success of the first application to depend on the extra processing.
I am using PHP for the application and would prefer not to use Java just for listener. Is there no way to do this?
Dewi
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: Wednesday, 11 July 2012 9:27 PM
To: solr-user@lucene.apache.org
Subject: RE: PostCommit Document
> Thanks for the reply. Yes I wanted the executable to run
> after the commit operation. I would like to have the doc XML
> though. The further processing might be intensive and so I
> didn't want the document to wait for the extra processing
> before committing (if I use UpdateRequestProcessor). The
> other reason is because we are not developing in Java.
Sorry but I don't fully follow your requirement. Can you elaborate more? What is this further processing, does it add new fields?
> I have managed to run an executable but I have no idea how
> to get the document. Would it be risky to use the timestamp
> instead?
> i.e. The executable check the current timestamp and queyr
> solr for documents with that exact time stamp. (Or with a
> few millisecond tolerance?)
DIH use this technique to capture delta changes. So you want to capture XML of newly committed documents? Documents changed after previous commit?
RE: PostCommit Document
Posted by Ahmet Arslan <io...@yahoo.com>.
> Thanks for the reply. Yes I wanted the executable to run
> after the commit operation. I would like to have the doc XML
> though. The further processing might be intensive and so I
> didn't want the document to wait for the extra processing
> before committing (if I use UpdateRequestProcessor). The
> other reason is because we are not developing in Java.
Sorry but I don't fully follow your requirement. Can you elaborate more? What is this further processing, does it add new fields?
> I have managed to run an executable but I have no idea how
> to get the document. Would it be risky to use the timestamp
> instead?
> i.e. The executable check the current timestamp and queyr
> solr for documents with that exact time stamp. (Or with a
> few millisecond tolerance?)
DIH use this technique to capture delta changes. So you want to capture XML of newly committed documents? Documents changed after previous commit?
RE: PostCommit Document
Posted by Dewi Wahyuni <de...@qut.edu.au>.
Hi
Thanks for the reply. Yes I wanted the executable to run after the commit operation. I would like to have the doc XML though. The further processing might be intensive and so I didn't want the document to wait for the extra processing before committing (if I use UpdateRequestProcessor). The other reason is because we are not developing in Java.
I have managed to run an executable but I have no idea how to get the document. Would it be risky to use the timestamp instead?
i.e. The executable check the current timestamp and queyr solr for documents with that exact time stamp. (Or with a few millisecond tolerance?)
Thanks
Dewi
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: Tuesday, 10 July 2012 2:30 AM
To: solr-user@lucene.apache.org
Subject: Re: PostCommit Document
> I would like to know how to use postCommit in SOLR properly.
> I would like to grab the indexed document and do further
> processing with it. How do I capture the documents
> being committed to the SOLR through the arguments in the
> postCommit config? I'm not using SolrJ and have no
> intention in using Java at the moment.
> If this is not possible, please let me know.
Not sure but, I think <listener event="postCommit" is used to run some executables after the commit operation. It does not let you to modify committed documents.
If you want further processing, http://wiki.apache.org/solr/UpdateRequestProcessor is used to modify solr documents being indexed.
Re: PostCommit Document
Posted by Ahmet Arslan <io...@yahoo.com>.
> I would like to know how to use postCommit in SOLR properly.
> I would like to grab the indexed document and do further
> processing with it. How do I capture the documents
> being committed to the SOLR through the arguments in the
> postCommit config? I'm not using SolrJ and have no
> intention in using Java at the moment.
> If this is not possible, please let me know.
Not sure but, I think <listener event="postCommit" is used to run some executables after the commit operation. It does not let you to modify committed documents.
If you want further processing, http://wiki.apache.org/solr/UpdateRequestProcessor is used to modify solr documents being indexed.