You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jack Park <ja...@topicquests.org> on 2013/08/07 19:04:48 UTC
Question about soft commit and updateRequestProcessorChain
If one allows for a soft commit (rather than a hard commit on each
request), when does the updateRequestProcessorChain fire? Does it fire
after the commit?
Many thanks
Jack
Re: Question about soft commit and updateRequestProcessorChain
Posted by Erick Erickson <er...@gmail.com>.
How are you allowing for a soft commit? IOW how are you triggering it?
And what do you speculate the updateRequestProcessorChain has to do with
soft commit?
Best
Erick
On Wed, Aug 7, 2013 at 1:04 PM, Jack Park <ja...@topicquests.org> wrote:
> If one allows for a soft commit (rather than a hard commit on each
> request), when does the updateRequestProcessorChain fire? Does it fire
> after the commit?
>
> Many thanks
> Jack
>
Re: Question about soft commit and updateRequestProcessorChain
Posted by Erick Erickson <er...@gmail.com>.
Well, in the Solr4 world, there's another option. Do
a hard commit with openSearcher=false. That will
guarantee that the documents are durably written
because it'll close the segment. What it will NOT do
is make the documents searchable, you need to do
a soft commit to make that happen.
The other thing to consider is "real time get" which
allows the latest copy of the document to be fetched
from the index regardless of the commit status....
Best
Erick
On Wed, Aug 7, 2013 at 11:24 PM, Alexandre Rafalovitch
<ar...@gmail.com>wrote:
> I noticed the example solrconfig.xml has event listeners for commit. I
> wonder if they could be useful here:
> <listener event="postCommit" class="solr.RunExecutableListener">
>
> I am not sure how they work with hard/soft commits though.
>
> Regards,
> Alex.
> P.s. Just to make things complicated, UpdateRequestProcessors
> have processCommit() method. But these seem to be a commit 'request', not
> commit 'execution'
>
> Personal website: http://www.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all at
> once. Lately, it doesn't seem to be working. (Anonymous - via GTD book)
>
>
> On Wed, Aug 7, 2013 at 7:51 PM, Jack Krupansky <jack@basetechnology.com
> >wrote:
>
> > No and No...
> >
> > Commit has a life of its own. Autocommit can occur based on time and
> > number of documents, independent of the update processor chain. For
> > example, you can send a few updates with "commit within" and sit there
> idle
> > doing no commands and then suddenly after the commitWithin interval the
> > commit magically happens. CommitWithin is a recommended approach - just
> > pick the desired time interval.
> >
> > Unless you have an explicit commit in your update command, there is no
> > guarantee of Run Update doing a commit.
> >
> > No, the document is not committed "after the first step in the update
> > processor chain" - the Run Update is usually the last or next to last
> (like
> > if you use the Log Update processor) processor in the chain. IFF you
> > requested commit, soft or hard, on your update command, the commit will
> > occur on the Run Update processor step of the chain.
> >
> >
> >
> >
> > -- Jack Krupansky
> > -----Original Message----- From: Jack Park
> > Sent: Wednesday, August 07, 2013 7:41 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Question about soft commit and updateRequestProcessorChain
> >
> >
> > Ok. So, running the update processor chain *is* the commit process?
> >
> > In answer to Erick's question: my habit, an old and apparently bad
> > one, has been to call a hard commit at the end of each update. My
> > question had to do with allowing soft commits to be controlled by
> > settings in solrconfig.xml, say every 30 seconds or something like
> > that (I really haven't studied such options yet).
> >
> > I ask this question because I add an additional call to the update
> > processor, which, after running Lucene, the document is then sent
> > outside to an agent network for further processing. I needed to know
> > if the document was already committed by that time.
> >
> > I am inferring from here that the document has been committed after
> > the first step in the update processor chain, even if that's based on
> > a soft commit.
> >
> > Thanks!
> > JackP
> >
> > On Wed, Aug 7, 2013 at 4:20 PM, Jack Krupansky <ja...@basetechnology.com>
> > wrote:
> >
> >> Most update processor chains will be configured with the Run Update
> >> processor as the last processor of the chain. That's were the Lucene
> index
> >> update and optional commit would be done.
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Jack Park
> >> Sent: Wednesday, August 07, 2013 1:04 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Question about soft commit and updateRequestProcessorChain
> >>
> >>
> >> If one allows for a soft commit (rather than a hard commit on each
> >> request), when does the updateRequestProcessorChain fire? Does it fire
> >> after the commit?
> >>
> >> Many thanks
> >> Jack
> >>
> >
> >
>
Re: Question about soft commit and updateRequestProcessorChain
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I noticed the example solrconfig.xml has event listeners for commit. I
wonder if they could be useful here:
<listener event="postCommit" class="solr.RunExecutableListener">
I am not sure how they work with hard/soft commits though.
Regards,
Alex.
P.s. Just to make things complicated, UpdateRequestProcessors
have processCommit() method. But these seem to be a commit 'request', not
commit 'execution'
Personal website: http://www.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all at
once. Lately, it doesn't seem to be working. (Anonymous - via GTD book)
On Wed, Aug 7, 2013 at 7:51 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> No and No...
>
> Commit has a life of its own. Autocommit can occur based on time and
> number of documents, independent of the update processor chain. For
> example, you can send a few updates with "commit within" and sit there idle
> doing no commands and then suddenly after the commitWithin interval the
> commit magically happens. CommitWithin is a recommended approach - just
> pick the desired time interval.
>
> Unless you have an explicit commit in your update command, there is no
> guarantee of Run Update doing a commit.
>
> No, the document is not committed "after the first step in the update
> processor chain" - the Run Update is usually the last or next to last (like
> if you use the Log Update processor) processor in the chain. IFF you
> requested commit, soft or hard, on your update command, the commit will
> occur on the Run Update processor step of the chain.
>
>
>
>
> -- Jack Krupansky
> -----Original Message----- From: Jack Park
> Sent: Wednesday, August 07, 2013 7:41 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Question about soft commit and updateRequestProcessorChain
>
>
> Ok. So, running the update processor chain *is* the commit process?
>
> In answer to Erick's question: my habit, an old and apparently bad
> one, has been to call a hard commit at the end of each update. My
> question had to do with allowing soft commits to be controlled by
> settings in solrconfig.xml, say every 30 seconds or something like
> that (I really haven't studied such options yet).
>
> I ask this question because I add an additional call to the update
> processor, which, after running Lucene, the document is then sent
> outside to an agent network for further processing. I needed to know
> if the document was already committed by that time.
>
> I am inferring from here that the document has been committed after
> the first step in the update processor chain, even if that's based on
> a soft commit.
>
> Thanks!
> JackP
>
> On Wed, Aug 7, 2013 at 4:20 PM, Jack Krupansky <ja...@basetechnology.com>
> wrote:
>
>> Most update processor chains will be configured with the Run Update
>> processor as the last processor of the chain. That's were the Lucene index
>> update and optional commit would be done.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Jack Park
>> Sent: Wednesday, August 07, 2013 1:04 PM
>> To: solr-user@lucene.apache.org
>> Subject: Question about soft commit and updateRequestProcessorChain
>>
>>
>> If one allows for a soft commit (rather than a hard commit on each
>> request), when does the updateRequestProcessorChain fire? Does it fire
>> after the commit?
>>
>> Many thanks
>> Jack
>>
>
>
Re: Question about soft commit and updateRequestProcessorChain
Posted by Jack Krupansky <ja...@basetechnology.com>.
No and No...
Commit has a life of its own. Autocommit can occur based on time and number
of documents, independent of the update processor chain. For example, you
can send a few updates with "commit within" and sit there idle doing no
commands and then suddenly after the commitWithin interval the commit
magically happens. CommitWithin is a recommended approach - just pick the
desired time interval.
Unless you have an explicit commit in your update command, there is no
guarantee of Run Update doing a commit.
No, the document is not committed "after the first step in the update
processor chain" - the Run Update is usually the last or next to last (like
if you use the Log Update processor) processor in the chain. IFF you
requested commit, soft or hard, on your update command, the commit will
occur on the Run Update processor step of the chain.
-- Jack Krupansky
-----Original Message-----
From: Jack Park
Sent: Wednesday, August 07, 2013 7:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Question about soft commit and updateRequestProcessorChain
Ok. So, running the update processor chain *is* the commit process?
In answer to Erick's question: my habit, an old and apparently bad
one, has been to call a hard commit at the end of each update. My
question had to do with allowing soft commits to be controlled by
settings in solrconfig.xml, say every 30 seconds or something like
that (I really haven't studied such options yet).
I ask this question because I add an additional call to the update
processor, which, after running Lucene, the document is then sent
outside to an agent network for further processing. I needed to know
if the document was already committed by that time.
I am inferring from here that the document has been committed after
the first step in the update processor chain, even if that's based on
a soft commit.
Thanks!
JackP
On Wed, Aug 7, 2013 at 4:20 PM, Jack Krupansky <ja...@basetechnology.com>
wrote:
> Most update processor chains will be configured with the Run Update
> processor as the last processor of the chain. That's were the Lucene index
> update and optional commit would be done.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Jack Park
> Sent: Wednesday, August 07, 2013 1:04 PM
> To: solr-user@lucene.apache.org
> Subject: Question about soft commit and updateRequestProcessorChain
>
>
> If one allows for a soft commit (rather than a hard commit on each
> request), when does the updateRequestProcessorChain fire? Does it fire
> after the commit?
>
> Many thanks
> Jack
Re: Question about soft commit and updateRequestProcessorChain
Posted by Jack Park <ja...@topicquests.org>.
Ok. So, running the update processor chain *is* the commit process?
In answer to Erick's question: my habit, an old and apparently bad
one, has been to call a hard commit at the end of each update. My
question had to do with allowing soft commits to be controlled by
settings in solrconfig.xml, say every 30 seconds or something like
that (I really haven't studied such options yet).
I ask this question because I add an additional call to the update
processor, which, after running Lucene, the document is then sent
outside to an agent network for further processing. I needed to know
if the document was already committed by that time.
I am inferring from here that the document has been committed after
the first step in the update processor chain, even if that's based on
a soft commit.
Thanks!
JackP
On Wed, Aug 7, 2013 at 4:20 PM, Jack Krupansky <ja...@basetechnology.com> wrote:
> Most update processor chains will be configured with the Run Update
> processor as the last processor of the chain. That's were the Lucene index
> update and optional commit would be done.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Jack Park
> Sent: Wednesday, August 07, 2013 1:04 PM
> To: solr-user@lucene.apache.org
> Subject: Question about soft commit and updateRequestProcessorChain
>
>
> If one allows for a soft commit (rather than a hard commit on each
> request), when does the updateRequestProcessorChain fire? Does it fire
> after the commit?
>
> Many thanks
> Jack
Re: Question about soft commit and updateRequestProcessorChain
Posted by Jack Krupansky <ja...@basetechnology.com>.
Most update processor chains will be configured with the Run Update
processor as the last processor of the chain. That's were the Lucene index
update and optional commit would be done.
-- Jack Krupansky
-----Original Message-----
From: Jack Park
Sent: Wednesday, August 07, 2013 1:04 PM
To: solr-user@lucene.apache.org
Subject: Question about soft commit and updateRequestProcessorChain
If one allows for a soft commit (rather than a hard commit on each
request), when does the updateRequestProcessorChain fire? Does it fire
after the commit?
Many thanks
Jack