You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by yaswanth kumar <ya...@gmail.com> on 2020/10/07 22:40:26 UTC
Question about solr commits
I have the below in my solrconfig.xml
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.Data.dir:}</str>
</updateLog>
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
</autoSoftCommit>
</updateHandler>
Does this mean even though we are always sending data with commit=false on
update solr api, the above should do the commit every minute (60000 ms)
right?
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: Question about solr commits
Posted by Erick Erickson <er...@gmail.com>.
Yes.
> On Oct 7, 2020, at 6:40 PM, yaswanth kumar <ya...@gmail.com> wrote:
>
> I have the below in my solrconfig.xml
>
> <updateHandler class="solr.DirectUpdateHandler2">
> <updateLog>
> <str name="dir">${solr.Data.dir:}</str>
> </updateLog>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
> </autoSoftCommit>
> </updateHandler>
>
> Does this mean even though we are always sending data with commit=false on
> update solr api, the above should do the commit every minute (60000 ms)
> right?
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: Question about solr commits
Posted by Erick Erickson <er...@gmail.com>.
This is a bit confused. There will be only one timer that starts at time T when
the first doc comes in. At T+ 15 seconds, all docs that have been received since
time T will be committed. The first doc to hit Solr _after_ T+15 seconds starts
a single new timer and the process repeats.
Best,
rick
> On Oct 8, 2020, at 2:26 PM, Rahul Goswami <ra...@gmail.com> wrote:
>
> Shawn,
> So if the autoCommit interval is 15 seconds, and one update request arrives
> at t=0 and another at t=10 seconds, then will there be two timers one
> expiring at t=15 and another at t=25 seconds, but this would amount to ONLY
> ONE commit at t=15 since that one would include changes from both updates.
> Is this understanding correct ?
>
> Thanks,
> Rahul
>
> On Wed, Oct 7, 2020 at 11:39 PM yaswanth kumar <ya...@gmail.com>
> wrote:
>
>> Thank you very much both Eric and Shawn
>>
>> Sent from my iPhone
>>
>>> On Oct 7, 2020, at 10:41 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>>>
>>> On 10/7/2020 4:40 PM, yaswanth kumar wrote:
>>>> I have the below in my solrconfig.xml
>>>> <updateHandler class="solr.DirectUpdateHandler2">
>>>> <updateLog>
>>>> <str name="dir">${solr.Data.dir:}</str>
>>>> </updateLog>
>>>> <autoCommit>
>>>> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
>>>> <openSearcher>false</openSearcher>
>>>> </autoCommit>
>>>> <autoSoftCommit>
>>>> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
>>>> </autoSoftCommit>
>>>> </updateHandler>
>>>> Does this mean even though we are always sending data with commit=false
>> on
>>>> update solr api, the above should do the commit every minute (60000 ms)
>>>> right?
>>>
>>> Assuming that you have not defined the "solr.autoCommit.maxTime" and/or
>> "solr.autoSoftCommit.maxTime" properties, this config has autoCommit set to
>> 60 seconds without opening a searcher, and autoSoftCommit set to 5 seconds.
>>>
>>> So five seconds after any indexing begins, Solr will do a soft commit.
>> When that commit finishes, changes to the index will be visible to
>> queries. One minute after any indexing begins, Solr will do a hard commit,
>> which guarantees that data is written to disk, but it will NOT open a new
>> searcher, which means that when the hard commit happens, any pending
>> changes to the index will not be visible.
>>>
>>> It's not "every five seconds" or "every 60 seconds" ... When any changes
>> are made, Solr starts a timer. When the timer expires, the commit is
>> fired. If no changes are made, no commits happen, because the timer isn't
>> started.
>>>
>>> Thanks,
>>> Shawn
>>
Re: Question about solr commits
Posted by Rahul Goswami <ra...@gmail.com>.
Shawn,
So if the autoCommit interval is 15 seconds, and one update request arrives
at t=0 and another at t=10 seconds, then will there be two timers one
expiring at t=15 and another at t=25 seconds, but this would amount to ONLY
ONE commit at t=15 since that one would include changes from both updates.
Is this understanding correct ?
Thanks,
Rahul
On Wed, Oct 7, 2020 at 11:39 PM yaswanth kumar <ya...@gmail.com>
wrote:
> Thank you very much both Eric and Shawn
>
> Sent from my iPhone
>
> > On Oct 7, 2020, at 10:41 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> >
> > On 10/7/2020 4:40 PM, yaswanth kumar wrote:
> >> I have the below in my solrconfig.xml
> >> <updateHandler class="solr.DirectUpdateHandler2">
> >> <updateLog>
> >> <str name="dir">${solr.Data.dir:}</str>
> >> </updateLog>
> >> <autoCommit>
> >> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> >> <openSearcher>false</openSearcher>
> >> </autoCommit>
> >> <autoSoftCommit>
> >> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
> >> </autoSoftCommit>
> >> </updateHandler>
> >> Does this mean even though we are always sending data with commit=false
> on
> >> update solr api, the above should do the commit every minute (60000 ms)
> >> right?
> >
> > Assuming that you have not defined the "solr.autoCommit.maxTime" and/or
> "solr.autoSoftCommit.maxTime" properties, this config has autoCommit set to
> 60 seconds without opening a searcher, and autoSoftCommit set to 5 seconds.
> >
> > So five seconds after any indexing begins, Solr will do a soft commit.
> When that commit finishes, changes to the index will be visible to
> queries. One minute after any indexing begins, Solr will do a hard commit,
> which guarantees that data is written to disk, but it will NOT open a new
> searcher, which means that when the hard commit happens, any pending
> changes to the index will not be visible.
> >
> > It's not "every five seconds" or "every 60 seconds" ... When any changes
> are made, Solr starts a timer. When the timer expires, the commit is
> fired. If no changes are made, no commits happen, because the timer isn't
> started.
> >
> > Thanks,
> > Shawn
>
Re: Question about solr commits
Posted by yaswanth kumar <ya...@gmail.com>.
Thank you very much both Eric and Shawn
Sent from my iPhone
> On Oct 7, 2020, at 10:41 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 10/7/2020 4:40 PM, yaswanth kumar wrote:
>> I have the below in my solrconfig.xml
>> <updateHandler class="solr.DirectUpdateHandler2">
>> <updateLog>
>> <str name="dir">${solr.Data.dir:}</str>
>> </updateLog>
>> <autoCommit>
>> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
>> <openSearcher>false</openSearcher>
>> </autoCommit>
>> <autoSoftCommit>
>> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
>> </autoSoftCommit>
>> </updateHandler>
>> Does this mean even though we are always sending data with commit=false on
>> update solr api, the above should do the commit every minute (60000 ms)
>> right?
>
> Assuming that you have not defined the "solr.autoCommit.maxTime" and/or "solr.autoSoftCommit.maxTime" properties, this config has autoCommit set to 60 seconds without opening a searcher, and autoSoftCommit set to 5 seconds.
>
> So five seconds after any indexing begins, Solr will do a soft commit. When that commit finishes, changes to the index will be visible to queries. One minute after any indexing begins, Solr will do a hard commit, which guarantees that data is written to disk, but it will NOT open a new searcher, which means that when the hard commit happens, any pending changes to the index will not be visible.
>
> It's not "every five seconds" or "every 60 seconds" ... When any changes are made, Solr starts a timer. When the timer expires, the commit is fired. If no changes are made, no commits happen, because the timer isn't started.
>
> Thanks,
> Shawn
Re: Question about solr commits
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/7/2020 4:40 PM, yaswanth kumar wrote:
> I have the below in my solrconfig.xml
>
> <updateHandler class="solr.DirectUpdateHandler2">
> <updateLog>
> <str name="dir">${solr.Data.dir:}</str>
> </updateLog>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>
> </autoSoftCommit>
> </updateHandler>
>
> Does this mean even though we are always sending data with commit=false on
> update solr api, the above should do the commit every minute (60000 ms)
> right?
Assuming that you have not defined the "solr.autoCommit.maxTime" and/or
"solr.autoSoftCommit.maxTime" properties, this config has autoCommit set
to 60 seconds without opening a searcher, and autoSoftCommit set to 5
seconds.
So five seconds after any indexing begins, Solr will do a soft commit.
When that commit finishes, changes to the index will be visible to
queries. One minute after any indexing begins, Solr will do a hard
commit, which guarantees that data is written to disk, but it will NOT
open a new searcher, which means that when the hard commit happens, any
pending changes to the index will not be visible.
It's not "every five seconds" or "every 60 seconds" ... When any changes
are made, Solr starts a timer. When the timer expires, the commit is
fired. If no changes are made, no commits happen, because the timer
isn't started.
Thanks,
Shawn