You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Ritvik Sharma <ri...@gmail.com> on 2021/03/19 11:06:08 UTC
tlog size issue- solr cloud 6.6
Hi Guys
We have faced an issue where tlog size is increasing unnecessarily. We are
using a "heavy indexing, heavy query" approach. We enabled hard commit
also,
solr cloud: 6.6
zk: 3.4.10
shards: 2, replication factor= 2
solrconfig,
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<maxDocs>10000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<!-- softAutoCommit is like autoCommit except it causes a
'soft' commit which only ensures that changes are visible
but does not ensure that data is synced to disk. This is
faster and more near-realtime friendly than a hard commit.
-->
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
</autoSoftCommit>
Re: tlog size issue- solr cloud 6.6
Posted by Ritvik Sharma <ri...@gmail.com>.
Yes, I did. With the use of CloudClollection Utility.
On Tue, 23 Mar 2021 at 12:01 AM, Dominique Bejean <do...@eolya.fr>
wrote:
> Hi,
>
> Did you try to force a hard commit in order to see the inpact on tlog ?
> http://localhost:8983/solr/[collection_name]/update?commit=true
>
> Did you read this article ?
>
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Regards
>
> Dominique
>
>
> Le lun. 22 mars 2021 à 17:29, Ritvik Sharma <ri...@gmail.com> a
> écrit :
>
>> Hi Dominique
>>
>> Any suggestions?
>>
>> On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <ri...@gmail.com>
>> wrote:
>>
>>> HI Dominique
>>>
>>> softcommit=false is coming in logs,
>>>
>>>
>>> INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1
>>> x:solrcollection_shard2_replica1]
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
>>> [solrcollection_shard2_replica1] webapp=/solr path=/update
>>> params={update.distrib=FROMLEADER&waitSearcher=true&
>>> *openSearcher=true&commit=true&softCommit=false*&distrib.from=
>>> http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
>>> 0 676
>>> INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1
>>> x:solrcollection_shard2_replica1]
>>> org.apache.solr.update.DirectUpdateHandler2; start
>>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>>>
>>>
>>> On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <
>>> dominique.bejean@eolya.fr> wrote:
>>>
>>>> Hi,
>>>>
>>>> If your tlogs grow, It looks like hard commit are never executed.
>>>>
>>>> Are you seeing hard commits in logs (grep 'softCommit=false') ?
>>>>
>>>> What is it happening if you force a hard commit ? Are tlog truncated ?
>>>> http://localhost:8983/solr/[collection_name]/update?commit=true
>>>>
>>>> Did you read this article ?
>>>>
>>>> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>
>>>> Regards
>>>>
>>>> Dominique
>>>>
>>>>
>>>> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ri...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Hi Dominique
>>>>>
>>>>> Heavy query means high query rate on solr.
>>>>>
>>>>> Honestly for 2-3 days we have stopped the queries on solr only we are
>>>>> doing indexing !!!
>>>>> As you see we have enabled Hard commit also to decrease tlog log size
>>>>> or more and the same is written on solr docs. Still the same behaviour is
>>>>> occurring on replicas. In some replicas, tlog size is of 300 GB.
>>>>>
>>>>> We may increase softcommit timeout but as per my understanding it only
>>>>> helps in searching availability prior to commit to indexes.
>>>>>
>>>>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
>>>>> dominique.bejean@eolya.fr> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> By heavy query, do you mean you have a high query rate and/or you need
>>>>>> index update be available fast after few seconds (NRT search) ?
>>>>>>
>>>>>> Do you see the hard commits in logs ?
>>>>>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Dominique
>>>>>>
>>>>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>> > Hi Guys
>>>>>> >
>>>>>> > We have faced an issue where tlog size is increasing unnecessarily.
>>>>>> We are
>>>>>> > using a "heavy indexing, heavy query" approach. We enabled hard
>>>>>> commit
>>>>>> > also,
>>>>>> >
>>>>>> > solr cloud: 6.6
>>>>>> > zk: 3.4.10
>>>>>> > shards: 2, replication factor= 2
>>>>>> >
>>>>>> >
>>>>>> > solrconfig,
>>>>>> >
>>>>>> > <autoCommit>
>>>>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>>>> > <maxDocs>10000</maxDocs>
>>>>>> > <openSearcher>false</openSearcher>
>>>>>> >
>>>>>> >
>>>>>> > </autoCommit>
>>>>>> >
>>>>>> > <!-- softAutoCommit is like autoCommit except it causes a
>>>>>> > 'soft' commit which only ensures that changes are visible
>>>>>> > but does not ensure that data is synced to disk. This is
>>>>>> > faster and more near-realtime friendly than a hard commit.
>>>>>> > -->
>>>>>> >
>>>>>> > <autoSoftCommit>
>>>>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>>>> > </autoSoftCommit>
>>>>>> >
>>>>>> > In every replica, tlog is increasing for more than 200GB which
>>>>>> exhaust disk
>>>>>> > space. Please suggest someting
>>>>>> >
>>>>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > Hi Guys
>>>>>> > >
>>>>>> > > We have faced an issue where tlog size is increasing
>>>>>> unnecessarily. We
>>>>>> > are
>>>>>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>>>>>> commit
>>>>>> > > also,
>>>>>> > >
>>>>>> > > solr cloud: 6.6
>>>>>> > > zk: 3.4.10
>>>>>> > > shards: 2, replication factor= 2
>>>>>> > >
>>>>>> > >
>>>>>> > > solrconfig,
>>>>>> > >
>>>>>> > > <autoCommit>
>>>>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>>>> > > <maxDocs>10000</maxDocs>
>>>>>> > > <openSearcher>false</openSearcher>
>>>>>> > >
>>>>>> > >
>>>>>> > > </autoCommit>
>>>>>> > >
>>>>>> > > <!-- softAutoCommit is like autoCommit except it causes a
>>>>>> > > 'soft' commit which only ensures that changes are visible
>>>>>> > > but does not ensure that data is synced to disk. This is
>>>>>> > > faster and more near-realtime friendly than a hard
>>>>>> commit.
>>>>>> > > -->
>>>>>> > >
>>>>>> > > <autoSoftCommit>
>>>>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>>>> > > </autoSoftCommit>
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: tlog size issue- solr cloud 6.6
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
Did you try to force a hard commit in order to see the inpact on tlog ?
http://localhost:8983/solr/[collection_name]/update?commit=true
Did you read this article ?
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Regards
Dominique
Le lun. 22 mars 2021 à 17:29, Ritvik Sharma <ri...@gmail.com> a
écrit :
> Hi Dominique
>
> Any suggestions?
>
> On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <ri...@gmail.com> wrote:
>
>> HI Dominique
>>
>> softcommit=false is coming in logs,
>>
>>
>> INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1
>> x:solrcollection_shard2_replica1]
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
>> [solrcollection_shard2_replica1] webapp=/solr path=/update
>> params={update.distrib=FROMLEADER&waitSearcher=true&
>> *openSearcher=true&commit=true&softCommit=false*&distrib.from=
>> http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
>> 0 676
>> INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1
>> x:solrcollection_shard2_replica1]
>> org.apache.solr.update.DirectUpdateHandler2; start
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>>
>>
>> On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <do...@eolya.fr>
>> wrote:
>>
>>> Hi,
>>>
>>> If your tlogs grow, It looks like hard commit are never executed.
>>>
>>> Are you seeing hard commits in logs (grep 'softCommit=false') ?
>>>
>>> What is it happening if you force a hard commit ? Are tlog truncated ?
>>> http://localhost:8983/solr/[collection_name]/update?commit=true
>>>
>>> Did you read this article ?
>>>
>>> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ri...@gmail.com> a
>>> écrit :
>>>
>>>> Hi Dominique
>>>>
>>>> Heavy query means high query rate on solr.
>>>>
>>>> Honestly for 2-3 days we have stopped the queries on solr only we are
>>>> doing indexing !!!
>>>> As you see we have enabled Hard commit also to decrease tlog log size
>>>> or more and the same is written on solr docs. Still the same behaviour is
>>>> occurring on replicas. In some replicas, tlog size is of 300 GB.
>>>>
>>>> We may increase softcommit timeout but as per my understanding it only
>>>> helps in searching availability prior to commit to indexes.
>>>>
>>>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
>>>> dominique.bejean@eolya.fr> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> By heavy query, do you mean you have a high query rate and/or you need
>>>>> index update be available fast after few seconds (NRT search) ?
>>>>>
>>>>> Do you see the hard commits in logs ?
>>>>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>>>>
>>>>> Regards
>>>>>
>>>>> Dominique
>>>>>
>>>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>> > Hi Guys
>>>>> >
>>>>> > We have faced an issue where tlog size is increasing unnecessarily.
>>>>> We are
>>>>> > using a "heavy indexing, heavy query" approach. We enabled hard
>>>>> commit
>>>>> > also,
>>>>> >
>>>>> > solr cloud: 6.6
>>>>> > zk: 3.4.10
>>>>> > shards: 2, replication factor= 2
>>>>> >
>>>>> >
>>>>> > solrconfig,
>>>>> >
>>>>> > <autoCommit>
>>>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>>> > <maxDocs>10000</maxDocs>
>>>>> > <openSearcher>false</openSearcher>
>>>>> >
>>>>> >
>>>>> > </autoCommit>
>>>>> >
>>>>> > <!-- softAutoCommit is like autoCommit except it causes a
>>>>> > 'soft' commit which only ensures that changes are visible
>>>>> > but does not ensure that data is synced to disk. This is
>>>>> > faster and more near-realtime friendly than a hard commit.
>>>>> > -->
>>>>> >
>>>>> > <autoSoftCommit>
>>>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>>> > </autoSoftCommit>
>>>>> >
>>>>> > In every replica, tlog is increasing for more than 200GB which
>>>>> exhaust disk
>>>>> > space. Please suggest someting
>>>>> >
>>>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Hi Guys
>>>>> > >
>>>>> > > We have faced an issue where tlog size is increasing
>>>>> unnecessarily. We
>>>>> > are
>>>>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>>>>> commit
>>>>> > > also,
>>>>> > >
>>>>> > > solr cloud: 6.6
>>>>> > > zk: 3.4.10
>>>>> > > shards: 2, replication factor= 2
>>>>> > >
>>>>> > >
>>>>> > > solrconfig,
>>>>> > >
>>>>> > > <autoCommit>
>>>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>>> > > <maxDocs>10000</maxDocs>
>>>>> > > <openSearcher>false</openSearcher>
>>>>> > >
>>>>> > >
>>>>> > > </autoCommit>
>>>>> > >
>>>>> > > <!-- softAutoCommit is like autoCommit except it causes a
>>>>> > > 'soft' commit which only ensures that changes are visible
>>>>> > > but does not ensure that data is synced to disk. This is
>>>>> > > faster and more near-realtime friendly than a hard commit.
>>>>> > > -->
>>>>> > >
>>>>> > > <autoSoftCommit>
>>>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>>> > > </autoSoftCommit>
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: tlog size issue- solr cloud 6.6
Posted by Ritvik Sharma <ri...@gmail.com>.
Hi Dominique
Any suggestions?
On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <ri...@gmail.com> wrote:
> HI Dominique
>
> softcommit=false is coming in logs,
>
>
> INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1
> x:solrcollection_shard2_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [solrcollection_shard2_replica1] webapp=/solr path=/update
> params={update.distrib=FROMLEADER&waitSearcher=true&
> *openSearcher=true&commit=true&softCommit=false*&distrib.from=
> http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 676
> INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1
> x:solrcollection_shard2_replica1]
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
>
> On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <do...@eolya.fr>
> wrote:
>
>> Hi,
>>
>> If your tlogs grow, It looks like hard commit are never executed.
>>
>> Are you seeing hard commits in logs (grep 'softCommit=false') ?
>>
>> What is it happening if you force a hard commit ? Are tlog truncated ?
>> http://localhost:8983/solr/[collection_name]/update?commit=true
>>
>> Did you read this article ?
>>
>> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> Regards
>>
>> Dominique
>>
>>
>> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ri...@gmail.com> a
>> écrit :
>>
>>> Hi Dominique
>>>
>>> Heavy query means high query rate on solr.
>>>
>>> Honestly for 2-3 days we have stopped the queries on solr only we are
>>> doing indexing !!!
>>> As you see we have enabled Hard commit also to decrease tlog log size or
>>> more and the same is written on solr docs. Still the same behaviour is
>>> occurring on replicas. In some replicas, tlog size is of 300 GB.
>>>
>>> We may increase softcommit timeout but as per my understanding it only
>>> helps in searching availability prior to commit to indexes.
>>>
>>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
>>> dominique.bejean@eolya.fr> wrote:
>>>
>>>> Hi,
>>>>
>>>> By heavy query, do you mean you have a high query rate and/or you need
>>>> index update be available fast after few seconds (NRT search) ?
>>>>
>>>> Do you see the hard commits in logs ?
>>>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>>>
>>>> Regards
>>>>
>>>> Dominique
>>>>
>>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
>>>> écrit :
>>>>
>>>> > Hi Guys
>>>> >
>>>> > We have faced an issue where tlog size is increasing unnecessarily.
>>>> We are
>>>> > using a "heavy indexing, heavy query" approach. We enabled hard
>>>> commit
>>>> > also,
>>>> >
>>>> > solr cloud: 6.6
>>>> > zk: 3.4.10
>>>> > shards: 2, replication factor= 2
>>>> >
>>>> >
>>>> > solrconfig,
>>>> >
>>>> > <autoCommit>
>>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>> > <maxDocs>10000</maxDocs>
>>>> > <openSearcher>false</openSearcher>
>>>> >
>>>> >
>>>> > </autoCommit>
>>>> >
>>>> > <!-- softAutoCommit is like autoCommit except it causes a
>>>> > 'soft' commit which only ensures that changes are visible
>>>> > but does not ensure that data is synced to disk. This is
>>>> > faster and more near-realtime friendly than a hard commit.
>>>> > -->
>>>> >
>>>> > <autoSoftCommit>
>>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>> > </autoSoftCommit>
>>>> >
>>>> > In every replica, tlog is increasing for more than 200GB which
>>>> exhaust disk
>>>> > space. Please suggest someting
>>>> >
>>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Hi Guys
>>>> > >
>>>> > > We have faced an issue where tlog size is increasing unnecessarily.
>>>> We
>>>> > are
>>>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>>>> commit
>>>> > > also,
>>>> > >
>>>> > > solr cloud: 6.6
>>>> > > zk: 3.4.10
>>>> > > shards: 2, replication factor= 2
>>>> > >
>>>> > >
>>>> > > solrconfig,
>>>> > >
>>>> > > <autoCommit>
>>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>>> > > <maxDocs>10000</maxDocs>
>>>> > > <openSearcher>false</openSearcher>
>>>> > >
>>>> > >
>>>> > > </autoCommit>
>>>> > >
>>>> > > <!-- softAutoCommit is like autoCommit except it causes a
>>>> > > 'soft' commit which only ensures that changes are visible
>>>> > > but does not ensure that data is synced to disk. This is
>>>> > > faster and more near-realtime friendly than a hard commit.
>>>> > > -->
>>>> > >
>>>> > > <autoSoftCommit>
>>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>>> > > </autoSoftCommit>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
Re: tlog size issue- solr cloud 6.6
Posted by Ritvik Sharma <ri...@gmail.com>.
HI Dominique
softcommit=false is coming in logs,
INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1
x:solrcollection_shard2_replica1]
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
[solrcollection_shard2_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&
*openSearcher=true&commit=true&softCommit=false*&distrib.from=
http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 676
INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1
x:solrcollection_shard2_replica1]
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <do...@eolya.fr>
wrote:
> Hi,
>
> If your tlogs grow, It looks like hard commit are never executed.
>
> Are you seeing hard commits in logs (grep 'softCommit=false') ?
>
> What is it happening if you force a hard commit ? Are tlog truncated ?
> http://localhost:8983/solr/[collection_name]/update?commit=true
>
> Did you read this article ?
>
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Regards
>
> Dominique
>
>
> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ri...@gmail.com> a
> écrit :
>
>> Hi Dominique
>>
>> Heavy query means high query rate on solr.
>>
>> Honestly for 2-3 days we have stopped the queries on solr only we are
>> doing indexing !!!
>> As you see we have enabled Hard commit also to decrease tlog log size or
>> more and the same is written on solr docs. Still the same behaviour is
>> occurring on replicas. In some replicas, tlog size is of 300 GB.
>>
>> We may increase softcommit timeout but as per my understanding it only
>> helps in searching availability prior to commit to indexes.
>>
>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
>> dominique.bejean@eolya.fr> wrote:
>>
>>> Hi,
>>>
>>> By heavy query, do you mean you have a high query rate and/or you need
>>> index update be available fast after few seconds (NRT search) ?
>>>
>>> Do you see the hard commits in logs ?
>>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
>>> écrit :
>>>
>>> > Hi Guys
>>> >
>>> > We have faced an issue where tlog size is increasing unnecessarily. We
>>> are
>>> > using a "heavy indexing, heavy query" approach. We enabled hard commit
>>> > also,
>>> >
>>> > solr cloud: 6.6
>>> > zk: 3.4.10
>>> > shards: 2, replication factor= 2
>>> >
>>> >
>>> > solrconfig,
>>> >
>>> > <autoCommit>
>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> > <maxDocs>10000</maxDocs>
>>> > <openSearcher>false</openSearcher>
>>> >
>>> >
>>> > </autoCommit>
>>> >
>>> > <!-- softAutoCommit is like autoCommit except it causes a
>>> > 'soft' commit which only ensures that changes are visible
>>> > but does not ensure that data is synced to disk. This is
>>> > faster and more near-realtime friendly than a hard commit.
>>> > -->
>>> >
>>> > <autoSoftCommit>
>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>> > </autoSoftCommit>
>>> >
>>> > In every replica, tlog is increasing for more than 200GB which exhaust
>>> disk
>>> > space. Please suggest someting
>>> >
>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Guys
>>> > >
>>> > > We have faced an issue where tlog size is increasing unnecessarily.
>>> We
>>> > are
>>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>>> commit
>>> > > also,
>>> > >
>>> > > solr cloud: 6.6
>>> > > zk: 3.4.10
>>> > > shards: 2, replication factor= 2
>>> > >
>>> > >
>>> > > solrconfig,
>>> > >
>>> > > <autoCommit>
>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> > > <maxDocs>10000</maxDocs>
>>> > > <openSearcher>false</openSearcher>
>>> > >
>>> > >
>>> > > </autoCommit>
>>> > >
>>> > > <!-- softAutoCommit is like autoCommit except it causes a
>>> > > 'soft' commit which only ensures that changes are visible
>>> > > but does not ensure that data is synced to disk. This is
>>> > > faster and more near-realtime friendly than a hard commit.
>>> > > -->
>>> > >
>>> > > <autoSoftCommit>
>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>> > > </autoSoftCommit>
>>> > >
>>> > >
>>> >
>>>
>>
Re: tlog size issue- solr cloud 6.6
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
If your tlogs grow, It looks like hard commit are never executed.
Are you seeing hard commits in logs (grep 'softCommit=false') ?
What is it happening if you force a hard commit ? Are tlog truncated ?
http://localhost:8983/solr/[collection_name]/update?commit=true
Did you read this article ?
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Regards
Dominique
Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <ri...@gmail.com> a
écrit :
> Hi Dominique
>
> Heavy query means high query rate on solr.
>
> Honestly for 2-3 days we have stopped the queries on solr only we are
> doing indexing !!!
> As you see we have enabled Hard commit also to decrease tlog log size or
> more and the same is written on solr docs. Still the same behaviour is
> occurring on replicas. In some replicas, tlog size is of 300 GB.
>
> We may increase softcommit timeout but as per my understanding it only
> helps in searching availability prior to commit to indexes.
>
> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
> dominique.bejean@eolya.fr> wrote:
>
>> Hi,
>>
>> By heavy query, do you mean you have a high query rate and/or you need
>> index update be available fast after few seconds (NRT search) ?
>>
>> Do you see the hard commits in logs ?
>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>
>> Regards
>>
>> Dominique
>>
>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
>> écrit :
>>
>> > Hi Guys
>> >
>> > We have faced an issue where tlog size is increasing unnecessarily. We
>> are
>> > using a "heavy indexing, heavy query" approach. We enabled hard commit
>> > also,
>> >
>> > solr cloud: 6.6
>> > zk: 3.4.10
>> > shards: 2, replication factor= 2
>> >
>> >
>> > solrconfig,
>> >
>> > <autoCommit>
>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>> > <maxDocs>10000</maxDocs>
>> > <openSearcher>false</openSearcher>
>> >
>> >
>> > </autoCommit>
>> >
>> > <!-- softAutoCommit is like autoCommit except it causes a
>> > 'soft' commit which only ensures that changes are visible
>> > but does not ensure that data is synced to disk. This is
>> > faster and more near-realtime friendly than a hard commit.
>> > -->
>> >
>> > <autoSoftCommit>
>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>> > </autoSoftCommit>
>> >
>> > In every replica, tlog is increasing for more than 200GB which exhaust
>> disk
>> > space. Please suggest someting
>> >
>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
>> wrote:
>> >
>> > > Hi Guys
>> > >
>> > > We have faced an issue where tlog size is increasing unnecessarily. We
>> > are
>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>> commit
>> > > also,
>> > >
>> > > solr cloud: 6.6
>> > > zk: 3.4.10
>> > > shards: 2, replication factor= 2
>> > >
>> > >
>> > > solrconfig,
>> > >
>> > > <autoCommit>
>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>> > > <maxDocs>10000</maxDocs>
>> > > <openSearcher>false</openSearcher>
>> > >
>> > >
>> > > </autoCommit>
>> > >
>> > > <!-- softAutoCommit is like autoCommit except it causes a
>> > > 'soft' commit which only ensures that changes are visible
>> > > but does not ensure that data is synced to disk. This is
>> > > faster and more near-realtime friendly than a hard commit.
>> > > -->
>> > >
>> > > <autoSoftCommit>
>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>> > > </autoSoftCommit>
>> > >
>> > >
>> >
>>
>
Re: tlog size issue- solr cloud 6.6
Posted by Ritvik Sharma <ri...@gmail.com>.
Hi Dominique
Heavy query means high query rate on solr.
Honestly for 2-3 days we have stopped the queries on solr only we are doing
indexing !!!
As you see we have enabled Hard commit also to decrease tlog log size or
more and the same is written on solr docs. Still the same behaviour is
occurring on replicas. In some replicas, tlog size is of 300 GB.
We may increase softcommit timeout but as per my understanding it only
helps in searching availability prior to commit to indexes.
On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <do...@eolya.fr>
wrote:
> Hi,
>
> By heavy query, do you mean you have a high query rate and/or you need
> index update be available fast after few seconds (NRT search) ?
>
> Do you see the hard commits in logs ?
> Can you try to increase autoSoftCommit to 30 seconds or more ?
>
> Regards
>
> Dominique
>
> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
> écrit :
>
> > Hi Guys
> >
> > We have faced an issue where tlog size is increasing unnecessarily. We
> are
> > using a "heavy indexing, heavy query" approach. We enabled hard commit
> > also,
> >
> > solr cloud: 6.6
> > zk: 3.4.10
> > shards: 2, replication factor= 2
> >
> >
> > solrconfig,
> >
> > <autoCommit>
> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> > <maxDocs>10000</maxDocs>
> > <openSearcher>false</openSearcher>
> >
> >
> > </autoCommit>
> >
> > <!-- softAutoCommit is like autoCommit except it causes a
> > 'soft' commit which only ensures that changes are visible
> > but does not ensure that data is synced to disk. This is
> > faster and more near-realtime friendly than a hard commit.
> > -->
> >
> > <autoSoftCommit>
> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
> > </autoSoftCommit>
> >
> > In every replica, tlog is increasing for more than 200GB which exhaust
> disk
> > space. Please suggest someting
> >
> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com>
> wrote:
> >
> > > Hi Guys
> > >
> > > We have faced an issue where tlog size is increasing unnecessarily. We
> > are
> > > using a "heavy indexing, heavy query" approach. We enabled hard commit
> > > also,
> > >
> > > solr cloud: 6.6
> > > zk: 3.4.10
> > > shards: 2, replication factor= 2
> > >
> > >
> > > solrconfig,
> > >
> > > <autoCommit>
> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> > > <maxDocs>10000</maxDocs>
> > > <openSearcher>false</openSearcher>
> > >
> > >
> > > </autoCommit>
> > >
> > > <!-- softAutoCommit is like autoCommit except it causes a
> > > 'soft' commit which only ensures that changes are visible
> > > but does not ensure that data is synced to disk. This is
> > > faster and more near-realtime friendly than a hard commit.
> > > -->
> > >
> > > <autoSoftCommit>
> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
> > > </autoSoftCommit>
> > >
> > >
> >
>
Re: tlog size issue- solr cloud 6.6
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
By heavy query, do you mean you have a high query rate and/or you need
index update be available fast after few seconds (NRT search) ?
Do you see the hard commits in logs ?
Can you try to increase autoSoftCommit to 30 seconds or more ?
Regards
Dominique
Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <ri...@gmail.com> a
écrit :
> Hi Guys
>
> We have faced an issue where tlog size is increasing unnecessarily. We are
> using a "heavy indexing, heavy query" approach. We enabled hard commit
> also,
>
> solr cloud: 6.6
> zk: 3.4.10
> shards: 2, replication factor= 2
>
>
> solrconfig,
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <maxDocs>10000</maxDocs>
> <openSearcher>false</openSearcher>
>
>
> </autoCommit>
>
> <!-- softAutoCommit is like autoCommit except it causes a
> 'soft' commit which only ensures that changes are visible
> but does not ensure that data is synced to disk. This is
> faster and more near-realtime friendly than a hard commit.
> -->
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
> </autoSoftCommit>
>
> In every replica, tlog is increasing for more than 200GB which exhaust disk
> space. Please suggest someting
>
> On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com> wrote:
>
> > Hi Guys
> >
> > We have faced an issue where tlog size is increasing unnecessarily. We
> are
> > using a "heavy indexing, heavy query" approach. We enabled hard commit
> > also,
> >
> > solr cloud: 6.6
> > zk: 3.4.10
> > shards: 2, replication factor= 2
> >
> >
> > solrconfig,
> >
> > <autoCommit>
> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> > <maxDocs>10000</maxDocs>
> > <openSearcher>false</openSearcher>
> >
> >
> > </autoCommit>
> >
> > <!-- softAutoCommit is like autoCommit except it causes a
> > 'soft' commit which only ensures that changes are visible
> > but does not ensure that data is synced to disk. This is
> > faster and more near-realtime friendly than a hard commit.
> > -->
> >
> > <autoSoftCommit>
> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
> > </autoSoftCommit>
> >
> >
>
Re: tlog size issue- solr cloud 6.6
Posted by Ritvik Sharma <ri...@gmail.com>.
Hi Guys
We have faced an issue where tlog size is increasing unnecessarily. We are
using a "heavy indexing, heavy query" approach. We enabled hard commit
also,
solr cloud: 6.6
zk: 3.4.10
shards: 2, replication factor= 2
solrconfig,
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<maxDocs>10000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<!-- softAutoCommit is like autoCommit except it causes a
'soft' commit which only ensures that changes are visible
but does not ensure that data is synced to disk. This is
faster and more near-realtime friendly than a hard commit.
-->
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
</autoSoftCommit>
In every replica, tlog is increasing for more than 200GB which exhaust disk
space. Please suggest someting
On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <ri...@gmail.com> wrote:
> Hi Guys
>
> We have faced an issue where tlog size is increasing unnecessarily. We are
> using a "heavy indexing, heavy query" approach. We enabled hard commit
> also,
>
> solr cloud: 6.6
> zk: 3.4.10
> shards: 2, replication factor= 2
>
>
> solrconfig,
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <maxDocs>10000</maxDocs>
> <openSearcher>false</openSearcher>
>
>
> </autoCommit>
>
> <!-- softAutoCommit is like autoCommit except it causes a
> 'soft' commit which only ensures that changes are visible
> but does not ensure that data is synced to disk. This is
> faster and more near-realtime friendly than a hard commit.
> -->
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
> </autoSoftCommit>
>
>