You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Nikhilesh Jannu <ni...@predictspring.com> on 2022/06/29 06:09:49 UTC
SOLR TRA Collection Question on recovery
Dear Users,
We are using the SOLR TRA collection for capturing the logs. We are
writing the logs to SOLR using the Rest API in a batch of 100 and also we
are using SOFT commit interval of 15000 and Hard commit interval of 60000.
Solr Version : 8.11.1.
When we restart the SOLR node in the cloud the current day's collection
goes in recovery mode and we see the following logs. It takes a long time
for the recovery process to complete. Not sure how to avoid it. Any
suggestions ?
Sample of the logs below.
2022-06-29 06:03:27.500 INFO
(recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29 s:shard1
r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log replay
status
tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
refcount=3} active=false starting pos=0 current pos=1119002110 current
size=3287152529 % read=34.0
Regards,
Nikhilesh Jannu
Re: SOLR TRA Collection Question on recovery
Posted by Radu Gheorghe <ra...@sematext.com>.
Hi Nikhilesh,
It's not necessary to stop traffic, it's just that the update traffic will
make replication more difficult, which is to be expected. Now, if you have
A LOT of update traffic and recovery is slow because of it, it would help
to pause the traffic for sure.
Best regards,
Radu
--
Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
Sematext Cloud - Full Stack Observability
https://sematext.com/ <http://sematext.com/>
On Thu, Jul 7, 2022 at 6:41 AM Nikhilesh Jannu <ni...@predictspring.com>
wrote:
> Hi Radu,
>
> I will check again when this happens if the recovery is going on. So if the
> recovery is going on should we be make sure there is no traffic sending
> data to be indexed to the collection ?
>
> Regards,
> Nikhilesh Jannu
>
>
>
> Nikhilesh Jannu Principal Software Engineer 405.609.4259
> <(405)%20741-9895>
>
>
> On Wed, Jul 6, 2022 at 1:51 AM Radu Gheorghe <ra...@sematext.com>
> wrote:
>
> > Hi Nikhilesh,
> >
> > Something sounds off, your transaction logs should be rotated unless
> > there's a recovery going on at that time. Is it?
> >
> > If not, can you double check:
> > - your solrconfig.xml - maybe you can post the relevant part here?
> > - your config overlay - maybe there's something that was posted by the
> > Config API that overwrote your solconfig settings?
> >
> > Best regards,
> > Radu
> > --
> > Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> > Sematext Cloud - Full Stack Observability
> > http://sematext.com/
> >
> >
> > On Tue, Jul 5, 2022 at 9:46 PM Nikhilesh Jannu <
> > nikhilesh@predictspring.com>
> > wrote:
> >
> > > Hi Radu,
> > >
> > > The issue happened again today and I see huge logs getting created even
> > > with the config where we want smaller files and smaller intervals.
> > >
> > > This happens in a particular scenario where we are trying to index
> > > documents. That API call is timing out and there is a connection
> > > expectation. The Tlogs keep growing in size. The collection is healthy
> > but
> > > we are constantly adding documents and there are no errors. Only part I
> > see
> > > is the Tlogs are now at 4GB or so.
> > >
> > > Regards,
> > > Nikhilesh Jannu
> > >
> > > Regards,
> > > Nikhilesh Jannu
> > > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > > <(405)%20741-9895>
> > >
> > >
> > > On Wed, Jun 29, 2022 at 10:15 PM Radu Gheorghe <
> > radu.gheorghe@sematext.com
> > > >
> > > wrote:
> > >
> > > > You're welcome, Nikhilesh!
> > > >
> > > > Best wishes,
> > > > Radu
> > > > --
> > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> > Training
> > > > Sematext Cloud - Full Stack Observability
> > > > http://sematext.com/
> > > >
> > > >
> > > > On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu <
> > > > nikhilesh@predictspring.com> wrote:
> > > >
> > > > > Thank you Radu for the quick response. I have updated the values as
> > > > > you suggested.
> > > > >
> > > > >
> > > > > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > > > > <(405)%20741-9895>
> > > > >
> > > > >
> > > > > On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <
> > > > radu.gheorghe@sematext.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Nikhilesh,
> > > > > >
> > > > > > Try hard-committing more often. This way you'll have smaller tlog
> > > files
> > > > > and
> > > > > > there will be less data to recover. My suggestion is to add a
> > maxSize
> > > > > > constraint to autoCommit. 100MB is a good rule of thumb, makes
> sure
> > > you
> > > > > > don't replay more than 100MB worth of data (even if you have an
> > > > indexing
> > > > > > spike).
> > > > > >
> > > > > > Best regards,
> > > > > > Radu
> > > > > > --
> > > > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> > > > Training
> > > > > > Sematext Cloud - Full Stack Observability
> > > > > > http://sematext.com/
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> > > > > > nikhilesh@predictspring.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear Users,
> > > > > > >
> > > > > > > We are using the SOLR TRA collection for capturing the logs.
> We
> > > are
> > > > > > > writing the logs to SOLR using the Rest API in a batch of 100
> and
> > > > also
> > > > > we
> > > > > > > are using SOFT commit interval of 15000 and Hard commit
> interval
> > of
> > > > > > 60000.
> > > > > > >
> > > > > > > Solr Version : 8.11.1.
> > > > > > >
> > > > > > > When we restart the SOLR node in the cloud the current day's
> > > > collection
> > > > > > > goes in recovery mode and we see the following logs. It takes a
> > > long
> > > > > time
> > > > > > > for the recovery process to complete. Not sure how to avoid it.
> > Any
> > > > > > > suggestions ?
> > > > > > >
> > > > > > > Sample of the logs below.
> > > > > > >
> > > > > > > 2022-06-29 06:03:27.500 INFO
> > > > > > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983
> > _solr
> > > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1
> c:logs__TRA__2022-06-29
> > > > > > s:shard1
> > > > > > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog
> log
> > > > replay
> > > > > > > status
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > > > > > > refcount=3} active=false starting pos=0 current pos=1119002110
> > > > current
> > > > > > > size=3287152529 % read=34.0
> > > > > > >
> > > > > > > Regards,
> > > > > > > Nikhilesh Jannu
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Nikhilesh Jannu <ni...@predictspring.com>.
Hi Radu,
I will check again when this happens if the recovery is going on. So if the
recovery is going on should we be make sure there is no traffic sending
data to be indexed to the collection ?
Regards,
Nikhilesh Jannu
Nikhilesh Jannu Principal Software Engineer 405.609.4259
<(405)%20741-9895>
On Wed, Jul 6, 2022 at 1:51 AM Radu Gheorghe <ra...@sematext.com>
wrote:
> Hi Nikhilesh,
>
> Something sounds off, your transaction logs should be rotated unless
> there's a recovery going on at that time. Is it?
>
> If not, can you double check:
> - your solrconfig.xml - maybe you can post the relevant part here?
> - your config overlay - maybe there's something that was posted by the
> Config API that overwrote your solconfig settings?
>
> Best regards,
> Radu
> --
> Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> Sematext Cloud - Full Stack Observability
> http://sematext.com/
>
>
> On Tue, Jul 5, 2022 at 9:46 PM Nikhilesh Jannu <
> nikhilesh@predictspring.com>
> wrote:
>
> > Hi Radu,
> >
> > The issue happened again today and I see huge logs getting created even
> > with the config where we want smaller files and smaller intervals.
> >
> > This happens in a particular scenario where we are trying to index
> > documents. That API call is timing out and there is a connection
> > expectation. The Tlogs keep growing in size. The collection is healthy
> but
> > we are constantly adding documents and there are no errors. Only part I
> see
> > is the Tlogs are now at 4GB or so.
> >
> > Regards,
> > Nikhilesh Jannu
> >
> > Regards,
> > Nikhilesh Jannu
> > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > <(405)%20741-9895>
> >
> >
> > On Wed, Jun 29, 2022 at 10:15 PM Radu Gheorghe <
> radu.gheorghe@sematext.com
> > >
> > wrote:
> >
> > > You're welcome, Nikhilesh!
> > >
> > > Best wishes,
> > > Radu
> > > --
> > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> Training
> > > Sematext Cloud - Full Stack Observability
> > > http://sematext.com/
> > >
> > >
> > > On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu <
> > > nikhilesh@predictspring.com> wrote:
> > >
> > > > Thank you Radu for the quick response. I have updated the values as
> > > > you suggested.
> > > >
> > > >
> > > > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > > > <(405)%20741-9895>
> > > >
> > > >
> > > > On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <
> > > radu.gheorghe@sematext.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Nikhilesh,
> > > > >
> > > > > Try hard-committing more often. This way you'll have smaller tlog
> > files
> > > > and
> > > > > there will be less data to recover. My suggestion is to add a
> maxSize
> > > > > constraint to autoCommit. 100MB is a good rule of thumb, makes sure
> > you
> > > > > don't replay more than 100MB worth of data (even if you have an
> > > indexing
> > > > > spike).
> > > > >
> > > > > Best regards,
> > > > > Radu
> > > > > --
> > > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> > > Training
> > > > > Sematext Cloud - Full Stack Observability
> > > > > http://sematext.com/
> > > > >
> > > > >
> > > > > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> > > > > nikhilesh@predictspring.com>
> > > > > wrote:
> > > > >
> > > > > > Dear Users,
> > > > > >
> > > > > > We are using the SOLR TRA collection for capturing the logs. We
> > are
> > > > > > writing the logs to SOLR using the Rest API in a batch of 100 and
> > > also
> > > > we
> > > > > > are using SOFT commit interval of 15000 and Hard commit interval
> of
> > > > > 60000.
> > > > > >
> > > > > > Solr Version : 8.11.1.
> > > > > >
> > > > > > When we restart the SOLR node in the cloud the current day's
> > > collection
> > > > > > goes in recovery mode and we see the following logs. It takes a
> > long
> > > > time
> > > > > > for the recovery process to complete. Not sure how to avoid it.
> Any
> > > > > > suggestions ?
> > > > > >
> > > > > > Sample of the logs below.
> > > > > >
> > > > > > 2022-06-29 06:03:27.500 INFO
> > > > > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983
> _solr
> > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29
> > > > > s:shard1
> > > > > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > > > > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log
> > > replay
> > > > > > status
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > > > > > refcount=3} active=false starting pos=0 current pos=1119002110
> > > current
> > > > > > size=3287152529 % read=34.0
> > > > > >
> > > > > > Regards,
> > > > > > Nikhilesh Jannu
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Radu Gheorghe <ra...@sematext.com>.
Hi Nikhilesh,
Something sounds off, your transaction logs should be rotated unless
there's a recovery going on at that time. Is it?
If not, can you double check:
- your solrconfig.xml - maybe you can post the relevant part here?
- your config overlay - maybe there's something that was posted by the
Config API that overwrote your solconfig settings?
Best regards,
Radu
--
Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
Sematext Cloud - Full Stack Observability
http://sematext.com/
On Tue, Jul 5, 2022 at 9:46 PM Nikhilesh Jannu <ni...@predictspring.com>
wrote:
> Hi Radu,
>
> The issue happened again today and I see huge logs getting created even
> with the config where we want smaller files and smaller intervals.
>
> This happens in a particular scenario where we are trying to index
> documents. That API call is timing out and there is a connection
> expectation. The Tlogs keep growing in size. The collection is healthy but
> we are constantly adding documents and there are no errors. Only part I see
> is the Tlogs are now at 4GB or so.
>
> Regards,
> Nikhilesh Jannu
>
> Regards,
> Nikhilesh Jannu
> Nikhilesh Jannu Principal Software Engineer 405.609.4259
> <(405)%20741-9895>
>
>
> On Wed, Jun 29, 2022 at 10:15 PM Radu Gheorghe <radu.gheorghe@sematext.com
> >
> wrote:
>
> > You're welcome, Nikhilesh!
> >
> > Best wishes,
> > Radu
> > --
> > Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> > Sematext Cloud - Full Stack Observability
> > http://sematext.com/
> >
> >
> > On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu <
> > nikhilesh@predictspring.com> wrote:
> >
> > > Thank you Radu for the quick response. I have updated the values as
> > > you suggested.
> > >
> > >
> > > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > > <(405)%20741-9895>
> > >
> > >
> > > On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <
> > radu.gheorghe@sematext.com
> > > >
> > > wrote:
> > >
> > > > Hi Nikhilesh,
> > > >
> > > > Try hard-committing more often. This way you'll have smaller tlog
> files
> > > and
> > > > there will be less data to recover. My suggestion is to add a maxSize
> > > > constraint to autoCommit. 100MB is a good rule of thumb, makes sure
> you
> > > > don't replay more than 100MB worth of data (even if you have an
> > indexing
> > > > spike).
> > > >
> > > > Best regards,
> > > > Radu
> > > > --
> > > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> > Training
> > > > Sematext Cloud - Full Stack Observability
> > > > http://sematext.com/
> > > >
> > > >
> > > > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> > > > nikhilesh@predictspring.com>
> > > > wrote:
> > > >
> > > > > Dear Users,
> > > > >
> > > > > We are using the SOLR TRA collection for capturing the logs. We
> are
> > > > > writing the logs to SOLR using the Rest API in a batch of 100 and
> > also
> > > we
> > > > > are using SOFT commit interval of 15000 and Hard commit interval of
> > > > 60000.
> > > > >
> > > > > Solr Version : 8.11.1.
> > > > >
> > > > > When we restart the SOLR node in the cloud the current day's
> > collection
> > > > > goes in recovery mode and we see the following logs. It takes a
> long
> > > time
> > > > > for the recovery process to complete. Not sure how to avoid it. Any
> > > > > suggestions ?
> > > > >
> > > > > Sample of the logs below.
> > > > >
> > > > > 2022-06-29 06:03:27.500 INFO
> > > > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
> > > > > x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29
> > > > s:shard1
> > > > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > > > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log
> > replay
> > > > > status
> > > > >
> > > > >
> > > >
> > >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > > > > refcount=3} active=false starting pos=0 current pos=1119002110
> > current
> > > > > size=3287152529 % read=34.0
> > > > >
> > > > > Regards,
> > > > > Nikhilesh Jannu
> > > > >
> > > >
> > >
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Nikhilesh Jannu <ni...@predictspring.com>.
Hi Radu,
The issue happened again today and I see huge logs getting created even
with the config where we want smaller files and smaller intervals.
This happens in a particular scenario where we are trying to index
documents. That API call is timing out and there is a connection
expectation. The Tlogs keep growing in size. The collection is healthy but
we are constantly adding documents and there are no errors. Only part I see
is the Tlogs are now at 4GB or so.
Regards,
Nikhilesh Jannu
Regards,
Nikhilesh Jannu
Nikhilesh Jannu Principal Software Engineer 405.609.4259
<(405)%20741-9895>
On Wed, Jun 29, 2022 at 10:15 PM Radu Gheorghe <ra...@sematext.com>
wrote:
> You're welcome, Nikhilesh!
>
> Best wishes,
> Radu
> --
> Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> Sematext Cloud - Full Stack Observability
> http://sematext.com/
>
>
> On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu <
> nikhilesh@predictspring.com> wrote:
>
> > Thank you Radu for the quick response. I have updated the values as
> > you suggested.
> >
> >
> > Nikhilesh Jannu Principal Software Engineer 405.609.4259
> > <(405)%20741-9895>
> >
> >
> > On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <
> radu.gheorghe@sematext.com
> > >
> > wrote:
> >
> > > Hi Nikhilesh,
> > >
> > > Try hard-committing more often. This way you'll have smaller tlog files
> > and
> > > there will be less data to recover. My suggestion is to add a maxSize
> > > constraint to autoCommit. 100MB is a good rule of thumb, makes sure you
> > > don't replay more than 100MB worth of data (even if you have an
> indexing
> > > spike).
> > >
> > > Best regards,
> > > Radu
> > > --
> > > Elasticsearch/OpenSearch & Solr Consulting, Production Support &
> Training
> > > Sematext Cloud - Full Stack Observability
> > > http://sematext.com/
> > >
> > >
> > > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> > > nikhilesh@predictspring.com>
> > > wrote:
> > >
> > > > Dear Users,
> > > >
> > > > We are using the SOLR TRA collection for capturing the logs. We are
> > > > writing the logs to SOLR using the Rest API in a batch of 100 and
> also
> > we
> > > > are using SOFT commit interval of 15000 and Hard commit interval of
> > > 60000.
> > > >
> > > > Solr Version : 8.11.1.
> > > >
> > > > When we restart the SOLR node in the cloud the current day's
> collection
> > > > goes in recovery mode and we see the following logs. It takes a long
> > time
> > > > for the recovery process to complete. Not sure how to avoid it. Any
> > > > suggestions ?
> > > >
> > > > Sample of the logs below.
> > > >
> > > > 2022-06-29 06:03:27.500 INFO
> > > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
> > > > x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29
> > > s:shard1
> > > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log
> replay
> > > > status
> > > >
> > > >
> > >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > > > refcount=3} active=false starting pos=0 current pos=1119002110
> current
> > > > size=3287152529 % read=34.0
> > > >
> > > > Regards,
> > > > Nikhilesh Jannu
> > > >
> > >
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Radu Gheorghe <ra...@sematext.com>.
You're welcome, Nikhilesh!
Best wishes,
Radu
--
Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
Sematext Cloud - Full Stack Observability
http://sematext.com/
On Wed, Jun 29, 2022 at 10:49 AM Nikhilesh Jannu <
nikhilesh@predictspring.com> wrote:
> Thank you Radu for the quick response. I have updated the values as
> you suggested.
>
>
> Nikhilesh Jannu Principal Software Engineer 405.609.4259
> <(405)%20741-9895>
>
>
> On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <radu.gheorghe@sematext.com
> >
> wrote:
>
> > Hi Nikhilesh,
> >
> > Try hard-committing more often. This way you'll have smaller tlog files
> and
> > there will be less data to recover. My suggestion is to add a maxSize
> > constraint to autoCommit. 100MB is a good rule of thumb, makes sure you
> > don't replay more than 100MB worth of data (even if you have an indexing
> > spike).
> >
> > Best regards,
> > Radu
> > --
> > Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> > Sematext Cloud - Full Stack Observability
> > http://sematext.com/
> >
> >
> > On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> > nikhilesh@predictspring.com>
> > wrote:
> >
> > > Dear Users,
> > >
> > > We are using the SOLR TRA collection for capturing the logs. We are
> > > writing the logs to SOLR using the Rest API in a batch of 100 and also
> we
> > > are using SOFT commit interval of 15000 and Hard commit interval of
> > 60000.
> > >
> > > Solr Version : 8.11.1.
> > >
> > > When we restart the SOLR node in the cloud the current day's collection
> > > goes in recovery mode and we see the following logs. It takes a long
> time
> > > for the recovery process to complete. Not sure how to avoid it. Any
> > > suggestions ?
> > >
> > > Sample of the logs below.
> > >
> > > 2022-06-29 06:03:27.500 INFO
> > > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
> > > x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29
> > s:shard1
> > > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log replay
> > > status
> > >
> > >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > > refcount=3} active=false starting pos=0 current pos=1119002110 current
> > > size=3287152529 % read=34.0
> > >
> > > Regards,
> > > Nikhilesh Jannu
> > >
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Nikhilesh Jannu <ni...@predictspring.com>.
Thank you Radu for the quick response. I have updated the values as
you suggested.
Nikhilesh Jannu Principal Software Engineer 405.609.4259
<(405)%20741-9895>
On Wed, Jun 29, 2022 at 12:03 AM Radu Gheorghe <ra...@sematext.com>
wrote:
> Hi Nikhilesh,
>
> Try hard-committing more often. This way you'll have smaller tlog files and
> there will be less data to recover. My suggestion is to add a maxSize
> constraint to autoCommit. 100MB is a good rule of thumb, makes sure you
> don't replay more than 100MB worth of data (even if you have an indexing
> spike).
>
> Best regards,
> Radu
> --
> Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
> Sematext Cloud - Full Stack Observability
> http://sematext.com/
>
>
> On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <
> nikhilesh@predictspring.com>
> wrote:
>
> > Dear Users,
> >
> > We are using the SOLR TRA collection for capturing the logs. We are
> > writing the logs to SOLR using the Rest API in a batch of 100 and also we
> > are using SOFT commit interval of 15000 and Hard commit interval of
> 60000.
> >
> > Solr Version : 8.11.1.
> >
> > When we restart the SOLR node in the cloud the current day's collection
> > goes in recovery mode and we see the following logs. It takes a long time
> > for the recovery process to complete. Not sure how to avoid it. Any
> > suggestions ?
> >
> > Sample of the logs below.
> >
> > 2022-06-29 06:03:27.500 INFO
> > (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
> > x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29
> s:shard1
> > r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> > x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log replay
> > status
> >
> >
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> > refcount=3} active=false starting pos=0 current pos=1119002110 current
> > size=3287152529 % read=34.0
> >
> > Regards,
> > Nikhilesh Jannu
> >
>
Re: SOLR TRA Collection Question on recovery
Posted by Radu Gheorghe <ra...@sematext.com>.
Hi Nikhilesh,
Try hard-committing more often. This way you'll have smaller tlog files and
there will be less data to recover. My suggestion is to add a maxSize
constraint to autoCommit. 100MB is a good rule of thumb, makes sure you
don't replay more than 100MB worth of data (even if you have an indexing
spike).
Best regards,
Radu
--
Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training
Sematext Cloud - Full Stack Observability
http://sematext.com/
On Wed, Jun 29, 2022 at 9:10 AM Nikhilesh Jannu <ni...@predictspring.com>
wrote:
> Dear Users,
>
> We are using the SOLR TRA collection for capturing the logs. We are
> writing the logs to SOLR using the Rest API in a batch of 100 and also we
> are using SOFT commit interval of 15000 and Hard commit interval of 60000.
>
> Solr Version : 8.11.1.
>
> When we restart the SOLR node in the cloud the current day's collection
> goes in recovery mode and we see the following logs. It takes a long time
> for the recovery process to complete. Not sure how to avoid it. Any
> suggestions ?
>
> Sample of the logs below.
>
> 2022-06-29 06:03:27.500 INFO
> (recoveryExecutor-67-thread-1-processing-n:10.0.42.157:8983_solr
> x:logs__TRA__2022-06-29_shard1_replica_n1 c:logs__TRA__2022-06-29 s:shard1
> r:core_node2) [c:logs__TRA__2022-06-29 s:shard1 r:core_node2
> x:logs__TRA__2022-06-29_shard1_replica_n1] o.a.s.u.UpdateLog log replay
> status
>
> tlog{file=/var/solr/data/logs__TRA__2022-06-29_shard1_replica_n1/data/tlog/tlog.0000000000000000243
> refcount=3} active=false starting pos=0 current pos=1119002110 current
> size=3287152529 % read=34.0
>
> Regards,
> Nikhilesh Jannu
>