You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Saur, Alexandre (ELS-AMS)" <a....@elsevier.com> on 2021/10/06 13:00:07 UTC
Solr 8 autocommit confirmation
Hello there,
I have a quick question regarding Solr 8 autocommit feature. This is my configuration:
<autoCommit>
<maxTime>600000</maxTime>
<openSearcher>true</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
I can check this is true using the console. However, I cannot see any thread doing the autocommit in the logs, nor any metric that gives me the time the autocommit is being done.
Is there any way to know for sure the autocommit is working as expected in Solr 8?
PS: Solr 6 had an entry in the logs that confirmed it was being executed, you just need to grep for "commitScheduler"
Thanks,
Alex
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33158992, Registered in The Netherlands.
Re: Solr 8 autocommit confirmation
Posted by Shankar R <ia...@gmail.com>.
Hi Alex,
You can set the autocommit time limit to 5 seconds. From your client code,
index some content without issuing explicit commit. From admin UI if you
search for the content which u indexed recently you will be able to find
it. It shows that Solr is inssuing autocommit automatically after 5 seconds
and you will be able to search in admin UI because your opensearcher value
is set to true. If you set it to 'false' then even though solr has done
auto commit it will not be visible because solr has not opened a new
searcher and is showing you the content from the cached searcher / previous
searcher.
Hope it helps
Regards
Ravi
On Wed, Oct 6, 2021 at 6:30 PM Saur, Alexandre (ELS-AMS) <
a.saur@elsevier.com> wrote:
> Hello there,
>
> I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any
> thread doing the autocommit in the logs, nor any metric that gives me the
> time the autocommit is being done.
>
> Is there any way to know for sure the autocommit is working as expected in
> Solr 8?
>
> PS: Solr 6 had an entry in the logs that confirmed it was being executed,
> you just need to grep for "commitScheduler"
>
> Thanks,
> Alex
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>
Re: Solr 8 autocommit confirmation
Posted by Joel Bernstein <jo...@gmail.com>.
I was just looking at the commit details in the logs. This has changed in
8x somewhere along the way. Now there is very little INFO level logging for
commits. Currently there is only this:
2021-10-08 16:39:43.073 INFO (qtp1080476785-21) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard
AutoCommit: if uncommitted for 30000ms;
2021-10-08 16:39:43.073 INFO (qtp1080476785-21) [c:collection1 s:shard1
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft
AutoCommit: if uncommitted for 15000ms;
New searcher registration is also still in the logs and looks like this:
2019-12-16 19:00:23.931 INFO (searcherExecutor-66-thread-1) [ ]
o.a.s.c.SolrCore [production1_shard35_replica_n1] Registered new searcher
Searcher@16ef5fac[production_cv_month_201912_shard35_replica_n1] ...
Joel Bernstein
http://joelsolr.blogspot.com/
On Fri, Oct 8, 2021 at 5:56 AM Saur, Alexandre (ELS-AMS) <
a.saur@elsevier.com> wrote:
> Hi Alex,
>
> Many thanks for the suggestions: I was able to enable debug log level and
> could confirm there was thread that was being called to make the commits:
>
> 2021-10-07 16:47:23.956 DEBUG (commitScheduler-37-thread-1) [...]
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1712980223806406656,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> This runs roughly against the schedule you specify in the autocommit
> configuration if you have updates to commit.
>
> Cheers!
>
> ________________________________
> From: Alexandre Rafalovitch <ar...@gmail.com>
> Sent: 06 October 2021 19:30
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: Solr 8 autocommit confirmation
>
> *** External email: use caution ***
>
>
>
> If you can afford to, in Admin UI, you can dynamically change log level for
> specific subsystem or even whole Solr (or whole Apache).
>
> Maybe turn it to Trace for a couple of minutes in the right window of time
> and see what happens. Make sure you are looking at log files and not
> console output, they have different log levels.
>
> Also solrconfig.xml had a setting about echoing lucene information to log.
> Maybe that influences segment generation information, which would be one of
> the things you could look for.
>
> If desperate, you could always just get a fresh Solr install instance for
> the same version and run tutorial commands and look for output there. That
> would allow to compare expectations with reality and you could definitely
> enable Trace debug level there.
>
> Regards,
> Alex
>
> On Wed., Oct. 6, 2021, 12:23 p.m. Saur, Alexandre (ELS-AMS), <
> a.saur@elsevier.com> wrote:
>
> > Hi Shawn,
> >
> > Thanks for the considerations. I also cannot see anything in the logs,
> > which is unfortunate. I'm trying to see if the autocommits have a direct
> > relationship with the bad Solr 8 throughput I'm experiencing. I'll start
> > experimenting with the values!
> >
> > Regards,
> > Alex
> >
> >
> > ________________________________
> > From: Shawn Heisey <ap...@elyograg.org>
> > Sent: 06 October 2021 16:59
> > To: users@solr.apache.org <us...@solr.apache.org>
> > Subject: Re: Solr 8 autocommit confirmation
> >
> > *** External email: use caution ***
> >
> >
> >
> > On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> > > I have a quick question regarding Solr 8 autocommit feature. This is my
> > configuration:
> > >
> > > <autoCommit>
> > > <maxTime>600000</maxTime>
> > > <openSearcher>true</openSearcher>
> > > </autoCommit>
> > >
> > > <autoSoftCommit>
> > > <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> > > </autoSoftCommit>
> > >
> > > I can check this is true using the console. However, I cannot see any
> > thread doing the autocommit in the logs, nor any metric that gives me the
> > time the autocommit is being done.
> >
> > The client (dovecot) that I have using my personal Solr server issues
> > commits itself, so I do not have any automatic commits in my log, and at
> > this time I do not use Solr at work. I have no idea if Solr's logfile
> > has specific mention of automatic commits. I would hope that it does.
> > I have done a full reindex several times and I do not see anything that
> > suggests an automatic commit ever fired. I have 60000 for autoCommit
> > and 120000 for autoSoftCommit.
> >
> > If you do a manual commit after every indexing batch, and the batch
> > takes less than 10 minutes (your autoCommit time) then autoCommit will
> > not actually do anything, and what I see in my own log suggests that
> > autoCommit and autoSoftCommit are smart enough to not even run if a
> > commit is done manually before they fire.
> >
> > This is what I would do if I were in your shoes, and I might further
> > reduce the autoSoftCommit, depending on how long commits actually take:
> >
> > <autoCommit>
> > <maxTime>60000</maxTime>
> > <openSearcher>false</openSearcher>
> > </autoCommit>
> >
> > <autoSoftCommit>
> > <maxTime>300000</maxTime>
> > </autoSoftCommit>
> >
> > Assuming you don't do manual commits in your indexing software, this
> > autoCommit would ensure the transaction logs will have less data in them,
> > which is a good thing, and the autoSoftCommit handles change visibility.
> > Of course, if your indexing software does its own commits and does them
> > with enough frequency, then the automatic commits may be essentially
> > unused.
> >
> > This article is valuable reading. The title says "SolrCloud" but it is
> > also applicable when NOT in cloud mode:
> >
> >
> >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2Fpost%2Funderstanding-transaction-logs-softcommit-and-commit-in-sorlcloud%2F&data=04%7C01%7Ca.saur%40elsevier.com%7C2a3f5750a51d4c13b21b08d988ef1c27%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637691383569136762%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MjFQ8c8j%2Bh7iIvCKpbCkVaAwKWUuEioQega%2B4DZDb5M%3D&reserved=0
> >
> > Thanks,
> > Shawn
> >
> >
> > ________________________________
> >
> > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> > Netherlands, Registration No. 33158992, Registered in The Netherlands.
> >
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>
Re: Solr 8 autocommit confirmation
Posted by "Saur, Alexandre (ELS-AMS)" <a....@elsevier.com>.
Hi Alex,
Many thanks for the suggestions: I was able to enable debug log level and could confirm there was thread that was being called to make the commits:
2021-10-07 16:47:23.956 DEBUG (commitScheduler-37-thread-1) [...] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1712980223806406656,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
This runs roughly against the schedule you specify in the autocommit configuration if you have updates to commit.
Cheers!
________________________________
From: Alexandre Rafalovitch <ar...@gmail.com>
Sent: 06 October 2021 19:30
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solr 8 autocommit confirmation
*** External email: use caution ***
If you can afford to, in Admin UI, you can dynamically change log level for
specific subsystem or even whole Solr (or whole Apache).
Maybe turn it to Trace for a couple of minutes in the right window of time
and see what happens. Make sure you are looking at log files and not
console output, they have different log levels.
Also solrconfig.xml had a setting about echoing lucene information to log.
Maybe that influences segment generation information, which would be one of
the things you could look for.
If desperate, you could always just get a fresh Solr install instance for
the same version and run tutorial commands and look for output there. That
would allow to compare expectations with reality and you could definitely
enable Trace debug level there.
Regards,
Alex
On Wed., Oct. 6, 2021, 12:23 p.m. Saur, Alexandre (ELS-AMS), <
a.saur@elsevier.com> wrote:
> Hi Shawn,
>
> Thanks for the considerations. I also cannot see anything in the logs,
> which is unfortunate. I'm trying to see if the autocommits have a direct
> relationship with the bad Solr 8 throughput I'm experiencing. I'll start
> experimenting with the values!
>
> Regards,
> Alex
>
>
> ________________________________
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: 06 October 2021 16:59
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: Solr 8 autocommit confirmation
>
> *** External email: use caution ***
>
>
>
> On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> > I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
> >
> > <autoCommit>
> > <maxTime>600000</maxTime>
> > <openSearcher>true</openSearcher>
> > </autoCommit>
> >
> > <autoSoftCommit>
> > <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> > </autoSoftCommit>
> >
> > I can check this is true using the console. However, I cannot see any
> thread doing the autocommit in the logs, nor any metric that gives me the
> time the autocommit is being done.
>
> The client (dovecot) that I have using my personal Solr server issues
> commits itself, so I do not have any automatic commits in my log, and at
> this time I do not use Solr at work. I have no idea if Solr's logfile
> has specific mention of automatic commits. I would hope that it does.
> I have done a full reindex several times and I do not see anything that
> suggests an automatic commit ever fired. I have 60000 for autoCommit
> and 120000 for autoSoftCommit.
>
> If you do a manual commit after every indexing batch, and the batch
> takes less than 10 minutes (your autoCommit time) then autoCommit will
> not actually do anything, and what I see in my own log suggests that
> autoCommit and autoSoftCommit are smart enough to not even run if a
> commit is done manually before they fire.
>
> This is what I would do if I were in your shoes, and I might further
> reduce the autoSoftCommit, depending on how long commits actually take:
>
> <autoCommit>
> <maxTime>60000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>300000</maxTime>
> </autoSoftCommit>
>
> Assuming you don't do manual commits in your indexing software, this
> autoCommit would ensure the transaction logs will have less data in them,
> which is a good thing, and the autoSoftCommit handles change visibility.
> Of course, if your indexing software does its own commits and does them
> with enough frequency, then the automatic commits may be essentially
> unused.
>
> This article is valuable reading. The title says "SolrCloud" but it is
> also applicable when NOT in cloud mode:
>
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2Fpost%2Funderstanding-transaction-logs-softcommit-and-commit-in-sorlcloud%2F&data=04%7C01%7Ca.saur%40elsevier.com%7C2a3f5750a51d4c13b21b08d988ef1c27%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637691383569136762%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MjFQ8c8j%2Bh7iIvCKpbCkVaAwKWUuEioQega%2B4DZDb5M%3D&reserved=0
>
> Thanks,
> Shawn
>
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33158992, Registered in The Netherlands.
Re: Solr 8 autocommit confirmation
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
If you can afford to, in Admin UI, you can dynamically change log level for
specific subsystem or even whole Solr (or whole Apache).
Maybe turn it to Trace for a couple of minutes in the right window of time
and see what happens. Make sure you are looking at log files and not
console output, they have different log levels.
Also solrconfig.xml had a setting about echoing lucene information to log.
Maybe that influences segment generation information, which would be one of
the things you could look for.
If desperate, you could always just get a fresh Solr install instance for
the same version and run tutorial commands and look for output there. That
would allow to compare expectations with reality and you could definitely
enable Trace debug level there.
Regards,
Alex
On Wed., Oct. 6, 2021, 12:23 p.m. Saur, Alexandre (ELS-AMS), <
a.saur@elsevier.com> wrote:
> Hi Shawn,
>
> Thanks for the considerations. I also cannot see anything in the logs,
> which is unfortunate. I'm trying to see if the autocommits have a direct
> relationship with the bad Solr 8 throughput I'm experiencing. I'll start
> experimenting with the values!
>
> Regards,
> Alex
>
>
> ________________________________
> From: Shawn Heisey <ap...@elyograg.org>
> Sent: 06 October 2021 16:59
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: Solr 8 autocommit confirmation
>
> *** External email: use caution ***
>
>
>
> On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> > I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
> >
> > <autoCommit>
> > <maxTime>600000</maxTime>
> > <openSearcher>true</openSearcher>
> > </autoCommit>
> >
> > <autoSoftCommit>
> > <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> > </autoSoftCommit>
> >
> > I can check this is true using the console. However, I cannot see any
> thread doing the autocommit in the logs, nor any metric that gives me the
> time the autocommit is being done.
>
> The client (dovecot) that I have using my personal Solr server issues
> commits itself, so I do not have any automatic commits in my log, and at
> this time I do not use Solr at work. I have no idea if Solr's logfile
> has specific mention of automatic commits. I would hope that it does.
> I have done a full reindex several times and I do not see anything that
> suggests an automatic commit ever fired. I have 60000 for autoCommit
> and 120000 for autoSoftCommit.
>
> If you do a manual commit after every indexing batch, and the batch
> takes less than 10 minutes (your autoCommit time) then autoCommit will
> not actually do anything, and what I see in my own log suggests that
> autoCommit and autoSoftCommit are smart enough to not even run if a
> commit is done manually before they fire.
>
> This is what I would do if I were in your shoes, and I might further
> reduce the autoSoftCommit, depending on how long commits actually take:
>
> <autoCommit>
> <maxTime>60000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>300000</maxTime>
> </autoSoftCommit>
>
> Assuming you don't do manual commits in your indexing software, this
> autoCommit would ensure the transaction logs will have less data in them,
> which is a good thing, and the autoSoftCommit handles change visibility.
> Of course, if your indexing software does its own commits and does them
> with enough frequency, then the automatic commits may be essentially
> unused.
>
> This article is valuable reading. The title says "SolrCloud" but it is
> also applicable when NOT in cloud mode:
>
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2Fpost%2Funderstanding-transaction-logs-softcommit-and-commit-in-sorlcloud%2F&data=04%7C01%7Ca.saur%40elsevier.com%7Cbb90d89a7de54274fec208d988d9ffac%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637691293307772302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xVcTSmftEFx3yRb3q%2Bi8GtSUppT0jZx35g4y3cDXLWc%3D&reserved=0
>
> Thanks,
> Shawn
>
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>
Re: Solr 8 autocommit confirmation
Posted by "Saur, Alexandre (ELS-AMS)" <a....@elsevier.com>.
Hi Shawn,
Thanks for the considerations. I also cannot see anything in the logs, which is unfortunate. I'm trying to see if the autocommits have a direct relationship with the bad Solr 8 throughput I'm experiencing. I'll start experimenting with the values!
Regards,
Alex
________________________________
From: Shawn Heisey <ap...@elyograg.org>
Sent: 06 October 2021 16:59
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solr 8 autocommit confirmation
*** External email: use caution ***
On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> I have a quick question regarding Solr 8 autocommit feature. This is my configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any thread doing the autocommit in the logs, nor any metric that gives me the time the autocommit is being done.
The client (dovecot) that I have using my personal Solr server issues
commits itself, so I do not have any automatic commits in my log, and at
this time I do not use Solr at work. I have no idea if Solr's logfile
has specific mention of automatic commits. I would hope that it does.
I have done a full reindex several times and I do not see anything that
suggests an automatic commit ever fired. I have 60000 for autoCommit
and 120000 for autoSoftCommit.
If you do a manual commit after every indexing batch, and the batch
takes less than 10 minutes (your autoCommit time) then autoCommit will
not actually do anything, and what I see in my own log suggests that
autoCommit and autoSoftCommit are smart enough to not even run if a
commit is done manually before they fire.
This is what I would do if I were in your shoes, and I might further
reduce the autoSoftCommit, depending on how long commits actually take:
<autoCommit>
<maxTime>60000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>300000</maxTime>
</autoSoftCommit>
Assuming you don't do manual commits in your indexing software, this autoCommit would ensure the transaction logs will have less data in them, which is a good thing, and the autoSoftCommit handles change visibility. Of course, if your indexing software does its own commits and does them with enough frequency, then the automatic commits may be essentially unused.
This article is valuable reading. The title says "SolrCloud" but it is also applicable when NOT in cloud mode:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2Fpost%2Funderstanding-transaction-logs-softcommit-and-commit-in-sorlcloud%2F&data=04%7C01%7Ca.saur%40elsevier.com%7Cbb90d89a7de54274fec208d988d9ffac%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637691293307772302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xVcTSmftEFx3yRb3q%2Bi8GtSUppT0jZx35g4y3cDXLWc%3D&reserved=0
Thanks,
Shawn
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33158992, Registered in The Netherlands.
Re: Solr 8 autocommit confirmation
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> I have a quick question regarding Solr 8 autocommit feature. This is my configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any thread doing the autocommit in the logs, nor any metric that gives me the time the autocommit is being done.
The client (dovecot) that I have using my personal Solr server issues
commits itself, so I do not have any automatic commits in my log, and at
this time I do not use Solr at work. I have no idea if Solr's logfile
has specific mention of automatic commits. I would hope that it does.
I have done a full reindex several times and I do not see anything that
suggests an automatic commit ever fired. I have 60000 for autoCommit
and 120000 for autoSoftCommit.
If you do a manual commit after every indexing batch, and the batch
takes less than 10 minutes (your autoCommit time) then autoCommit will
not actually do anything, and what I see in my own log suggests that
autoCommit and autoSoftCommit are smart enough to not even run if a
commit is done manually before they fire.
This is what I would do if I were in your shoes, and I might further
reduce the autoSoftCommit, depending on how long commits actually take:
<autoCommit>
<maxTime>60000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>300000</maxTime>
</autoSoftCommit>
Assuming you don't do manual commits in your indexing software, this autoCommit would ensure the transaction logs will have less data in them, which is a good thing, and the autoSoftCommit handles change visibility. Of course, if your indexing software does its own commits and does them with enough frequency, then the automatic commits may be essentially unused.
This article is valuable reading. The title says "SolrCloud" but it is also applicable when NOT in cloud mode:
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
Re: Solr 8 autocommit confirmation
Posted by "Saur, Alexandre (ELS-AMS)" <a....@elsevier.com>.
Thanks for the tip, Sudhir!
I can now see that autocommits are happening, but there is no information on when it occurred (what time). Is there a way to know it?
________________________________
From: sudhir kumar <go...@gmail.com>
Sent: 06 October 2021 15:42
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solr 8 autocommit confirmation
*** External email: use caution ***
Hey Alex,
you can use api -
http://solrurl:8983/solr/admin/metrics?group=core&prefix=UPDATE.updateHandler
Find the collection shard and check for *autoCommits. *you should see the
count/number of autocommits triggered.
You can also check how much time(600000) it takes to increment the
autocommit count.
Thanks,
Sudhir.
On Wed, Oct 6, 2021 at 6:30 PM Saur, Alexandre (ELS-AMS) <
a.saur@elsevier.com> wrote:
> Hello there,
>
> I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any
> thread doing the autocommit in the logs, nor any metric that gives me the
> time the autocommit is being done.
>
> Is there any way to know for sure the autocommit is working as expected in
> Solr 8?
>
> PS: Solr 6 had an entry in the logs that confirmed it was being executed,
> you just need to grep for "commitScheduler"
>
> Thanks,
> Alex
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33158992, Registered in The Netherlands.
Re: Solr 8 autocommit confirmation
Posted by sudhir kumar <go...@gmail.com>.
Hey Alex,
you can use api -
http://solrurl:8983/solr/admin/metrics?group=core&prefix=UPDATE.updateHandler
Find the collection shard and check for *autoCommits. *you should see the
count/number of autocommits triggered.
You can also check how much time(600000) it takes to increment the
autocommit count.
Thanks,
Sudhir.
On Wed, Oct 6, 2021 at 6:30 PM Saur, Alexandre (ELS-AMS) <
a.saur@elsevier.com> wrote:
> Hello there,
>
> I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any
> thread doing the autocommit in the logs, nor any metric that gives me the
> time the autocommit is being done.
>
> Is there any way to know for sure the autocommit is working as expected in
> Solr 8?
>
> PS: Solr 6 had an entry in the logs that confirmed it was being executed,
> you just need to grep for "commitScheduler"
>
> Thanks,
> Alex
>
> ________________________________
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33158992, Registered in The Netherlands.
>