You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Satyaprashant Bezwada <Sa...@nasdaq.com> on 2018/01/05 19:36:52 UTC
Solr cloud issue : Solr 6.5
Not sure where we are going wrong in our implementation.
We have a Solr cloud environment (Solr 6.5), with 2 solr nodes and 3 Zookeeper servers. The environment was running without any issues, but lately noticed that one of the Solr node keeps shutting down frequently. We have replication in place where in we stated one per node.
This is what I see in the logs and it happens only on one particular Solr node, which keeps shutting down frequently.
2018-01-04 07:34:59.168 INFO (qtp127618319-21) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.c.PluginBag Going to create a new requestHandler with {type = requestHandler,name = /update/extract,class = solr.extraction.ExtractingRequestHandler,attributes = {startup=lazy, name=/update/extract, class=solr.extraction.ExtractingRequestHandler},args = {defaults={uprefix=ignored_,captureAttr=true,df=_text_}}}
2018-01-04 07:35:00.490 ERROR (updateExecutor-2-thread-9-processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2 x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.StreamingSolrClients error
java.net.ConnectException: Connection refused
2018-01-04 07:35:00.493 ERROR (qtp127618319-21) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed - retrying ... retries: 1 add{,id=0DODD:S8IYAB-13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/ rsp:-1:java.net.ConnectException: Connection refused
2018-01-04 07:35:00.995 ERROR (updateExecutor-2-thread-9-processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2 x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.StreamingSolrClients error
java.net.ConnectException: Connection refused
2018-01-04 07:35:00.996 ERROR (qtp127618319-21) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed - retrying ... retries: 2 add{,id=0DODD:S8IYAB-13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/ rsp:-1:java.net.ConnectException: Connection refused
2018-01-04 07:35:01.497 ERROR (updateExecutor-2-thread-9-processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2 x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.StreamingSolrClients error
java.net.ConnectException: Connection refused
………
2018-01-04 07:35:12.544 ERROR (qtp127618319-21) [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed - retrying ... retries: 25 add{,id=0DODD:S8IYAB-13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/ rsp:-1:java.net.ConnectException: Connection refused
……..
2018-01-05 03:54:15.506 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer (id=98943164658291709-sr01:8983_solr-n_0000000071) closing
2018-01-05 03:54:15.507 INFO (OverseerStateUpdate-98943164658291709-sr01:8983_solr-n_0000000071) [ ] o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2018-01-05 03:54:17.013 WARN (zkCallback-4-thread-20-processing-n:sr01:8983_solr) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
2018-01-05 03:54:17.013 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
***********************************************************
Re: Solr cloud issue : Solr 6.5
Posted by Satyaprashant Bezwada <Sa...@nasdaq.com>.
Thanks a lot Shawn. That really helps, I believe the changes were made during our load testing phase to introduce some changes in the startup script to manage the max allocated memory.
That’s where someone introduced that line. We removed it and I’ll check how it behaves now.
Regards
Prashant
On 1/5/18, 2:09 PM, "Shawn Heisey" <ap...@elyograg.org> wrote:
WARNING - External email; exercise caution.
On 1/5/2018 2:35 PM, Satyaprashant Bezwada wrote:
> Thanks a lot. It helped, I noticed the error in the solr console log.
<snip>
> # java.lang.OutOfMemoryError: Metaspace
> # -XX:OnOutOfMemoryError="/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"
> # Executing /bin/sh -c "/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"...
> Running OOM killer script for process 24431 for Solr on port 8983
> Killed process 24431
>
> Wasn’t sure if it would help, but I increased the max memory allocated for Metaspace size from 64 to 128. Would continue to monitor and see if it works.
> Is there any ideal memory space that is suggested?
>
> SOLR_OPTS="$SOLR_OPTS -XX:MaxMetaspaceSize=128m"
The information I've found says Java 8 doesn't set a limit on metaspace
unless you specify that startup option. It can use as much native
memory as the machine has. Since Solr 6.x requires Java 8, I don't
think you're running an older Java version.
I do not see the -XX:MaxMetaSpaceSize option in the standard scripts
when I look in the master branch, or in the 6.5.1 version you are
running, so unless your startup script was edited to include that
option, metaspace would not be limited to 64 megabytes. That particular
OOME exception should not be possible with the standard script unless
the amount of memory in the machine is limited. If you *did* find that
option in your script set to 64m, then somebody at your location added
it. Unless you have a really good reason for having the option, I would
recommend removing it, so there is no artificial limit.
This is the first time I've seen a metaspace OOME related to Solr, so
although it is *possible* that there's a metaspace/permgen memory leak,
it is unlikely. If there were a bug like that in Solr, I would expect
to see that error from more than one person.
Thanks,
Shawn
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
***********************************************************
Re: Solr cloud issue : Solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/5/2018 2:35 PM, Satyaprashant Bezwada wrote:
> Thanks a lot. It helped, I noticed the error in the solr console log.
<snip>
> # java.lang.OutOfMemoryError: Metaspace
> # -XX:OnOutOfMemoryError="/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"
> # Executing /bin/sh -c "/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"...
> Running OOM killer script for process 24431 for Solr on port 8983
> Killed process 24431
>
> Wasn’t sure if it would help, but I increased the max memory allocated for Metaspace size from 64 to 128. Would continue to monitor and see if it works.
> Is there any ideal memory space that is suggested?
>
> SOLR_OPTS="$SOLR_OPTS -XX:MaxMetaspaceSize=128m"
The information I've found says Java 8 doesn't set a limit on metaspace
unless you specify that startup option. It can use as much native
memory as the machine has. Since Solr 6.x requires Java 8, I don't
think you're running an older Java version.
I do not see the -XX:MaxMetaSpaceSize option in the standard scripts
when I look in the master branch, or in the 6.5.1 version you are
running, so unless your startup script was edited to include that
option, metaspace would not be limited to 64 megabytes. That particular
OOME exception should not be possible with the standard script unless
the amount of memory in the machine is limited. If you *did* find that
option in your script set to 64m, then somebody at your location added
it. Unless you have a really good reason for having the option, I would
recommend removing it, so there is no artificial limit.
This is the first time I've seen a metaspace OOME related to Solr, so
although it is *possible* that there's a metaspace/permgen memory leak,
it is unlikely. If there were a bug like that in Solr, I would expect
to see that error from more than one person.
Thanks,
Shawn
Re: Solr cloud issue : Solr 6.5
Posted by Satyaprashant Bezwada <Sa...@nasdaq.com>.
Thanks a lot. It helped, I noticed the error in the solr console log.
cat solr-8983-console.log
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://xxx.xx.x.xxx:8778/jolokia/
2018-01-05 03:56:30.824 INFO (main) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
2018-01-05 03:56:31.209 INFO (main) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 6.5.1
2018-01-05 03:56:31.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port 8983
2018-01-05 03:56:31.231 INFO (main) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: /usr/local/solr
2018-01-05 03:56:31.248 INFO (main) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-05T03:56:31.232Z
2018-01-05 03:56:31.249 INFO (main) [ ] o.a.s.s.StartupLoggingUtils Property solr.log.muteconsole given. Muting ConsoleAppender named CONSOLE
#
# java.lang.OutOfMemoryError: Metaspace
# -XX:OnOutOfMemoryError="/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"
# Executing /bin/sh -c "/usr/local/solr/bin/oom_solr.sh 8983 /var/solr/logs"...
Running OOM killer script for process 24431 for Solr on port 8983
Killed process 24431
Wasn’t sure if it would help, but I increased the max memory allocated for Metaspace size from 64 to 128. Would continue to monitor and see if it works.
Is there any ideal memory space that is suggested?
SOLR_OPTS="$SOLR_OPTS -XX:MaxMetaspaceSize=128m"
Thanks
Prashant
On 1/5/18, 11:56 AM, "Erick Erickson" <er...@gmail.com> wrote:
WARNING - External email; exercise caution.
If this has been working fine for a while and suddenly started this
behavior my first suspicion would be excessive GC, i.e. you've been adding
docs and your heap is no longer adequate. If Java needs to do a
stop-the-world garbage collection you can get these kinds of errors.
So I'd enable GC logging on my Solr instances (and ZooKeeper) and then use
something like GCViewer to figure out whether I have excessive
stop-the-world pauses first.
Best,
Erick
On Fri, Jan 5, 2018 at 11:36 AM, Satyaprashant Bezwada <
Satyaprashant.Bezwada@nasdaq.com> wrote:
> Not sure where we are going wrong in our implementation.
>
> We have a Solr cloud environment (Solr 6.5), with 2 solr nodes and 3
> Zookeeper servers. The environment was running without any issues, but
> lately noticed that one of the Solr node keeps shutting down frequently. We
> have replication in place where in we stated one per node.
>
> This is what I see in the logs and it happens only on one particular Solr
> node, which keeps shutting down frequently.
>
> 2018-01-04 07:34:59.168 INFO (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.c.PluginBag Going to create a
> new requestHandler with {type = requestHandler,name = /update/extract,class
> = solr.extraction.ExtractingRequestHandler,attributes = {startup=lazy,
> name=/update/extract, class=solr.extraction.ExtractingRequestHandler},args
> = {defaults={uprefix=ignored_,captureAttr=true,df=_text_}}}
> 2018-01-04 07:35:00.490 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
>
> 2018-01-04 07:35:00.493 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 1 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.995 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.996 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 2 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:01.497 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
> ………
>
>
> 2018-01-04 07:35:12.544 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 25 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
> ……..
>
> 2018-01-05 03:54:15.506 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
> Overseer (id=98943164658291709-sr01:8983_solr-n_0000000071) closing
> 2018-01-05 03:54:15.507 INFO (OverseerStateUpdate-
> 98943164658291709-sr01:8983_solr-n_0000000071) [ ] o.a.s.c.Overseer
> Overseer Loop exiting : sr01:8983_solr
> 2018-01-05 03:54:17.013 WARN (zkCallback-4-thread-20-processing-n:sr01:8983_solr)
> [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot
> talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
> 2018-01-05 03:54:17.013 INFO (ShutdownMonitor) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
>
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the
> exclusive and confidential use of the intended recipient and may constitute
> non-public information. If you received this e-mail in error, disclosing,
> copying, distributing or taking any action in reliance of this e-mail is
> strictly prohibited and may be unlawful. Instead, please notify us
> immediately by return e-mail and promptly delete this message and its
> attachments from your computer system. We do not waive any work product or
> other applicable legal privilege(s) by the transmission of this message.
> ***********************************************************
>
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message.
***********************************************************
Re: Solr cloud issue : Solr 6.5
Posted by Erick Erickson <er...@gmail.com>.
If this has been working fine for a while and suddenly started this
behavior my first suspicion would be excessive GC, i.e. you've been adding
docs and your heap is no longer adequate. If Java needs to do a
stop-the-world garbage collection you can get these kinds of errors.
So I'd enable GC logging on my Solr instances (and ZooKeeper) and then use
something like GCViewer to figure out whether I have excessive
stop-the-world pauses first.
Best,
Erick
On Fri, Jan 5, 2018 at 11:36 AM, Satyaprashant Bezwada <
Satyaprashant.Bezwada@nasdaq.com> wrote:
> Not sure where we are going wrong in our implementation.
>
> We have a Solr cloud environment (Solr 6.5), with 2 solr nodes and 3
> Zookeeper servers. The environment was running without any issues, but
> lately noticed that one of the Solr node keeps shutting down frequently. We
> have replication in place where in we stated one per node.
>
> This is what I see in the logs and it happens only on one particular Solr
> node, which keeps shutting down frequently.
>
> 2018-01-04 07:34:59.168 INFO (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.c.PluginBag Going to create a
> new requestHandler with {type = requestHandler,name = /update/extract,class
> = solr.extraction.ExtractingRequestHandler,attributes = {startup=lazy,
> name=/update/extract, class=solr.extraction.ExtractingRequestHandler},args
> = {defaults={uprefix=ignored_,captureAttr=true,df=_text_}}}
> 2018-01-04 07:35:00.490 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
>
> 2018-01-04 07:35:00.493 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 1 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.995 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.996 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 2 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:01.497 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
> ………
>
>
> 2018-01-04 07:35:12.544 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 25 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
> ……..
>
> 2018-01-05 03:54:15.506 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
> Overseer (id=98943164658291709-sr01:8983_solr-n_0000000071) closing
> 2018-01-05 03:54:15.507 INFO (OverseerStateUpdate-
> 98943164658291709-sr01:8983_solr-n_0000000071) [ ] o.a.s.c.Overseer
> Overseer Loop exiting : sr01:8983_solr
> 2018-01-05 03:54:17.013 WARN (zkCallback-4-thread-20-processing-n:sr01:8983_solr)
> [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot
> talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
> 2018-01-05 03:54:17.013 INFO (ShutdownMonitor) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
>
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the
> exclusive and confidential use of the intended recipient and may constitute
> non-public information. If you received this e-mail in error, disclosing,
> copying, distributing or taking any action in reliance of this e-mail is
> strictly prohibited and may be unlawful. Instead, please notify us
> immediately by return e-mail and promptly delete this message and its
> attachments from your computer system. We do not waive any work product or
> other applicable legal privilege(s) by the transmission of this message.
> ***********************************************************
>