You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by HariBabu kuruva <ha...@gmail.com> on 2021/09/07 10:00:29 UTC
Solr | OOM | Lazyfield
Hi All,
We are getting OOM errors in the solr logs for only specific solr stores.
And in the solr logs we see the below error. Is the OOM error could be
because of the below error.
Also i see below Lazyfield error is spanned across thousands of lines.
Please advise. This is a PROD environment.
---------------------------------
AsyncLogger error handling event seq=163,
value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
Message=org.apache.solr.
common.SolrException: ERROR: [doc=1-121180294489] multiple values
encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
che.lucene.document.LazyDocument$LazyField@2b119ab6,
org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
org.apache.lucene.docu
ment.LazyDocument$LazyField@7dba464d,
org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
org.apache.lucene.document.LazyDocume
nt$LazyField@4b19be97,
org.apache.lucene.document.LazyDocument$LazyField@5978e924,
org.apache.lucene.document.LazyDocument$LazyField@76
--------------------------
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr | OOM | Lazyfield
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/7/2021 4:00 AM, HariBabu kuruva wrote:
> We are getting OOM errors in the solr logs for only specific solr stores.
>
> And in the solr logs we see the below error. Is the OOM error could be
> because of the below error.
There are precisely two ways to deal with OOME. One is to increase the
size of the resource that has been depleted. The other is to change
something so less of that resource is required. Very frequently it is
not possible to accomplish the second option. Increasing the resource
is very often the only solution.
Note that it might not actually be memory that is being depleted. Java
throws OOME for several different resource exhaustion scenarios. Some
examples of things that might run out before memory are max processes
per user or max open files.
You haven't shown us the OOME error, so we cannot advise you about what
you need to do. Assuming that it is actually memory that is depleted...
Out of the box, Solr's max heap defaults to 512MB. This is VERY small
and almost every user will need to increase it. We made the default
heap small so that Solr would start on just about any hardware without
changing the config.
It is very unlikely that the place in the code where the OOME occurred
will reveal anything useful. We just want to see it so we can see the
message logged at the beginning. Also, any other errors you are seeing
are likely unrelated to the OOME.
If you're running Solr on a non-windows system, the bin/solr script
starts Solr with a Java option that causes Solr to kill itself when OOME
occurs. It does this to protect itself -- Java program operation after
OOME is completely unpredictable and in the case of Solr/Lucene, could
corrupt the index. We haven't yet done this for Windows.
Thanks,
Shawn
Re: Solr | OOM | Lazyfield
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/8/2021 8:05 AM, HariBabu kuruva wrote:
> I have checked with the DEV team regarding multivalue error. They say that :
> the fields which require/have multiple values were already set to
> multivalued=true . For non-multi valued fields also we are getting multiple
> values. That is the problem.
Are you using the atomic update feature? There are very specific schema
requirements for Atomic Update. If you're not using Atomic Update, then
you can ignore this whole message. From the ref guide:
-------
The core functionality of atomically updating a document requires that
all fields in your schema must be configured as stored (stored="true")
or docValues (docValues="true") except for fields which are <copyField/>
destinations, which must be configured as stored="false". Atomic updates
are applied to the document represented by the existing stored field
values. All data in copyField destinations fields must originate from
ONLY copyField sources.
If <copyField/> destinations are configured as stored, then Solr will
attempt to index both the current value of the field as well as an
additional copy from any source fields. If such fields contain some
information that comes from the indexing program and some information
that comes from copyField, then the information which originally came
from the indexing program will be lost when an atomic update is made.
There are other kinds of derived fields that must also be set so they
aren't stored. Some spatial field types, such as BBoxField and
LatLonType, use derived fields. CurrencyFieldType also uses derived
fields. These types create additional fields which are normally
specified by a dynamic field definition. That dynamic field definition
must be not stored, or indexing will fail.
-------
That requires a little bit of an update -- definitions that must be
unstored must also have docValues set to false.
If copyField is in use and destinations are stored, that could lead to
multiple values when using atomic update.
Thanks,
Shawn
Re: Solr | OOM | Lazyfield
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi Charlie,
I have checked with the DEV team regarding multivalue error. They say that :
the fields which require/have multiple values were already set to
multivalued=true . For non-multi valued fields also we are getting multiple
values. That is the problem.
On Tue, Sep 7, 2021 at 9:05 PM Charlie Hull <ch...@opensourceconnections.com>
wrote:
> I don't know, but perhaps 8.8.1 checks it more than 8.1? - it's still
> worth your dev team reviewing as I suggested as it may be linked to why
> the nodes are going down. The error is occurring for a reason.
>
> Charlie
>
> On 07/09/2021 14:30, HariBabu kuruva wrote:
> > Hi Charlie,
> >
> > The multiple values error is not occurring only for that field. Its also
> > occurring for multiple fields and multiple stores(collections).
> > Ex:
> > 2021-09-07 13:05:53.646 ERROR (qtp1198197478-1200) [c:quoteStore s:shard1
> > r:core_node8 x:quoteStore_shard1_replica_n7] o.a.s.h.RequestH
> > andlerBase org.apache.solr.common.SolrException: ERROR:
> > [doc=1-116339633161] multiple values encountered for non multiValued
> field
> > quot
> > eAgeCompleted: [N, N]
> >
> > Our Dev Team says that its only after the Solr upgrade from 8.1 to 8.8.1
> >
> > Also, Solr is going down very frequently (after every one hour) on the
> > nodes only where a specific store (quotestore) is available. Other nodes
> > are fine.
> > For the nodes which are going down, I see OOM error for one node and for
> > the other node we see zookeeper connection timeout error as below.
> >
> > --------------------------------
> > 2021-09-07 13:06:13.907 WARN (main-SendThread(
> > zookeeperhost.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Client
> session
> > timed out, ha
> > ve not heard from server in 23226ms for session id 0x40000139b3e00d3
> > 2021-09-07 13:06:13.907 WARN (main-SendThread(
> > lxeisprdas10.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Session
> > 0x40000139b3e00d3 fo
> > r sever zookeeperhost.corp.equinix.com/10.250.12.54:2185, Closing socket
> > connection. Attempting reconnect except it is a SessionExpiredE
> > xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> > Client session timed out, have not heard from server in 23226ms f
> > or session id 0x40000139b3e00d3
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> > org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
> > timed out, have not heard from server in 23226ms for session id
> > 0x40000139b3e00d3
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> > ~[zookeeper-3.6.2.jar:3.6.2]
> > 2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
> > ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.commo
> > n.cloud.ConnectionManager@5521aab name: ZooKeeperConnection Watcher:
> > zk1.corp.equinix.com:2181,zk2.corp.equinix.com:21
> > 82,zk3.corp.equinix.com:2183,zk4.corp.equinix.com:2184,
> > zk5.corp.equinix.com:2185 got event WatchedEvent stat
> > e:Disconnected type:None path:null path: null type: None
> > 2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
> > ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> > 2021-09-07 13:06:39.494 WARN (main-SendThread(
> > lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Client
> session
> > timed out, ha
> > ve not heard from server in 23533ms for session id 0x40000139b3e00d3
> > 2021-09-07 13:06:39.494 WARN (main-SendThread(
> > lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Session
> > 0x40000139b3e00d3 fo
> > r sever zk4.corp.equinix.com/10.250.**.**:2184, Closing socket
> connection.
> > Attempting reconnect except it is a SessionExpiredE
> > xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> > Client session timed out, have not heard from server in 23533ms f
> > or session id 0x40000139b3e00d3
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> > org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
> > timed out, have not heard from server in 23533ms for session id
> > 0x40000139b3e00d3
> > -------------------------------------------------------
> >
> > Could you please suggest.
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Sep 7, 2021 at 5:49 PM Charlie Hull <
> chull@opensourceconnections.com>
> > wrote:
> >
> >> Sorry I should have worded that better - I suggest that you review your
> >> schema file and indexing process, so that you can understand why
> >> multiple values are being sent to the field 'quoteSLOCurrentStatus' when
> >> it's probably not defined as multivalued.
> >>
> >> C
> >>
> >> On 07/09/2021 12:30, HariBabu kuruva wrote:
> >>> Thank you Charlie.
> >>>
> >>> Please let me know if anything is required from my end.
> >>>
> >>> Also, we have added below filed as a work around
> >>> <enableLazyFieldLoading>false</enableLazyFieldLoading> in
> solrconfig.xml
> >>>
> >>> On Tue, Sep 7, 2021 at 4:03 PM Charlie Hull <
> >> chull@opensourceconnections.com>
> >>> wrote:
> >>>
> >>>> I doubt these are related, that second error looks like something
> >>>> triggered by indexing - attempting to add multiple values to a field
> >>>> that hasn't been defined in the schema as multivalued. I'd review your
> >>>> indexing process and schema first.
> >>>>
> >>>> Charlie
> >>>>
> >>>> On 07/09/2021 11:00, HariBabu kuruva wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> We are getting OOM errors in the solr logs for only specific solr
> >> stores.
> >>>>> And in the solr logs we see the below error. Is the OOM error could
> be
> >>>>> because of the below error.
> >>>>>
> >>>>> Also i see below Lazyfield error is spanned across thousands of
> lines.
> >>>>>
> >>>>> Please advise. This is a PROD environment.
> >>>>>
> >>>>> ---------------------------------
> >>>>> AsyncLogger error handling event seq=163,
> >>>>> value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
> >>>>> Message=org.apache.solr.
> >>>>> common.SolrException: ERROR: [doc=1-121180294489] multiple values
> >>>>> encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
> >>>>> che.lucene.document.LazyDocument$LazyField@2b119ab6,
> >>>>> org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
> >>>>> org.apache.lucene.docu
> >>>>> ment.LazyDocument$LazyField@7dba464d,
> >>>>> org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
> >>>>> org.apache.lucene.document.LazyDocume
> >>>>> nt$LazyField@4b19be97,
> >>>>> org.apache.lucene.document.LazyDocument$LazyField@5978e924,
> >>>>> org.apache.lucene.document.LazyDocument$LazyField@76
> >>>>> --------------------------
> >>>>>
> >>>>>
> >>>>> Thanks and Regards,
> >>>>> Hari
> >>>>> Mobile:9790756568
> >>>>>
> >>>> --
> >>>> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> >>>> <www.o19s.com>
> >>>> Founding member of The Search Network <https://thesearchnetwork.com/>
> >>>> and co-author of Searching the Enterprise
> >>>> <https://opensourceconnections.com/about-us/books-resources/>
> >>>> tel/fax: +44 (0)8700 118334
> >>>> mobile: +44 (0)7767 825828
> >>>>
> >>>> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> >>>> Amtsgericht Charlottenburg | HRB 230712 B
> >>>> Geschäftsführer: John M. Woodell | David E. Pugh
> >>>> Finanzamt: Berlin Finanzamt für Körperschaften II
> >>>>
> >> --
> >> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> >> <www.o19s.com>
> >> Founding member of The Search Network <https://thesearchnetwork.com/>
> >> and co-author of Searching the Enterprise
> >> <https://opensourceconnections.com/about-us/books-resources/>
> >> tel/fax: +44 (0)8700 118334
> >> mobile: +44 (0)7767 825828
> >>
> >> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> >> Amtsgericht Charlottenburg | HRB 230712 B
> >> Geschäftsführer: John M. Woodell | David E. Pugh
> >> Finanzamt: Berlin Finanzamt für Körperschaften II
> >>
> >
>
> --
> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> <www.o19s.com>
> Founding member of The Search Network <https://thesearchnetwork.com/>
> and co-author of Searching the Enterprise
> <https://opensourceconnections.com/about-us/books-resources/>
> tel/fax: +44 (0)8700 118334
> mobile: +44 (0)7767 825828
>
> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> Amtsgericht Charlottenburg | HRB 230712 B
> Geschäftsführer: John M. Woodell | David E. Pugh
> Finanzamt: Berlin Finanzamt für Körperschaften II
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr | OOM | Lazyfield
Posted by Charlie Hull <ch...@opensourceconnections.com>.
I don't know, but perhaps 8.8.1 checks it more than 8.1? - it's still
worth your dev team reviewing as I suggested as it may be linked to why
the nodes are going down. The error is occurring for a reason.
Charlie
On 07/09/2021 14:30, HariBabu kuruva wrote:
> Hi Charlie,
>
> The multiple values error is not occurring only for that field. Its also
> occurring for multiple fields and multiple stores(collections).
> Ex:
> 2021-09-07 13:05:53.646 ERROR (qtp1198197478-1200) [c:quoteStore s:shard1
> r:core_node8 x:quoteStore_shard1_replica_n7] o.a.s.h.RequestH
> andlerBase org.apache.solr.common.SolrException: ERROR:
> [doc=1-116339633161] multiple values encountered for non multiValued field
> quot
> eAgeCompleted: [N, N]
>
> Our Dev Team says that its only after the Solr upgrade from 8.1 to 8.8.1
>
> Also, Solr is going down very frequently (after every one hour) on the
> nodes only where a specific store (quotestore) is available. Other nodes
> are fine.
> For the nodes which are going down, I see OOM error for one node and for
> the other node we see zookeeper connection timeout error as below.
>
> --------------------------------
> 2021-09-07 13:06:13.907 WARN (main-SendThread(
> zookeeperhost.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Client session
> timed out, ha
> ve not heard from server in 23226ms for session id 0x40000139b3e00d3
> 2021-09-07 13:06:13.907 WARN (main-SendThread(
> lxeisprdas10.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Session
> 0x40000139b3e00d3 fo
> r sever zookeeperhost.corp.equinix.com/10.250.12.54:2185, Closing socket
> connection. Attempting reconnect except it is a SessionExpiredE
> xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> Client session timed out, have not heard from server in 23226ms f
> or session id 0x40000139b3e00d3
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
> timed out, have not heard from server in 23226ms for session id
> 0x40000139b3e00d3
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> ~[zookeeper-3.6.2.jar:3.6.2]
> 2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
> ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.commo
> n.cloud.ConnectionManager@5521aab name: ZooKeeperConnection Watcher:
> zk1.corp.equinix.com:2181,zk2.corp.equinix.com:21
> 82,zk3.corp.equinix.com:2183,zk4.corp.equinix.com:2184,
> zk5.corp.equinix.com:2185 got event WatchedEvent stat
> e:Disconnected type:None path:null path: null type: None
> 2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
> ] o.a.s.c.c.ConnectionManager zkClient has disconnected
> 2021-09-07 13:06:39.494 WARN (main-SendThread(
> lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Client session
> timed out, ha
> ve not heard from server in 23533ms for session id 0x40000139b3e00d3
> 2021-09-07 13:06:39.494 WARN (main-SendThread(
> lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Session
> 0x40000139b3e00d3 fo
> r sever zk4.corp.equinix.com/10.250.**.**:2184, Closing socket connection.
> Attempting reconnect except it is a SessionExpiredE
> xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> Client session timed out, have not heard from server in 23533ms f
> or session id 0x40000139b3e00d3
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
> timed out, have not heard from server in 23533ms for session id
> 0x40000139b3e00d3
> -------------------------------------------------------
>
> Could you please suggest.
>
>
>
>
>
>
>
> On Tue, Sep 7, 2021 at 5:49 PM Charlie Hull <ch...@opensourceconnections.com>
> wrote:
>
>> Sorry I should have worded that better - I suggest that you review your
>> schema file and indexing process, so that you can understand why
>> multiple values are being sent to the field 'quoteSLOCurrentStatus' when
>> it's probably not defined as multivalued.
>>
>> C
>>
>> On 07/09/2021 12:30, HariBabu kuruva wrote:
>>> Thank you Charlie.
>>>
>>> Please let me know if anything is required from my end.
>>>
>>> Also, we have added below filed as a work around
>>> <enableLazyFieldLoading>false</enableLazyFieldLoading> in solrconfig.xml
>>>
>>> On Tue, Sep 7, 2021 at 4:03 PM Charlie Hull <
>> chull@opensourceconnections.com>
>>> wrote:
>>>
>>>> I doubt these are related, that second error looks like something
>>>> triggered by indexing - attempting to add multiple values to a field
>>>> that hasn't been defined in the schema as multivalued. I'd review your
>>>> indexing process and schema first.
>>>>
>>>> Charlie
>>>>
>>>> On 07/09/2021 11:00, HariBabu kuruva wrote:
>>>>> Hi All,
>>>>>
>>>>> We are getting OOM errors in the solr logs for only specific solr
>> stores.
>>>>> And in the solr logs we see the below error. Is the OOM error could be
>>>>> because of the below error.
>>>>>
>>>>> Also i see below Lazyfield error is spanned across thousands of lines.
>>>>>
>>>>> Please advise. This is a PROD environment.
>>>>>
>>>>> ---------------------------------
>>>>> AsyncLogger error handling event seq=163,
>>>>> value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
>>>>> Message=org.apache.solr.
>>>>> common.SolrException: ERROR: [doc=1-121180294489] multiple values
>>>>> encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
>>>>> che.lucene.document.LazyDocument$LazyField@2b119ab6,
>>>>> org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
>>>>> org.apache.lucene.docu
>>>>> ment.LazyDocument$LazyField@7dba464d,
>>>>> org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
>>>>> org.apache.lucene.document.LazyDocume
>>>>> nt$LazyField@4b19be97,
>>>>> org.apache.lucene.document.LazyDocument$LazyField@5978e924,
>>>>> org.apache.lucene.document.LazyDocument$LazyField@76
>>>>> --------------------------
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Hari
>>>>> Mobile:9790756568
>>>>>
>>>> --
>>>> Charlie Hull - Managing Consultant at OpenSource Connections Limited
>>>> <www.o19s.com>
>>>> Founding member of The Search Network <https://thesearchnetwork.com/>
>>>> and co-author of Searching the Enterprise
>>>> <https://opensourceconnections.com/about-us/books-resources/>
>>>> tel/fax: +44 (0)8700 118334
>>>> mobile: +44 (0)7767 825828
>>>>
>>>> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
>>>> Amtsgericht Charlottenburg | HRB 230712 B
>>>> Geschäftsführer: John M. Woodell | David E. Pugh
>>>> Finanzamt: Berlin Finanzamt für Körperschaften II
>>>>
>> --
>> Charlie Hull - Managing Consultant at OpenSource Connections Limited
>> <www.o19s.com>
>> Founding member of The Search Network <https://thesearchnetwork.com/>
>> and co-author of Searching the Enterprise
>> <https://opensourceconnections.com/about-us/books-resources/>
>> tel/fax: +44 (0)8700 118334
>> mobile: +44 (0)7767 825828
>>
>> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
>> Amtsgericht Charlottenburg | HRB 230712 B
>> Geschäftsführer: John M. Woodell | David E. Pugh
>> Finanzamt: Berlin Finanzamt für Körperschaften II
>>
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
<www.o19s.com>
Founding member of The Search Network <https://thesearchnetwork.com/>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/about-us/books-resources/>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II
Re: Solr | OOM | Lazyfield
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi Charlie,
The multiple values error is not occurring only for that field. Its also
occurring for multiple fields and multiple stores(collections).
Ex:
2021-09-07 13:05:53.646 ERROR (qtp1198197478-1200) [c:quoteStore s:shard1
r:core_node8 x:quoteStore_shard1_replica_n7] o.a.s.h.RequestH
andlerBase org.apache.solr.common.SolrException: ERROR:
[doc=1-116339633161] multiple values encountered for non multiValued field
quot
eAgeCompleted: [N, N]
Our Dev Team says that its only after the Solr upgrade from 8.1 to 8.8.1
Also, Solr is going down very frequently (after every one hour) on the
nodes only where a specific store (quotestore) is available. Other nodes
are fine.
For the nodes which are going down, I see OOM error for one node and for
the other node we see zookeeper connection timeout error as below.
--------------------------------
2021-09-07 13:06:13.907 WARN (main-SendThread(
zookeeperhost.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Client session
timed out, ha
ve not heard from server in 23226ms for session id 0x40000139b3e00d3
2021-09-07 13:06:13.907 WARN (main-SendThread(
lxeisprdas10.corp.equinix.com:2185)) [ ] o.a.z.ClientCnxn Session
0x40000139b3e00d3 fo
r sever zookeeperhost.corp.equinix.com/10.250.12.54:2185, Closing socket
connection. Attempting reconnect except it is a SessionExpiredE
xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
Client session timed out, have not heard from server in 23226ms f
or session id 0x40000139b3e00d3
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
timed out, have not heard from server in 23226ms for session id
0x40000139b3e00d3
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
~[zookeeper-3.6.2.jar:3.6.2]
2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.commo
n.cloud.ConnectionManager@5521aab name: ZooKeeperConnection Watcher:
zk1.corp.equinix.com:2181,zk2.corp.equinix.com:21
82,zk3.corp.equinix.com:2183,zk4.corp.equinix.com:2184,
zk5.corp.equinix.com:2185 got event WatchedEvent stat
e:Disconnected type:None path:null path: null type: None
2021-09-07 13:06:14.010 WARN (zkConnectionManagerCallback-13-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has disconnected
2021-09-07 13:06:39.494 WARN (main-SendThread(
lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Client session
timed out, ha
ve not heard from server in 23533ms for session id 0x40000139b3e00d3
2021-09-07 13:06:39.494 WARN (main-SendThread(
lxeisprdas09.corp.equinix.com:2184)) [ ] o.a.z.ClientCnxn Session
0x40000139b3e00d3 fo
r sever zk4.corp.equinix.com/10.250.**.**:2184, Closing socket connection.
Attempting reconnect except it is a SessionExpiredE
xception. => org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
Client session timed out, have not heard from server in 23533ms f
or session id 0x40000139b3e00d3
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session
timed out, have not heard from server in 23533ms for session id
0x40000139b3e00d3
-------------------------------------------------------
Could you please suggest.
On Tue, Sep 7, 2021 at 5:49 PM Charlie Hull <ch...@opensourceconnections.com>
wrote:
> Sorry I should have worded that better - I suggest that you review your
> schema file and indexing process, so that you can understand why
> multiple values are being sent to the field 'quoteSLOCurrentStatus' when
> it's probably not defined as multivalued.
>
> C
>
> On 07/09/2021 12:30, HariBabu kuruva wrote:
> > Thank you Charlie.
> >
> > Please let me know if anything is required from my end.
> >
> > Also, we have added below filed as a work around
> > <enableLazyFieldLoading>false</enableLazyFieldLoading> in solrconfig.xml
> >
> > On Tue, Sep 7, 2021 at 4:03 PM Charlie Hull <
> chull@opensourceconnections.com>
> > wrote:
> >
> >> I doubt these are related, that second error looks like something
> >> triggered by indexing - attempting to add multiple values to a field
> >> that hasn't been defined in the schema as multivalued. I'd review your
> >> indexing process and schema first.
> >>
> >> Charlie
> >>
> >> On 07/09/2021 11:00, HariBabu kuruva wrote:
> >>> Hi All,
> >>>
> >>> We are getting OOM errors in the solr logs for only specific solr
> stores.
> >>>
> >>> And in the solr logs we see the below error. Is the OOM error could be
> >>> because of the below error.
> >>>
> >>> Also i see below Lazyfield error is spanned across thousands of lines.
> >>>
> >>> Please advise. This is a PROD environment.
> >>>
> >>> ---------------------------------
> >>> AsyncLogger error handling event seq=163,
> >>> value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
> >>> Message=org.apache.solr.
> >>> common.SolrException: ERROR: [doc=1-121180294489] multiple values
> >>> encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
> >>> che.lucene.document.LazyDocument$LazyField@2b119ab6,
> >>> org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
> >>> org.apache.lucene.docu
> >>> ment.LazyDocument$LazyField@7dba464d,
> >>> org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
> >>> org.apache.lucene.document.LazyDocume
> >>> nt$LazyField@4b19be97,
> >>> org.apache.lucene.document.LazyDocument$LazyField@5978e924,
> >>> org.apache.lucene.document.LazyDocument$LazyField@76
> >>> --------------------------
> >>>
> >>>
> >>> Thanks and Regards,
> >>> Hari
> >>> Mobile:9790756568
> >>>
> >> --
> >> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> >> <www.o19s.com>
> >> Founding member of The Search Network <https://thesearchnetwork.com/>
> >> and co-author of Searching the Enterprise
> >> <https://opensourceconnections.com/about-us/books-resources/>
> >> tel/fax: +44 (0)8700 118334
> >> mobile: +44 (0)7767 825828
> >>
> >> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> >> Amtsgericht Charlottenburg | HRB 230712 B
> >> Geschäftsführer: John M. Woodell | David E. Pugh
> >> Finanzamt: Berlin Finanzamt für Körperschaften II
> >>
> >
>
> --
> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> <www.o19s.com>
> Founding member of The Search Network <https://thesearchnetwork.com/>
> and co-author of Searching the Enterprise
> <https://opensourceconnections.com/about-us/books-resources/>
> tel/fax: +44 (0)8700 118334
> mobile: +44 (0)7767 825828
>
> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> Amtsgericht Charlottenburg | HRB 230712 B
> Geschäftsführer: John M. Woodell | David E. Pugh
> Finanzamt: Berlin Finanzamt für Körperschaften II
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr | OOM | Lazyfield
Posted by Charlie Hull <ch...@opensourceconnections.com>.
Sorry I should have worded that better - I suggest that you review your
schema file and indexing process, so that you can understand why
multiple values are being sent to the field 'quoteSLOCurrentStatus' when
it's probably not defined as multivalued.
C
On 07/09/2021 12:30, HariBabu kuruva wrote:
> Thank you Charlie.
>
> Please let me know if anything is required from my end.
>
> Also, we have added below filed as a work around
> <enableLazyFieldLoading>false</enableLazyFieldLoading> in solrconfig.xml
>
> On Tue, Sep 7, 2021 at 4:03 PM Charlie Hull <ch...@opensourceconnections.com>
> wrote:
>
>> I doubt these are related, that second error looks like something
>> triggered by indexing - attempting to add multiple values to a field
>> that hasn't been defined in the schema as multivalued. I'd review your
>> indexing process and schema first.
>>
>> Charlie
>>
>> On 07/09/2021 11:00, HariBabu kuruva wrote:
>>> Hi All,
>>>
>>> We are getting OOM errors in the solr logs for only specific solr stores.
>>>
>>> And in the solr logs we see the below error. Is the OOM error could be
>>> because of the below error.
>>>
>>> Also i see below Lazyfield error is spanned across thousands of lines.
>>>
>>> Please advise. This is a PROD environment.
>>>
>>> ---------------------------------
>>> AsyncLogger error handling event seq=163,
>>> value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
>>> Message=org.apache.solr.
>>> common.SolrException: ERROR: [doc=1-121180294489] multiple values
>>> encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
>>> che.lucene.document.LazyDocument$LazyField@2b119ab6,
>>> org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
>>> org.apache.lucene.docu
>>> ment.LazyDocument$LazyField@7dba464d,
>>> org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
>>> org.apache.lucene.document.LazyDocume
>>> nt$LazyField@4b19be97,
>>> org.apache.lucene.document.LazyDocument$LazyField@5978e924,
>>> org.apache.lucene.document.LazyDocument$LazyField@76
>>> --------------------------
>>>
>>>
>>> Thanks and Regards,
>>> Hari
>>> Mobile:9790756568
>>>
>> --
>> Charlie Hull - Managing Consultant at OpenSource Connections Limited
>> <www.o19s.com>
>> Founding member of The Search Network <https://thesearchnetwork.com/>
>> and co-author of Searching the Enterprise
>> <https://opensourceconnections.com/about-us/books-resources/>
>> tel/fax: +44 (0)8700 118334
>> mobile: +44 (0)7767 825828
>>
>> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
>> Amtsgericht Charlottenburg | HRB 230712 B
>> Geschäftsführer: John M. Woodell | David E. Pugh
>> Finanzamt: Berlin Finanzamt für Körperschaften II
>>
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
<www.o19s.com>
Founding member of The Search Network <https://thesearchnetwork.com/>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/about-us/books-resources/>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II
Re: Solr | OOM | Lazyfield
Posted by HariBabu kuruva <ha...@gmail.com>.
Thank you Charlie.
Please let me know if anything is required from my end.
Also, we have added below filed as a work around
<enableLazyFieldLoading>false</enableLazyFieldLoading> in solrconfig.xml
On Tue, Sep 7, 2021 at 4:03 PM Charlie Hull <ch...@opensourceconnections.com>
wrote:
> I doubt these are related, that second error looks like something
> triggered by indexing - attempting to add multiple values to a field
> that hasn't been defined in the schema as multivalued. I'd review your
> indexing process and schema first.
>
> Charlie
>
> On 07/09/2021 11:00, HariBabu kuruva wrote:
> > Hi All,
> >
> > We are getting OOM errors in the solr logs for only specific solr stores.
> >
> > And in the solr logs we see the below error. Is the OOM error could be
> > because of the below error.
> >
> > Also i see below Lazyfield error is spanned across thousands of lines.
> >
> > Please advise. This is a PROD environment.
> >
> > ---------------------------------
> > AsyncLogger error handling event seq=163,
> > value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
> > Message=org.apache.solr.
> > common.SolrException: ERROR: [doc=1-121180294489] multiple values
> > encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
> > che.lucene.document.LazyDocument$LazyField@2b119ab6,
> > org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
> > org.apache.lucene.docu
> > ment.LazyDocument$LazyField@7dba464d,
> > org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
> > org.apache.lucene.document.LazyDocume
> > nt$LazyField@4b19be97,
> > org.apache.lucene.document.LazyDocument$LazyField@5978e924,
> > org.apache.lucene.document.LazyDocument$LazyField@76
> > --------------------------
> >
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
> --
> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> <www.o19s.com>
> Founding member of The Search Network <https://thesearchnetwork.com/>
> and co-author of Searching the Enterprise
> <https://opensourceconnections.com/about-us/books-resources/>
> tel/fax: +44 (0)8700 118334
> mobile: +44 (0)7767 825828
>
> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> Amtsgericht Charlottenburg | HRB 230712 B
> Geschäftsführer: John M. Woodell | David E. Pugh
> Finanzamt: Berlin Finanzamt für Körperschaften II
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr | OOM | Lazyfield
Posted by Charlie Hull <ch...@opensourceconnections.com>.
I doubt these are related, that second error looks like something
triggered by indexing - attempting to add multiple values to a field
that hasn't been defined in the schema as multivalued. I'd review your
indexing process and schema first.
Charlie
On 07/09/2021 11:00, HariBabu kuruva wrote:
> Hi All,
>
> We are getting OOM errors in the solr logs for only specific solr stores.
>
> And in the solr logs we see the below error. Is the OOM error could be
> because of the below error.
>
> Also i see below Lazyfield error is spanned across thousands of lines.
>
> Please advise. This is a PROD environment.
>
> ---------------------------------
> AsyncLogger error handling event seq=163,
> value='Logger=org.apache.solr.handler.RequestHandlerBase Level=ERROR
> Message=org.apache.solr.
> common.SolrException: ERROR: [doc=1-121180294489] multiple values
> encountered for non multiValued field quoteSLOCurrentStatus: [org.apa
> che.lucene.document.LazyDocument$LazyField@2b119ab6,
> org.apache.lucene.document.LazyDocument$LazyField@70dc55ed,
> org.apache.lucene.docu
> ment.LazyDocument$LazyField@7dba464d,
> org.apache.lucene.document.LazyDocument$LazyField@6cfa4d94,
> org.apache.lucene.document.LazyDocume
> nt$LazyField@4b19be97,
> org.apache.lucene.document.LazyDocument$LazyField@5978e924,
> org.apache.lucene.document.LazyDocument$LazyField@76
> --------------------------
>
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
<www.o19s.com>
Founding member of The Search Network <https://thesearchnetwork.com/>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/about-us/books-resources/>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II