You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by HariBabu kuruva <ha...@gmail.com> on 2023/02/14 13:00:11 UTC
PROD Solr errors
Hi All,
We are receiving below errors when the manual indexing is initialized for
one of the stores, and its working fine for all other stores. The certs are
updated and in place.
*Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for all
10 nodes. ~48 GB Heap is assigned.
******************************************
org.apache.solr.client.solrj.SolrServerException: No live SolrServers
available to handle this request:[https://*****.com:8981/solr/s_doc_quote,
https://l******.com:8981/solr/s_doc_quote]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
at
eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
at
eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
at
eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
at
eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at
org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
occurred when talking to server at: https://
*****.***.com:8981/solr/s_doc_quote
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
... 18 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at
java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
... 22 common frames omitted
ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in range
data processing ->
java.lang.NullPointerException: null
*************************************************
Also we could see Broken pipe errors in 4-5 nodes.
*Suppressed: java.io.IOException: Broken pipe*
*Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused by:
java.io.IOException: Broken pipe*
Could you please advise.
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: PROD Solr errors
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi ,
We are still seeing Broken Pipe errors in some of the Solr nodes. Could you
please let me know if something needs to be tuned ? We don't have any
network issues, we can't expect network issues because it's a PROD
environment. And the error is very frequent. Please help
*Logs below:*
*java.io.IOException: javax.net.ssl.SSLException: Received fatal alert:
internal_error*
at
org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:197)
~[jetty-client-9.4.44.v20210
927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:151)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:145)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df
70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b1
1819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:172)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e
83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:106)
~[solr-solrj-8.
11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy -
2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
~[solr-solrj-8.11.1.jar:8.11.1 0b
002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:351)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b1181
9df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.
java:237) ~[solr-solrj-8.11.1.jar:8.11.1
0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[s
olr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-
core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr-solrj-8
.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy -
2021-12-14 13:50:57]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*Suppressed: java.io.IOException: javax.net.ssl.SSLException:
Received fatal alert: internal_error*
at
org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:197)
~[jetty-client-9.4.4
4.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.j
ava:151) ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.j
ava:145) ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
~[solr-solrj-8.11.1.jar:8.11.1 0b002
b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
~[solr-solrj-8.11.1.jar:8.11.1
0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1277)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819d
f70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:107)
~[solr-
solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy -
2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
~[solr-solrj-8.11.1.jar:8
.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14
13:50:57]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:351)
~[solr-solrj-8.11.1.jar:8.11.1 0b
002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2Sol
rClient.java:237) ~[solr-solrj-8.11.1.jar:8.11.1
0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:
181) ~[solr-solrj-8.11.1.jar:8.11.1
0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[
metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr
-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy
- 2021-12-14 13:50:57]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLException: Received fatal alert:
internal_error
at
sun.security.ssl.Alert.createSSLException(Alert.java:129) ~[?:?]
at
sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
at
sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279) ~[?:?]
at
sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) ~[?:?]
at
sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
at
sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672) ~[?:?]
at
sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627) ~[?:?]
at
sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443) ~[?:?]
at
sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
at
org.eclipse.jetty.io.ssl.SslConnection.unwrap(SslConnection.java:429)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v2021
0927]
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:718)
~[jetty-io-9.4.44.v20210927.
jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.NegotiatingClientConnection.fill(NegotiatingClientConnection.java:99)
~[jetty-io-9.4.44.v2021
0927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:81)
~[jetty-io-9.4.44
.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
~[jetty-io-9.4.44.v202
10927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927
]
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
~[jetty-io-9.4.44.v202
10927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v
20210927]
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.
v20210927]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927
]
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210
927]
... 4 more
*Suppressed: java.io.IOException: Broken pipe*
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1135)
~[jetty-io-9.4.44.
v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v202
10927]
at
org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v202
10927]
at
org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
~[jetty-io-9.4.44.v20210927.jar:9.4
.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:295)
~[http2-common-9.4.44.v20210927.jar:9.
4.44.v20210927]
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
~[jetty-util-9.4.44.v20210
927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
~[jetty-util-9.4.44.v20210927
.jar:9.4.44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:729)
~[http2-common-9.4.44.v20210927.jar:9.4.
44.v20210927]
at
org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:701)
~[http2-common-9.4.44.v20210927.jar:9.4
.44.v20210927]
at
org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnec
tionFactory.java:116) ~[http2-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:444)
~[jetty-io-9.4.44.v20210927.jar:9
.4.44.v20210927]
at
org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:114)
~
[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:84)
~[jetty-i
o-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
~[jetty-io-9.4
.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v
20210927]
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
~[jetty-io-9.4
.44.v20210927.jar:9.4.44.v20210927]
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
~[jetty-io-9.4.44.v20210927.jar:
9.4.44.v20210927]
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
~[jetty-io-9.4.44.v20210927.jar
:9.4.44.v20210927]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.44.v20210927.jar:9.4.44.v
20210927]
On Fri, Feb 17, 2023 at 9:27 AM HariBabu kuruva <ha...@gmail.com>
wrote:
> Hi
>
> Thank you for your reply. I will try to check if there was any network
> issue.
>
> Yes we were facing OOM issues and we had to increase the HEAP. We will try
> to see if we can reduce it.
>
> On Thu, Feb 16, 2023 at 10:12 PM Gus Heck <gu...@gmail.com> wrote:
>
>> If you are 100% sure no machines or solr software were restarted, you had
>> a
>> networking issue that broke communications, perhaps a switch or router
>> restart, someone re-plugging cables, or a router that got overwhelmed, (or
>> is failing)?
>>
>> Unrelated side note:
>>
>> 48GB heap is not usually a great setting. @32GB the JVM switches to 64
>> bit
>> pointers, and so going above 31GB is actually less memory until somewhere
>> around 38-40GB. Meanwhile you operating system can't use that memory for
>> disk caching. Of course if you had heavy use cases that were causing you
>> to
>> see OutOfMemoryError that forced you into that setting nevermind.
>> Certainly
>> sometimes super-sized heap is necessary, but usually not. If that setting
>> was chosen via "hmmm, this looks good" ... it doesn't look good to me ;)
>>
>> On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva <
>> hari2708.kuruva@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Thanks for your reply.
>> >
>> > But, I can see all the solr nodes are up and running, and I have
>> manually
>> > accessed the solr url with each node , it's accessible.
>> >
>> > On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma <
>> markus.jelsma@openindex.io>
>> > wrote:
>> >
>> > > Hello Hari,
>> > >
>> > > One or more Solr servers was down when sending your data. The broken
>> pipe
>> > > messages indicate the same, it happens when connections are broken.
>> > >
>> > > Find out which servers went offline/unavailable, and see what they
>> wrote
>> > in
>> > > their logs.
>> > >
>> > > Regards,
>> > > Markus
>> > >
>> > > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <
>> > > hari2708.kuruva@gmail.com>:
>> > >
>> > > > Hi All,
>> > > >
>> > > > We are receiving below errors when the manual indexing is
>> initialized
>> > for
>> > > > one of the stores, and its working fine for all other stores. The
>> certs
>> > > are
>> > > > updated and in place.
>> > > >
>> > > > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu
>> for
>> > > all
>> > > > 10 nodes. ~48 GB Heap is assigned.
>> > > >
>> > > > ******************************************
>> > > > org.apache.solr.client.solrj.SolrServerException: No live
>> SolrServers
>> > > > available to handle this request:[https://
>> > > *****.com:8981/solr/s_doc_quote,
>> > > > https://l******.com:8981/solr/s_doc_quote]
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
>> > > > at
>> > > >
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
>> > > > at
>> > > >
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
>> > > > at
>> > > >
>> > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
>> > > > at
>> > > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:834)
>> > > > Caused by: org.apache.solr.client.solrj.SolrServerException:
>> > IOException
>> > > > occurred when talking to server at: https://
>> > > > *****.***.com:8981/solr/s_doc_quote
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
>> > > > ... 18 common frames omitted
>> > > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not
>> > > authenticated
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
>> > > > at
>> > > > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
>> > > > ... 22 common frames omitted
>> > > > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
>> > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in
>> > > range
>> > > > data processing ->
>> > > > java.lang.NullPointerException: null
>> > > >
>> > > > *************************************************
>> > > >
>> > > > Also we could see Broken pipe errors in 4-5 nodes.
>> > > > *Suppressed: java.io.IOException: Broken pipe*
>> > > >
>> > > > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused
>> by:
>> > > > java.io.IOException: Broken pipe*
>> > > >
>> > > >
>> > > > Could you please advise.
>> > > >
>> > > > --
>> > > >
>> > > > Thanks and Regards,
>> > > > Hari
>> > > > Mobile:9790756568
>> > > >
>> > >
>> >
>> >
>> > --
>> >
>> > Thanks and Regards,
>> > Hari
>> > Mobile:9790756568
>> >
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: PROD Solr errors
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi
Thank you for your reply. I will try to check if there was any network
issue.
Yes we were facing OOM issues and we had to increase the HEAP. We will try
to see if we can reduce it.
On Thu, Feb 16, 2023 at 10:12 PM Gus Heck <gu...@gmail.com> wrote:
> If you are 100% sure no machines or solr software were restarted, you had a
> networking issue that broke communications, perhaps a switch or router
> restart, someone re-plugging cables, or a router that got overwhelmed, (or
> is failing)?
>
> Unrelated side note:
>
> 48GB heap is not usually a great setting. @32GB the JVM switches to 64 bit
> pointers, and so going above 31GB is actually less memory until somewhere
> around 38-40GB. Meanwhile you operating system can't use that memory for
> disk caching. Of course if you had heavy use cases that were causing you to
> see OutOfMemoryError that forced you into that setting nevermind. Certainly
> sometimes super-sized heap is necessary, but usually not. If that setting
> was chosen via "hmmm, this looks good" ... it doesn't look good to me ;)
>
> On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva <hari2708.kuruva@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > Thanks for your reply.
> >
> > But, I can see all the solr nodes are up and running, and I have manually
> > accessed the solr url with each node , it's accessible.
> >
> > On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma <
> markus.jelsma@openindex.io>
> > wrote:
> >
> > > Hello Hari,
> > >
> > > One or more Solr servers was down when sending your data. The broken
> pipe
> > > messages indicate the same, it happens when connections are broken.
> > >
> > > Find out which servers went offline/unavailable, and see what they
> wrote
> > in
> > > their logs.
> > >
> > > Regards,
> > > Markus
> > >
> > > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <
> > > hari2708.kuruva@gmail.com>:
> > >
> > > > Hi All,
> > > >
> > > > We are receiving below errors when the manual indexing is initialized
> > for
> > > > one of the stores, and its working fine for all other stores. The
> certs
> > > are
> > > > updated and in place.
> > > >
> > > > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu
> for
> > > all
> > > > 10 nodes. ~48 GB Heap is assigned.
> > > >
> > > > ******************************************
> > > > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> > > > available to handle this request:[https://
> > > *****.com:8981/solr/s_doc_quote,
> > > > https://l******.com:8981/solr/s_doc_quote]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
> > > > at
> > > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
> > > > at
> > > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
> > > > at
> > > >
> > > >
> > >
> >
> eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
> > > > at
> > > >
> > > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
> > > > at
> > > >
> > > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
> > > > at
> > > >
> > > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
> > > > at
> > > >
> > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> > > > at
> > > >
> > > >
> > >
> >
> org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
> > > > at
> > > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:834)
> > > > Caused by: org.apache.solr.client.solrj.SolrServerException:
> > IOException
> > > > occurred when talking to server at: https://
> > > > *****.***.com:8981/solr/s_doc_quote
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
> > > > ... 18 common frames omitted
> > > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not
> > > authenticated
> > > > at
> > > >
> > > >
> > >
> >
> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> > > > at
> > > >
> > >
> >
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> > > > at
> > > > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> > > > at
> > > >
> > >
> >
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> > > > ... 22 common frames omitted
> > > > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
> > > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in
> > > range
> > > > data processing ->
> > > > java.lang.NullPointerException: null
> > > >
> > > > *************************************************
> > > >
> > > > Also we could see Broken pipe errors in 4-5 nodes.
> > > > *Suppressed: java.io.IOException: Broken pipe*
> > > >
> > > > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused
> by:
> > > > java.io.IOException: Broken pipe*
> > > >
> > > >
> > > > Could you please advise.
> > > >
> > > > --
> > > >
> > > > Thanks and Regards,
> > > > Hari
> > > > Mobile:9790756568
> > > >
> > >
> >
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: PROD Solr errors
Posted by Gus Heck <gu...@gmail.com>.
If you are 100% sure no machines or solr software were restarted, you had a
networking issue that broke communications, perhaps a switch or router
restart, someone re-plugging cables, or a router that got overwhelmed, (or
is failing)?
Unrelated side note:
48GB heap is not usually a great setting. @32GB the JVM switches to 64 bit
pointers, and so going above 31GB is actually less memory until somewhere
around 38-40GB. Meanwhile you operating system can't use that memory for
disk caching. Of course if you had heavy use cases that were causing you to
see OutOfMemoryError that forced you into that setting nevermind. Certainly
sometimes super-sized heap is necessary, but usually not. If that setting
was chosen via "hmmm, this looks good" ... it doesn't look good to me ;)
On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva <ha...@gmail.com>
wrote:
> Hi,
>
> Thanks for your reply.
>
> But, I can see all the solr nodes are up and running, and I have manually
> accessed the solr url with each node , it's accessible.
>
> On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma <ma...@openindex.io>
> wrote:
>
> > Hello Hari,
> >
> > One or more Solr servers was down when sending your data. The broken pipe
> > messages indicate the same, it happens when connections are broken.
> >
> > Find out which servers went offline/unavailable, and see what they wrote
> in
> > their logs.
> >
> > Regards,
> > Markus
> >
> > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <
> > hari2708.kuruva@gmail.com>:
> >
> > > Hi All,
> > >
> > > We are receiving below errors when the manual indexing is initialized
> for
> > > one of the stores, and its working fine for all other stores. The certs
> > are
> > > updated and in place.
> > >
> > > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for
> > all
> > > 10 nodes. ~48 GB Heap is assigned.
> > >
> > > ******************************************
> > > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> > > available to handle this request:[https://
> > *****.com:8981/solr/s_doc_quote,
> > > https://l******.com:8981/solr/s_doc_quote]
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
> > > at
> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
> > > at
> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
> > > at
> > >
> > >
> >
> eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
> > > at
> > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
> > > at
> > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
> > > at
> > >
> > >
> >
> eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
> > > at
> > >
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> > > at
> > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
> > > at
> > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > at
> > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> > > at
> > >
> > >
> >
> org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
> > > at
> > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:834)
> > > Caused by: org.apache.solr.client.solrj.SolrServerException:
> IOException
> > > occurred when talking to server at: https://
> > > *****.***.com:8981/solr/s_doc_quote
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
> > > ... 18 common frames omitted
> > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not
> > authenticated
> > > at
> > >
> > >
> >
> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
> > > at
> > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
> > > at
> > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> > > at
> > >
> > >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> > > at
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> > > at
> > >
> > >
> >
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> > > at
> > >
> > >
> >
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> > > at
> > >
> > >
> >
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> > > at
> > >
> >
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> > > at
> > > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> > > at
> > >
> >
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> > > at
> > >
> > >
> >
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> > > at
> > >
> > >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> > > at
> > >
> > >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> > > at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> > > ... 22 common frames omitted
> > > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
> > > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in
> > range
> > > data processing ->
> > > java.lang.NullPointerException: null
> > >
> > > *************************************************
> > >
> > > Also we could see Broken pipe errors in 4-5 nodes.
> > > *Suppressed: java.io.IOException: Broken pipe*
> > >
> > > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused by:
> > > java.io.IOException: Broken pipe*
> > >
> > >
> > > Could you please advise.
> > >
> > > --
> > >
> > > Thanks and Regards,
> > > Hari
> > > Mobile:9790756568
> > >
> >
>
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: PROD Solr errors
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi,
Thanks for your reply.
But, I can see all the solr nodes are up and running, and I have manually
accessed the solr url with each node , it's accessible.
On Tue, Feb 14, 2023 at 6:36 PM Markus Jelsma <ma...@openindex.io>
wrote:
> Hello Hari,
>
> One or more Solr servers was down when sending your data. The broken pipe
> messages indicate the same, it happens when connections are broken.
>
> Find out which servers went offline/unavailable, and see what they wrote in
> their logs.
>
> Regards,
> Markus
>
> Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <
> hari2708.kuruva@gmail.com>:
>
> > Hi All,
> >
> > We are receiving below errors when the manual indexing is initialized for
> > one of the stores, and its working fine for all other stores. The certs
> are
> > updated and in place.
> >
> > *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for
> all
> > 10 nodes. ~48 GB Heap is assigned.
> >
> > ******************************************
> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> > available to handle this request:[https://
> *****.com:8981/solr/s_doc_quote,
> > https://l******.com:8981/solr/s_doc_quote]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
> > at
> >
> >
> eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
> > at
> >
> >
> eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
> > at
> >
> >
> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
> > at
> >
> >
> eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
> > at
> > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> > at
> >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
> > at
> >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> >
> >
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> > at
> >
> >
> org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:834)
> > Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
> > occurred when talking to server at: https://
> > *****.***.com:8981/solr/s_doc_quote
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
> > ... 18 common frames omitted
> > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not
> authenticated
> > at
> >
> >
> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
> > at
> >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
> > at
> >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> > at
> >
> >
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> > at
> >
> >
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> > at
> >
> >
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> > at
> >
> >
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> > at
> >
> >
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> > at
> >
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> > at
> > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> > at
> >
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> > at
> >
> >
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> > at
> >
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> > at
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> > ... 22 common frames omitted
> > ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
> > eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in
> range
> > data processing ->
> > java.lang.NullPointerException: null
> >
> > *************************************************
> >
> > Also we could see Broken pipe errors in 4-5 nodes.
> > *Suppressed: java.io.IOException: Broken pipe*
> >
> > *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused by:
> > java.io.IOException: Broken pipe*
> >
> >
> > Could you please advise.
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: PROD Solr errors
Posted by Markus Jelsma <ma...@openindex.io>.
Hello Hari,
One or more Solr servers was down when sending your data. The broken pipe
messages indicate the same, it happens when connections are broken.
Find out which servers went offline/unavailable, and see what they wrote in
their logs.
Regards,
Markus
Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <
hari2708.kuruva@gmail.com>:
> Hi All,
>
> We are receiving below errors when the manual indexing is initialized for
> one of the stores, and its working fine for all other stores. The certs are
> updated and in place.
>
> *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for all
> 10 nodes. ~48 GB Heap is assigned.
>
> ******************************************
> org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> available to handle this request:[https://*****.com:8981/solr/s_doc_quote,
> https://l******.com:8981/solr/s_doc_quote]
> at
>
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:346)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1171)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
> at
>
> eisp.cdc.data.indexer.adapter.EispSolrCloudServiceImpl.getCursorSolrResponse(EispSolrCloudServiceImpl.java:197)
> at
>
> eisp.kafka.cdc.task.MainIndexerTask.dataIndexerProcess(MainIndexerTask.java:208)
> at
>
> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest(MainIndexerTask.java:147)
> at
>
> eisp.kafka.cdc.task.MainIndexerTask$$FastClassBySpringCGLIB$$88fcf6c3.invoke(<generated>)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
>
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
>
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> at
>
> org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
> occurred when talking to server at: https://
> *****.***.com:8981/solr/s_doc_quote
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> at
>
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
> at
>
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
> ... 18 common frames omitted
> Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
> at
>
> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
> at
>
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:504)
> at
>
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> at
>
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> at
>
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
>
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> at
>
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> at
>
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
>
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
>
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
> ... 22 common frames omitted
> ERROR <hostname) 14-02-2023 07:37:28.954 [mainRequestHandler-1]
> eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Error in range
> data processing ->
> java.lang.NullPointerException: null
>
> *************************************************
>
> Also we could see Broken pipe errors in 4-5 nodes.
> *Suppressed: java.io.IOException: Broken pipe*
>
> *Caused by: javax.net.ssl.SSLHandshakeException: Broken pipeCaused by:
> java.io.IOException: Broken pipe*
>
>
> Could you please advise.
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>