You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by yaswanth kumar <ya...@gmail.com> on 2020/06/02 02:00:00 UTC
solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
configurations with all the certs in place, but the issue what I am seeing
is when trying to hit /update api on non-leader solr node , its throwing an
error
configured 2 solr nodes with 1 zookeeper.
metadata":[
"error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
"root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
"msg":"Async exception during distributed update:
javax.crypto.BadPaddingException: RSA private key operation failed",
"trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update:
javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.finish........
Strangely this is happening when we try to hit a non-leader node, hitting
leader node its working fine without any issue and getting the data indexed.
Not able to track down where the exact issue is happening.
Thanks,
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by Jörn Franke <jo...@gmail.com>.
If the keystore and/or truststore is encrypted you need to provide the Passwort in solr.in.sh
> Am 04.06.2020 um 18:38 schrieb yaswanth kumar <ya...@gmail.com>:
>
> I haven't done any changes on jetty xml , I am just using what it comes
> with the solr package. just doing it in solr.in.sh but I am still seeing
> the same issue.
>
> Thanks,
>
>> On Thu, Jun 4, 2020 at 12:23 PM Jörn Franke <jo...@gmail.com> wrote:
>>
>> I think you should not do it in the Jetty xml
>> Follow the official reference guide.
>> It should be in solr.in.sh
>>
>> https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html
>>
>>
>>
>>
>>>> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <ya...@gmail.com>:
>>>
>>> Hi Franke,
>>>
>>> I suspect its because of the certificate encryption ?? But will wait for
>>> you to confirm the same. We are trying to generate a certs with RSA 2048
>>> and finally combining them to a single JKS and that's what we are
>> referring
>>> as a keystore and truststore, let me know if it doesn't work or if there
>> is
>>> a standard procedure to do this certs.
>>>
>>> Thanks,
>>>
>>>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <ya...@gmail.com>
>> wrote:
>>>>
>>>> thanks Franke,
>>>>
>>>> I now made the use of the default jetty-ssl.xml that comes with the solr
>>>> package, but the issue is still happening when I try to push data to a
>>>> non-leader node.
>>>>
>>>> Do you still think if its something to do with the configurations ??
>>>>
>>>> Thanks,
>>>>
>>>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com>
>> wrote:
>>>>>
>>>>> Why in the jetty-ssl.xml?
>>>>>
>>>>> Should this not be configured in the solr.in.sh?
>>>>>
>>>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <yaswanthcse@gmail.com
>>> :
>>>>>>
>>>>>> Thanks Franke, but yes for all these questions I did configured it
>>>>>> properly, I made sure to include
>>>>>>
>>>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>>>> default="JKS"/></Set>
>>>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>>>> default="JKS"/></Set>
>>>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>>>
>>>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>>>> using the same file for both keystore and truststore as below
>>>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>>>> default="./etc/solr-keystore.jks"/></Set>
>>>>>> <Set name="KeyStorePassword"><Property
>>>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>>>> default="./etc/solr-keystore.jks"/></Set>
>>>>>> <Set name="TrustStorePassword"><Property
>>>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>>>
>>>>>> also urlScheme for ZK is set to https
>>>>>>
>>>>>>
>>>>>> Also the main error that I posted is the one that I am seeing as a
>>>>> return
>>>>>> response where as the below one is what I see from solr logs
>>>>>>
>>>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>>>> null:org.apache.solr.update.processor.Distr$
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>>>> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>>>> at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>>>> at
>>>>>>
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>>> at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>>>
>>>>>>
>>>>>> One strange observation is that when I hit update api on the leader
>> node
>>>>>> its working without any error, and now immediately if I hit non-leader
>>>>> its
>>>>>> working fine (only once or twice), but if I keep on trying to hit this
>>>>> node
>>>>>> again and again its then throwing the above error and once the error
>>>>>> started happening , its consistent again.
>>>>>>
>>>>>> Please let me know if you need more information or if I am missing
>>>>>> something else
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>> Have you looked in the logfiles?
>>>>>>>
>>>>>>> Keystore Type correctly defined on all nodes?
>>>>>>>
>>>>>>> Have you configured the truststore on all nodes correctly?
>>>>>>>
>>>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <
>> yaswanthcse@gmail.com
>>>>>> :
>>>>>>>>
>>>>>>>> team, can someone help me on the above topic?
>>>>>>>>
>>>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>>>> yaswanthcse@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the
>> ssl
>>>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>>>> seeing
>>>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>>>> throwing an
>>>>>>>>> error
>>>>>>>>>
>>>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>>>
>>>>>>>>> metadata":[
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>>>> "msg":"Async exception during distributed update:
>>>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>> failed",
>>>>>>>>>
>>>>>>>
>>>>>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>>>> Async exception during distributed update:
>>>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>>>> failed\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>>>
>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>>>
>>>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>>>> hitting
>>>>>>>>> leader node its working fine without any issue and getting the data
>>>>>>> indexed.
>>>>>>>>>
>>>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Yaswanth Kumar Konathala.
>>>>>>>>> yaswanthcse@gmail.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks & Regards,
>>>>>>>> Yaswanth Kumar Konathala.
>>>>>>>> yaswanthcse@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> yaswanthcse@gmail.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Yaswanth Kumar Konathala.
>>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
I haven't done any changes on jetty xml , I am just using what it comes
with the solr package. just doing it in solr.in.sh but I am still seeing
the same issue.
Thanks,
On Thu, Jun 4, 2020 at 12:23 PM Jörn Franke <jo...@gmail.com> wrote:
> I think you should not do it in the Jetty xml
> Follow the official reference guide.
> It should be in solr.in.sh
>
> https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html
>
>
>
>
> > Am 04.06.2020 um 06:48 schrieb yaswanth kumar <ya...@gmail.com>:
> >
> > Hi Franke,
> >
> > I suspect its because of the certificate encryption ?? But will wait for
> > you to confirm the same. We are trying to generate a certs with RSA 2048
> > and finally combining them to a single JKS and that's what we are
> referring
> > as a keystore and truststore, let me know if it doesn't work or if there
> is
> > a standard procedure to do this certs.
> >
> > Thanks,
> >
> >> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <ya...@gmail.com>
> wrote:
> >>
> >> thanks Franke,
> >>
> >> I now made the use of the default jetty-ssl.xml that comes with the solr
> >> package, but the issue is still happening when I try to push data to a
> >> non-leader node.
> >>
> >> Do you still think if its something to do with the configurations ??
> >>
> >> Thanks,
> >>
> >>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com>
> wrote:
> >>>
> >>> Why in the jetty-ssl.xml?
> >>>
> >>> Should this not be configured in the solr.in.sh?
> >>>
> >>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <yaswanthcse@gmail.com
> >:
> >>>>
> >>>> Thanks Franke, but yes for all these questions I did configured it
> >>>> properly, I made sure to include
> >>>>
> >>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> >>>> default="JKS"/></Set>
> >>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> >>>> default="JKS"/></Set>
> >>>> in the jetty-ssl.xml along with the path keystore and truststore.
> >>>>
> >>>> Also I have made sure that trusstore exists on all nodes and also I am
> >>>> using the same file for both keystore and truststore as below
> >>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> >>>> default="./etc/solr-keystore.jks"/></Set>
> >>>> <Set name="KeyStorePassword"><Property
> >>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
> >>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> >>>> default="./etc/solr-keystore.jks"/></Set>
> >>>> <Set name="TrustStorePassword"><Property
> >>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
> >>>>
> >>>> also urlScheme for ZK is set to https
> >>>>
> >>>>
> >>>> Also the main error that I posted is the one that I am seeing as a
> >>> return
> >>>> response where as the below one is what I see from solr logs
> >>>>
> >>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> >>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> >>>> null:org.apache.solr.update.processor.Distr$
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>> at
> >>>>
> >>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> >>>> at
> >>>>
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> >>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> >>>> at
> >>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> >>>> at
> >>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> >>>> at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> >>>> at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >>>>
> >>>>
> >>>> One strange observation is that when I hit update api on the leader
> node
> >>>> its working without any error, and now immediately if I hit non-leader
> >>> its
> >>>> working fine (only once or twice), but if I keep on trying to hit this
> >>> node
> >>>> again and again its then throwing the above error and once the error
> >>>> started happening , its consistent again.
> >>>>
> >>>> Please let me know if you need more information or if I am missing
> >>>> something else
> >>>>
> >>>> Thanks,
> >>>>
> >>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> Have you looked in the logfiles?
> >>>>>
> >>>>> Keystore Type correctly defined on all nodes?
> >>>>>
> >>>>> Have you configured the truststore on all nodes correctly?
> >>>>>
> >>>>> Have you set clusterprop urlScheme to htttps in ZK?
> >>>>>
> >>>>>
> >>>>>
> >>>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <
> yaswanthcse@gmail.com
> >>>> :
> >>>>>>
> >>>>>> team, can someone help me on the above topic?
> >>>>>>
> >>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
> >>> yaswanthcse@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the
> ssl
> >>>>>>> configurations with all the certs in place, but the issue what I am
> >>>>> seeing
> >>>>>>> is when trying to hit /update api on non-leader solr node , its
> >>>>> throwing an
> >>>>>>> error
> >>>>>>>
> >>>>>>> configured 2 solr nodes with 1 zookeeper.
> >>>>>>>
> >>>>>>> metadata":[
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >>>>>>> "msg":"Async exception during distributed update:
> >>>>>>> javax.crypto.BadPaddingException: RSA private key operation
> failed",
> >>>>>>>
> >>>>>
> >>>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >>>>>>> Async exception during distributed update:
> >>>>>>> javax.crypto.BadPaddingException: RSA private key operation
> >>> failed\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >>>>>>>
> >>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>>>>>>
> >>>>>>> Strangely this is happening when we try to hit a non-leader node,
> >>>>> hitting
> >>>>>>> leader node its working fine without any issue and getting the data
> >>>>> indexed.
> >>>>>>>
> >>>>>>> Not able to track down where the exact issue is happening.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> --
> >>>>>>> Thanks & Regards,
> >>>>>>> Yaswanth Kumar Konathala.
> >>>>>>> yaswanthcse@gmail.com
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Thanks & Regards,
> >>>>>> Yaswanth Kumar Konathala.
> >>>>>> yaswanthcse@gmail.com
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks & Regards,
> >>>> Yaswanth Kumar Konathala.
> >>>> yaswanthcse@gmail.com
> >>>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Yaswanth Kumar Konathala.
> >> yaswanthcse@gmail.com
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by Jörn Franke <jo...@gmail.com>.
I think you should not do it in the Jetty xml
Follow the official reference guide.
It should be in solr.in.sh
https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html
> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <ya...@gmail.com>:
>
> Hi Franke,
>
> I suspect its because of the certificate encryption ?? But will wait for
> you to confirm the same. We are trying to generate a certs with RSA 2048
> and finally combining them to a single JKS and that's what we are referring
> as a keystore and truststore, let me know if it doesn't work or if there is
> a standard procedure to do this certs.
>
> Thanks,
>
>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <ya...@gmail.com> wrote:
>>
>> thanks Franke,
>>
>> I now made the use of the default jetty-ssl.xml that comes with the solr
>> package, but the issue is still happening when I try to push data to a
>> non-leader node.
>>
>> Do you still think if its something to do with the configurations ??
>>
>> Thanks,
>>
>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com> wrote:
>>>
>>> Why in the jetty-ssl.xml?
>>>
>>> Should this not be configured in the solr.in.sh?
>>>
>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <ya...@gmail.com>:
>>>>
>>>> Thanks Franke, but yes for all these questions I did configured it
>>>> properly, I made sure to include
>>>>
>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>> default="JKS"/></Set>
>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>> default="JKS"/></Set>
>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>
>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>> using the same file for both keystore and truststore as below
>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="KeyStorePassword"><Property
>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="TrustStorePassword"><Property
>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>
>>>> also urlScheme for ZK is set to https
>>>>
>>>>
>>>> Also the main error that I posted is the one that I am seeing as a
>>> return
>>>> response where as the below one is what I see from solr logs
>>>>
>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>> null:org.apache.solr.update.processor.Distr$
>>>> at
>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>> at
>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>> at
>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>> at
>>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>> at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>> at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>> at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>> at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>> at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>> at
>>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>
>>>>
>>>> One strange observation is that when I hit update api on the leader node
>>>> its working without any error, and now immediately if I hit non-leader
>>> its
>>>> working fine (only once or twice), but if I keep on trying to hit this
>>> node
>>>> again and again its then throwing the above error and once the error
>>>> started happening , its consistent again.
>>>>
>>>> Please let me know if you need more information or if I am missing
>>>> something else
>>>>
>>>> Thanks,
>>>>
>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
>>> wrote:
>>>>>
>>>>> Have you looked in the logfiles?
>>>>>
>>>>> Keystore Type correctly defined on all nodes?
>>>>>
>>>>> Have you configured the truststore on all nodes correctly?
>>>>>
>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>
>>>>>
>>>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>
>>>>>
>>>>>
>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <yaswanthcse@gmail.com
>>>> :
>>>>>>
>>>>>> team, can someone help me on the above topic?
>>>>>>
>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>> yaswanthcse@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>> seeing
>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>> throwing an
>>>>>>> error
>>>>>>>
>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>
>>>>>>> metadata":[
>>>>>>>
>>>>>>>
>>>>>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>
>>>>>>>
>>>>>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>> "msg":"Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>>>>
>>>>>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>> Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>
>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>> hitting
>>>>>>> leader node its working fine without any issue and getting the data
>>>>> indexed.
>>>>>>>
>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> Thanks & Regards,
>>>>>>> Yaswanth Kumar Konathala.
>>>>>>> yaswanthcse@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> yaswanthcse@gmail.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by Jörn Franke <jo...@gmail.com>.
You need to separate keystore and truststore.
I would leave the stores in their original format and provide the type in solr.in.sh
There is no need to convert them to JKS, PKCS12 is perfectly supported
> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <ya...@gmail.com>:
>
> Hi Franke,
>
> I suspect its because of the certificate encryption ?? But will wait for
> you to confirm the same. We are trying to generate a certs with RSA 2048
> and finally combining them to a single JKS and that's what we are referring
> as a keystore and truststore, let me know if it doesn't work or if there is
> a standard procedure to do this certs.
>
> Thanks,
>
>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <ya...@gmail.com> wrote:
>>
>> thanks Franke,
>>
>> I now made the use of the default jetty-ssl.xml that comes with the solr
>> package, but the issue is still happening when I try to push data to a
>> non-leader node.
>>
>> Do you still think if its something to do with the configurations ??
>>
>> Thanks,
>>
>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com> wrote:
>>>
>>> Why in the jetty-ssl.xml?
>>>
>>> Should this not be configured in the solr.in.sh?
>>>
>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <ya...@gmail.com>:
>>>>
>>>> Thanks Franke, but yes for all these questions I did configured it
>>>> properly, I made sure to include
>>>>
>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>> default="JKS"/></Set>
>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>> default="JKS"/></Set>
>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>
>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>> using the same file for both keystore and truststore as below
>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="KeyStorePassword"><Property
>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="TrustStorePassword"><Property
>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>
>>>> also urlScheme for ZK is set to https
>>>>
>>>>
>>>> Also the main error that I posted is the one that I am seeing as a
>>> return
>>>> response where as the below one is what I see from solr logs
>>>>
>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>> null:org.apache.solr.update.processor.Distr$
>>>> at
>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>> at
>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>> at
>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>> at
>>>>
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>> at
>>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>> at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>> at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>> at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>> at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>> at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>> at
>>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>
>>>>
>>>> One strange observation is that when I hit update api on the leader node
>>>> its working without any error, and now immediately if I hit non-leader
>>> its
>>>> working fine (only once or twice), but if I keep on trying to hit this
>>> node
>>>> again and again its then throwing the above error and once the error
>>>> started happening , its consistent again.
>>>>
>>>> Please let me know if you need more information or if I am missing
>>>> something else
>>>>
>>>> Thanks,
>>>>
>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
>>> wrote:
>>>>>
>>>>> Have you looked in the logfiles?
>>>>>
>>>>> Keystore Type correctly defined on all nodes?
>>>>>
>>>>> Have you configured the truststore on all nodes correctly?
>>>>>
>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>
>>>>>
>>>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>
>>>>>
>>>>>
>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <yaswanthcse@gmail.com
>>>> :
>>>>>>
>>>>>> team, can someone help me on the above topic?
>>>>>>
>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>> yaswanthcse@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>> seeing
>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>> throwing an
>>>>>>> error
>>>>>>>
>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>
>>>>>>> metadata":[
>>>>>>>
>>>>>>>
>>>>>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>
>>>>>>>
>>>>>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>> "msg":"Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>>>>
>>>>>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>> Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>
>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>> hitting
>>>>>>> leader node its working fine without any issue and getting the data
>>>>> indexed.
>>>>>>>
>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> Thanks & Regards,
>>>>>>> Yaswanth Kumar Konathala.
>>>>>>> yaswanthcse@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> yaswanthcse@gmail.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
Hi Franke,
I suspect its because of the certificate encryption ?? But will wait for
you to confirm the same. We are trying to generate a certs with RSA 2048
and finally combining them to a single JKS and that's what we are referring
as a keystore and truststore, let me know if it doesn't work or if there is
a standard procedure to do this certs.
Thanks,
On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <ya...@gmail.com> wrote:
> thanks Franke,
>
> I now made the use of the default jetty-ssl.xml that comes with the solr
> package, but the issue is still happening when I try to push data to a
> non-leader node.
>
> Do you still think if its something to do with the configurations ??
>
> Thanks,
>
> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com> wrote:
>
>> Why in the jetty-ssl.xml?
>>
>> Should this not be configured in the solr.in.sh?
>>
>> > Am 03.06.2020 um 00:38 schrieb yaswanth kumar <ya...@gmail.com>:
>> >
>> > Thanks Franke, but yes for all these questions I did configured it
>> > properly, I made sure to include
>> >
>> > <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>> > default="JKS"/></Set>
>> > <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>> > default="JKS"/></Set>
>> > in the jetty-ssl.xml along with the path keystore and truststore.
>> >
>> > Also I have made sure that trusstore exists on all nodes and also I am
>> > using the same file for both keystore and truststore as below
>> > <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>> > default="./etc/solr-keystore.jks"/></Set>
>> > <Set name="KeyStorePassword"><Property
>> > name="solr.jetty.keystore.password" default="xxxx"/></Set>
>> > <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>> > default="./etc/solr-keystore.jks"/></Set>
>> > <Set name="TrustStorePassword"><Property
>> > name="solr.jetty.truststore.password" default="xxxx"/></Set>
>> >
>> > also urlScheme for ZK is set to https
>> >
>> >
>> > Also the main error that I posted is the one that I am seeing as a
>> return
>> > response where as the below one is what I see from solr logs
>> >
>> > 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>> > r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>> > null:org.apache.solr.update.processor.Distr$
>> > at
>> >
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>> > at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>> > at
>> >
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> > at
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>> > at
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>> > at
>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>> > at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> > at
>> >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> > at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> > at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> >
>> >
>> > One strange observation is that when I hit update api on the leader node
>> > its working without any error, and now immediately if I hit non-leader
>> its
>> > working fine (only once or twice), but if I keep on trying to hit this
>> node
>> > again and again its then throwing the above error and once the error
>> > started happening , its consistent again.
>> >
>> > Please let me know if you need more information or if I am missing
>> > something else
>> >
>> > Thanks,
>> >
>> >> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
>> wrote:
>> >>
>> >> Have you looked in the logfiles?
>> >>
>> >> Keystore Type correctly defined on all nodes?
>> >>
>> >> Have you configured the truststore on all nodes correctly?
>> >>
>> >> Have you set clusterprop urlScheme to htttps in ZK?
>> >>
>> >>
>> >>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>> >>
>> >>
>> >>
>> >>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <yaswanthcse@gmail.com
>> >:
>> >>>
>> >>> team, can someone help me on the above topic?
>> >>>
>> >>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>> yaswanthcse@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> >>>> configurations with all the certs in place, but the issue what I am
>> >> seeing
>> >>>> is when trying to hit /update api on non-leader solr node , its
>> >> throwing an
>> >>>> error
>> >>>>
>> >>>> configured 2 solr nodes with 1 zookeeper.
>> >>>>
>> >>>> metadata":[
>> >>>>
>> >>>>
>> >>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>> >>>>
>> >>>>
>> >>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> >>>> "msg":"Async exception during distributed update:
>> >>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>> >>>>
>> >>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> >>>> Async exception during distributed update:
>> >>>> javax.crypto.BadPaddingException: RSA private key operation
>> failed\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> >>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>> >>>>
>> >>>> Strangely this is happening when we try to hit a non-leader node,
>> >> hitting
>> >>>> leader node its working fine without any issue and getting the data
>> >> indexed.
>> >>>>
>> >>>> Not able to track down where the exact issue is happening.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> --
>> >>>> Thanks & Regards,
>> >>>> Yaswanth Kumar Konathala.
>> >>>> yaswanthcse@gmail.com
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Thanks & Regards,
>> >>> Yaswanth Kumar Konathala.
>> >>> yaswanthcse@gmail.com
>> >>
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Yaswanth Kumar Konathala.
>> > yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
thanks Franke,
I now made the use of the default jetty-ssl.xml that comes with the solr
package, but the issue is still happening when I try to push data to a
non-leader node.
Do you still think if its something to do with the configurations ??
Thanks,
On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <jo...@gmail.com> wrote:
> Why in the jetty-ssl.xml?
>
> Should this not be configured in the solr.in.sh?
>
> > Am 03.06.2020 um 00:38 schrieb yaswanth kumar <ya...@gmail.com>:
> >
> > Thanks Franke, but yes for all these questions I did configured it
> > properly, I made sure to include
> >
> > <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> > default="JKS"/></Set>
> > <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> > default="JKS"/></Set>
> > in the jetty-ssl.xml along with the path keystore and truststore.
> >
> > Also I have made sure that trusstore exists on all nodes and also I am
> > using the same file for both keystore and truststore as below
> > <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> > default="./etc/solr-keystore.jks"/></Set>
> > <Set name="KeyStorePassword"><Property
> > name="solr.jetty.keystore.password" default="xxxx"/></Set>
> > <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> > default="./etc/solr-keystore.jks"/></Set>
> > <Set name="TrustStorePassword"><Property
> > name="solr.jetty.truststore.password" default="xxxx"/></Set>
> >
> > also urlScheme for ZK is set to https
> >
> >
> > Also the main error that I posted is the one that I am seeing as a return
> > response where as the below one is what I see from solr logs
> >
> > 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> > r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> > null:org.apache.solr.update.processor.Distr$
> > at
> >
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> > at
> >
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> > at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >
> >
> > One strange observation is that when I hit update api on the leader node
> > its working without any error, and now immediately if I hit non-leader
> its
> > working fine (only once or twice), but if I keep on trying to hit this
> node
> > again and again its then throwing the above error and once the error
> > started happening , its consistent again.
> >
> > Please let me know if you need more information or if I am missing
> > something else
> >
> > Thanks,
> >
> >> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com>
> wrote:
> >>
> >> Have you looked in the logfiles?
> >>
> >> Keystore Type correctly defined on all nodes?
> >>
> >> Have you configured the truststore on all nodes correctly?
> >>
> >> Have you set clusterprop urlScheme to htttps in ZK?
> >>
> >>
> >>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
> >>
> >>
> >>
> >>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <yaswanthcse@gmail.com
> >:
> >>>
> >>> team, can someone help me on the above topic?
> >>>
> >>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <yaswanthcse@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> >>>> configurations with all the certs in place, but the issue what I am
> >> seeing
> >>>> is when trying to hit /update api on non-leader solr node , its
> >> throwing an
> >>>> error
> >>>>
> >>>> configured 2 solr nodes with 1 zookeeper.
> >>>>
> >>>> metadata":[
> >>>>
> >>>>
> >>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>>>
> >>>>
> >>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >>>> "msg":"Async exception during distributed update:
> >>>> javax.crypto.BadPaddingException: RSA private key operation failed",
> >>>>
> >>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >>>> Async exception during distributed update:
> >>>> javax.crypto.BadPaddingException: RSA private key operation
> failed\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>>>
> >>>> Strangely this is happening when we try to hit a non-leader node,
> >> hitting
> >>>> leader node its working fine without any issue and getting the data
> >> indexed.
> >>>>
> >>>> Not able to track down where the exact issue is happening.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> --
> >>>> Thanks & Regards,
> >>>> Yaswanth Kumar Konathala.
> >>>> yaswanthcse@gmail.com
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Yaswanth Kumar Konathala.
> >>> yaswanthcse@gmail.com
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by Jörn Franke <jo...@gmail.com>.
Why in the jetty-ssl.xml?
Should this not be configured in the solr.in.sh?
> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <ya...@gmail.com>:
>
> Thanks Franke, but yes for all these questions I did configured it
> properly, I made sure to include
>
> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> default="JKS"/></Set>
> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> default="JKS"/></Set>
> in the jetty-ssl.xml along with the path keystore and truststore.
>
> Also I have made sure that trusstore exists on all nodes and also I am
> using the same file for both keystore and truststore as below
> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="KeyStorePassword"><Property
> name="solr.jetty.keystore.password" default="xxxx"/></Set>
> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="TrustStorePassword"><Property
> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>
> also urlScheme for ZK is set to https
>
>
> Also the main error that I posted is the one that I am seeing as a return
> response where as the below one is what I see from solr logs
>
> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> null:org.apache.solr.update.processor.Distr$
> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>
>
> One strange observation is that when I hit update api on the leader node
> its working without any error, and now immediately if I hit non-leader its
> working fine (only once or twice), but if I keep on trying to hit this node
> again and again its then throwing the above error and once the error
> started happening , its consistent again.
>
> Please let me know if you need more information or if I am missing
> something else
>
> Thanks,
>
>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com> wrote:
>>
>> Have you looked in the logfiles?
>>
>> Keystore Type correctly defined on all nodes?
>>
>> Have you configured the truststore on all nodes correctly?
>>
>> Have you set clusterprop urlScheme to htttps in ZK?
>>
>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>
>>
>>
>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <ya...@gmail.com>:
>>>
>>> team, can someone help me on the above topic?
>>>
>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <ya...@gmail.com>
>>>> wrote:
>>>>
>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>> configurations with all the certs in place, but the issue what I am
>> seeing
>>>> is when trying to hit /update api on non-leader solr node , its
>> throwing an
>>>> error
>>>>
>>>> configured 2 solr nodes with 1 zookeeper.
>>>>
>>>> metadata":[
>>>>
>>>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>
>>>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>> "msg":"Async exception during distributed update:
>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>> Async exception during distributed update:
>>>> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>
>>>> Strangely this is happening when we try to hit a non-leader node,
>> hitting
>>>> leader node its working fine without any issue and getting the data
>> indexed.
>>>>
>>>> Not able to track down where the exact issue is happening.
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> yaswanthcse@gmail.com
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Yaswanth Kumar Konathala.
>>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
also forgot to update before that I have enabled basicauthentication and
provided the details in security.json and uploaded it via zookeeper.
Thanks,
On Tue, Jun 2, 2020 at 6:42 PM yaswanth kumar <ya...@gmail.com> wrote:
> also I am seeing the below error as a parent one from solr.log
>
> at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
> org.apache.solr.common.SolrException: javax.crypto.BadPaddingException:
> RSA private key operation failed
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
> ishan - 2020-01-10 1$
> at
> org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d0$
> at
> org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
> at
> org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 $
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:469)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.initOutStream(Http2SolrClient.java:324)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 - i$
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:227)
> ~[solr-solrj-8.4.1.jar:8.4.1 83$
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd918709$
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf6978$
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown
> Source) ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: javax.crypto.BadPaddingException: RSA private key operation
> failed
> at
> sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
> ~[?:?]
> at sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) ~[?:?]
> at sun.security.rsa.RSACore.rsa(RSACore.java:149) ~[?:?]
> at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355)
> ~[?:?]
> at
> com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) ~[?:?]
> at javax.crypto.Cipher.doFinal(Cipher.java:2260) ~[?:?]
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:364)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
> ishan - 2020-01-10 1$
> ... 13 more
>
> On Tue, Jun 2, 2020 at 6:37 PM yaswanth kumar <ya...@gmail.com>
> wrote:
>
>> Thanks Franke, but yes for all these questions I did configured it
>> properly, I made sure to include
>>
>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>> default="JKS"/></Set>
>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>> default="JKS"/></Set>
>> in the jetty-ssl.xml along with the path keystore and truststore.
>>
>> Also I have made sure that trusstore exists on all nodes and also I am
>> using the same file for both keystore and truststore as below
>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>> default="./etc/solr-keystore.jks"/></Set>
>> <Set name="KeyStorePassword"><Property
>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>> default="./etc/solr-keystore.jks"/></Set>
>> <Set name="TrustStorePassword"><Property
>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>
>> also urlScheme for ZK is set to https
>>
>>
>> Also the main error that I posted is the one that I am seeing as a return
>> response where as the below one is what I see from solr logs
>>
>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>> null:org.apache.solr.update.processor.Distr$
>> at
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>> at
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>> at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>> at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>
>>
>> One strange observation is that when I hit update api on the leader node
>> its working without any error, and now immediately if I hit non-leader its
>> working fine (only once or twice), but if I keep on trying to hit this node
>> again and again its then throwing the above error and once the error
>> started happening , its consistent again.
>>
>> Please let me know if you need more information or if I am missing
>> something else
>>
>> Thanks,
>>
>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com> wrote:
>>
>>> Have you looked in the logfiles?
>>>
>>> Keystore Type correctly defined on all nodes?
>>>
>>> Have you configured the truststore on all nodes correctly?
>>>
>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>
>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>
>>>
>>>
>>> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <ya...@gmail.com>:
>>> >
>>> > team, can someone help me on the above topic?
>>> >
>>> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <yaswanthcse@gmail.com
>>> >
>>> >> wrote:
>>> >>
>>> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>> >> configurations with all the certs in place, but the issue what I am
>>> seeing
>>> >> is when trying to hit /update api on non-leader solr node , its
>>> throwing an
>>> >> error
>>> >>
>>> >> configured 2 solr nodes with 1 zookeeper.
>>> >>
>>> >> metadata":[
>>> >>
>>> >>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>> >>
>>> >>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>> >> "msg":"Async exception during distributed update:
>>> >> javax.crypto.BadPaddingException: RSA private key operation failed",
>>> >>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>> >> Async exception during distributed update:
>>> >> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>> >>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>> >>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>> >>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>> >>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>> >>
>>> >> Strangely this is happening when we try to hit a non-leader node,
>>> hitting
>>> >> leader node its working fine without any issue and getting the data
>>> indexed.
>>> >>
>>> >> Not able to track down where the exact issue is happening.
>>> >>
>>> >> Thanks,
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Yaswanth Kumar Konathala.
>>> >> yaswanthcse@gmail.com
>>> >>
>>> >
>>> >
>>> > --
>>> > Thanks & Regards,
>>> > Yaswanth Kumar Konathala.
>>> > yaswanthcse@gmail.com
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
also I am seeing the below error as a parent one from solr.log
at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
private key operation failed
at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
at
org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d0$
at
org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
at
org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 $
at
org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:469)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
at
org.apache.solr.client.solrj.impl.Http2SolrClient.initOutStream(Http2SolrClient.java:324)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 - i$
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:227)
~[solr-solrj-8.4.1.jar:8.4.1 83$
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd918709$
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
~[metrics-core-4.0.5.jar:4.0.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf6978$
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown
Source) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.crypto.BadPaddingException: RSA private key operation
failed
at
sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
~[?:?]
at sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) ~[?:?]
at sun.security.rsa.RSACore.rsa(RSACore.java:149) ~[?:?]
at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355)
~[?:?]
at
com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) ~[?:?]
at javax.crypto.Cipher.doFinal(Cipher.java:2260) ~[?:?]
at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:364)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
... 13 more
On Tue, Jun 2, 2020 at 6:37 PM yaswanth kumar <ya...@gmail.com> wrote:
> Thanks Franke, but yes for all these questions I did configured it
> properly, I made sure to include
>
> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> default="JKS"/></Set>
> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> default="JKS"/></Set>
> in the jetty-ssl.xml along with the path keystore and truststore.
>
> Also I have made sure that trusstore exists on all nodes and also I am
> using the same file for both keystore and truststore as below
> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="KeyStorePassword"><Property
> name="solr.jetty.keystore.password" default="xxxx"/></Set>
> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="TrustStorePassword"><Property
> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>
> also urlScheme for ZK is set to https
>
>
> Also the main error that I posted is the one that I am seeing as a return
> response where as the below one is what I see from solr logs
>
> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> null:org.apache.solr.update.processor.Distr$
> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>
>
> One strange observation is that when I hit update api on the leader node
> its working without any error, and now immediately if I hit non-leader its
> working fine (only once or twice), but if I keep on trying to hit this node
> again and again its then throwing the above error and once the error
> started happening , its consistent again.
>
> Please let me know if you need more information or if I am missing
> something else
>
> Thanks,
>
> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com> wrote:
>
>> Have you looked in the logfiles?
>>
>> Keystore Type correctly defined on all nodes?
>>
>> Have you configured the truststore on all nodes correctly?
>>
>> Have you set clusterprop urlScheme to htttps in ZK?
>>
>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>
>>
>>
>> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <ya...@gmail.com>:
>> >
>> > team, can someone help me on the above topic?
>> >
>> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <ya...@gmail.com>
>> >> wrote:
>> >>
>> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> >> configurations with all the certs in place, but the issue what I am
>> seeing
>> >> is when trying to hit /update api on non-leader solr node , its
>> throwing an
>> >> error
>> >>
>> >> configured 2 solr nodes with 1 zookeeper.
>> >>
>> >> metadata":[
>> >>
>> >>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>> >>
>> >>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> >> "msg":"Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation failed",
>> >>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> >> Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation
>> failed\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>> >>
>> >> Strangely this is happening when we try to hit a non-leader node,
>> hitting
>> >> leader node its working fine without any issue and getting the data
>> indexed.
>> >>
>> >> Not able to track down where the exact issue is happening.
>> >>
>> >> Thanks,
>> >>
>> >> --
>> >> Thanks & Regards,
>> >> Yaswanth Kumar Konathala.
>> >> yaswanthcse@gmail.com
>> >>
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Yaswanth Kumar Konathala.
>> > yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
Thanks Franke, but yes for all these questions I did configured it
properly, I made sure to include
<Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
default="JKS"/></Set>
<Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
default="JKS"/></Set>
in the jetty-ssl.xml along with the path keystore and truststore.
Also I have made sure that trusstore exists on all nodes and also I am
using the same file for both keystore and truststore as below
<Set name="KeyStorePath"><Property name="solr.jetty.keystore"
default="./etc/solr-keystore.jks"/></Set>
<Set name="KeyStorePassword"><Property
name="solr.jetty.keystore.password" default="xxxx"/></Set>
<Set name="TrustStorePath"><Property name="solr.jetty.truststore"
default="./etc/solr-keystore.jks"/></Set>
<Set name="TrustStorePassword"><Property
name="solr.jetty.truststore.password" default="xxxx"/></Set>
also urlScheme for ZK is set to https
Also the main error that I posted is the one that I am seeing as a return
response where as the below one is what I see from solr logs
2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
null:org.apache.solr.update.processor.Distr$
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
One strange observation is that when I hit update api on the leader node
its working without any error, and now immediately if I hit non-leader its
working fine (only once or twice), but if I keep on trying to hit this node
again and again its then throwing the above error and once the error
started happening , its consistent again.
Please let me know if you need more information or if I am missing
something else
Thanks,
On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jo...@gmail.com> wrote:
> Have you looked in the logfiles?
>
> Keystore Type correctly defined on all nodes?
>
> Have you configured the truststore on all nodes correctly?
>
> Have you set clusterprop urlScheme to htttps in ZK?
>
>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>
>
>
> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <ya...@gmail.com>:
> >
> > team, can someone help me on the above topic?
> >
> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <ya...@gmail.com>
> >> wrote:
> >>
> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> >> configurations with all the certs in place, but the issue what I am
> seeing
> >> is when trying to hit /update api on non-leader solr node , its
> throwing an
> >> error
> >>
> >> configured 2 solr nodes with 1 zookeeper.
> >>
> >> metadata":[
> >>
> >>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>
> >>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >> "msg":"Async exception during distributed update:
> >> javax.crypto.BadPaddingException: RSA private key operation failed",
> >>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> Async exception during distributed update:
> >> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
> >>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>
> >> Strangely this is happening when we try to hit a non-leader node,
> hitting
> >> leader node its working fine without any issue and getting the data
> indexed.
> >>
> >> Not able to track down where the exact issue is happening.
> >>
> >> Thanks,
> >>
> >> --
> >> Thanks & Regards,
> >> Yaswanth Kumar Konathala.
> >> yaswanthcse@gmail.com
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by Jörn Franke <jo...@gmail.com>.
Have you looked in the logfiles?
Keystore Type correctly defined on all nodes?
Have you configured the truststore on all nodes correctly?
Have you set clusterprop urlScheme to htttps in ZK?
https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <ya...@gmail.com>:
>
> team, can someone help me on the above topic?
>
>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <ya...@gmail.com>
>> wrote:
>>
>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> configurations with all the certs in place, but the issue what I am seeing
>> is when trying to hit /update api on non-leader solr node , its throwing an
>> error
>>
>> configured 2 solr nodes with 1 zookeeper.
>>
>> metadata":[
>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> "msg":"Async exception during distributed update:
>> javax.crypto.BadPaddingException: RSA private key operation failed",
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> Async exception during distributed update:
>> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>
>> Strangely this is happening when we try to hit a non-leader node, hitting
>> leader node its working fine without any issue and getting the data indexed.
>>
>> Not able to track down where the exact issue is happening.
>>
>> Thanks,
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanthcse@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node
Posted by yaswanth kumar <ya...@gmail.com>.
team, can someone help me on the above topic?
On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <ya...@gmail.com>
wrote:
> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> configurations with all the certs in place, but the issue what I am seeing
> is when trying to hit /update api on non-leader solr node , its throwing an
> error
>
> configured 2 solr nodes with 1 zookeeper.
>
> metadata":[
>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> "msg":"Async exception during distributed update:
> javax.crypto.BadPaddingException: RSA private key operation failed",
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> Async exception during distributed update:
> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>
> Strangely this is happening when we try to hit a non-leader node, hitting
> leader node its working fine without any issue and getting the data indexed.
>
> Not able to track down where the exact issue is happening.
>
> Thanks,
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com