You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID> on 2023/08/24 03:34:23 UTC
RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
[11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
{
"responseHeader":{
"status":0,
"QTime":255},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":219}}}}
[11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
{
"responseHeader":{
"status":0,
"QTime":237},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":216}}}}
[11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Authentication failed, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
<table>
<tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
*[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
[11:51 dbh19850s 1155]$ less solr.log
...
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
...
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] has role, but request principal cannot be identified; forbidding access
-----Original Message-----
From: Kevin Risden <kr...@apache.org>
Sent: Monday, August 14, 2023 10:31 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
The issue w/ allowPaths was reported here:
https://issues.apache.org/jira/browse/SOLR-16905
Basically allowPaths was not being reflected in the security manager
policy. You can update security.policy yourself or disable security manager
as you found.
Kevin Risden
On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
> SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
> and incremental=true)
>
>
> -----Original Message-----
> From: Jan Høydahl <ja...@cominvent.com>
> Sent: Monday, August 14, 2023 3:12 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>
> This is probably due to SecurityManager rejecting an unknown path. Try
> starting Solr with:
>
> -Dsolr.allowPaths=/netmnt/
>
> or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
>
> Then try again. Alternatively you can try disabling SecMgr withenv.var
> SOLR_SECURITY_MANAGER_ENABLED=false
>
>
> https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
>
> Jan
>
> > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.INVALID>:
> >
> > Thanks: I was forgetting the fact that (in order to get backups in the
> format we expected) I added "incremental=false" to our backup script when
> we upgraded from Solr8.5 to Solr8.11.
> >
> > It trying to test this functionality, however, I now find that backups
> do not seem to be working for me *at all* in Solr9.2, with or without
> "incremental=false"
> >
> > I keep getting a stack trace such as:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <response>
> >
> > <lst name="responseHeader">
> > <int name="status">500</int>
> > <int name="QTime">9</int>
> > </lst>
> > <lst name="error">
> > <str name="msg">access denied ("java.io.FilePermission"
> "/netmnt/[redacted-directory]" "read")</str>
> > <str name="trace">java.security.AccessControlException: access denied
> ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
> > at
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > at
> java.base/java.security.AccessController.checkPermission(AccessController.java:897)
> > at
> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
> > at
> java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
> > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
> > at
> java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
> > at java.base/java.nio.file.Files.exists(Files.java:2435)
> > at
> org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
> > at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
> > at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> > at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> > at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> > at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:100)
> > at org.eclipse.jetty.io
> .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > </str>
> > <int name="code">500</int>
> > </lst>
> > </response>
> >
> >
> > It should be noted that the Linux account which runs Solr has read and
> write access to the specified location, which is a subdirectory of what is
> listed in allowPaths in all the solr.xml files of this SolrCloud, and which
> is mounted on all hosts of the SolrCloud.
> >
> > I have tried both V1 API and V2 API as found at
> solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
> >
> > Not being familiar with V2 API, I tried various permutations of
> "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
> the same stack trace
> >
> > Does anyone have any example of a successful backup from Solr9.2?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Dominique Bejean <do...@eolya.fr>
> > Sent: Friday, August 11, 2023 9:32 AM
> > To: users@solr.apache.org
> > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
> replication
> >
> > Hi Craig,
> >
> > Yes, starting with the 8.9 and 9.0 versions, Collection API allows
> > incremental backup and much more (corruption check, backup to Amazon S3
> or
> > Google Cloud Storage) .
> > Take a look at this umbrella JIRA
> > https://issues.apache.org/jira/browse/SOLR-15086
> >
> > Regards
> >
> > Dominique
> >
> >
> >
> > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.invalid> a écrit :
> >
> >> RE: a better option is to use the backup/restore functionality in the
> >> Collections API.
> >>
> >> My impression was that there is no facility for _incremental_ backup and
> >> retore in the collections API: is there? To do backup and scp and
> restore
> >> of terabytes of data every few minutes does not sound practical.
> >>
> >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
> as
> >> well as Solr) is redundancy (for failover) in our main Data Center, and
> >> redundancy in read-only reasonably-concurrent copies in our second Data
> >> Center. For Solr8.11 (and earlier), we have had SolrCloud for
> redundancy in
> >> our main Data Center, and Leader/Follower replication to the read-only
> >> SolrClouds in the second Data Center. At one time, we were hoping that
> CDCR
> >> would work better: but we never managed to get CDCR to work reliably
> (and
> >> perhaps others also found it unreliable, leading to it being deprecated
> >> rather than being fixed). We have found that SolrCloud does not work
> >> reliably when spread across Data Centers, so Leader/Follower replication
> >> (formerly known as Master/Slave replication) has been the only way we
> have
> >> found to keep our (read-only) copies in the second Data Center only a
> few
> >> minutes behind the data in the main Data Center (the only way to provide
> >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
> and
> >> MySQL). My supervisor was asking for clarification whether you are
> implying
> >> that Leader/Follower replication is being deprecated.
> >>
> >>
> >> In my continuing attempts to resolve these issues, I have come across a
> >> related question. The error message about SolrAuthV2 prompted me to
> wonder
> >> about the V1 and V2 syntax options (such as shown at
> >>
> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
> );
> >> and I was wondering whether Leader/Follower replication changed from
> using
> >> syntax V1 to using syntax V2, and if that might contribute to the
> breaking
> >> of permissions. As an experiment in our test environment, I setup a
> >> permission in security.json to allow RELOAD collection without a
> password.
> >> After confirming that my V2 syntax does work _with_ a password, I then
> >> attempted RELOAD collection without a password using both syntax V1 and
> >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
> several
> >> permutations in security.json to allow RELOAD without password in syntax
> >> V2, but have not yet found a successful permutation. Are there any
> >> clarifications what security.json changes are needed for syntax V2? Can
> it
> >> be confirmed whether Leader/Follower replication is using V2 (in other
> >> words, whether that may be contributing to the permission problem)?
> >>
> >>
> >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
> 'Content-Type:
> >> application/json' -d '{"reload":{}}'
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":255},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":219}}}}
> >> [11:51 dbh19850s 1152]$ curl "http://localhost:
> $p/solr/admin/collections?action=RELOAD&name=helpdocs"
> >>
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":237},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":216}}}}
> >> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
> $p/api/collections/helpdocs"
> >> -H 'Content-Type: application/json' -d '{"reload":{}}'
> >> <html>
> >> <head>
> >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> >> <title>Error 401 Authentication failed, Response code: 401</title>
> >> </head>
> >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> >> <table>
> >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> >> <tr><th>STATUS:</th><td>401</td></tr>
> >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> >> 401</td></tr>
> >> <tr><th>SERVLET:</th><td>default</td></tr>
> >> </table>
> >>
> >> </body>
> >> </html>
> >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> >>
> >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 401 491
> >> [11:51 dbh19850s 1155]$ less solr.log
> >> ...
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> >> [/admin/collections]
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> >> ...
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] has role, but request principal cannot be identified;
> >> forbidding access
> >>
> >>
> >> As a side note, in our experience, the only thing that has been
> cluttering
> >> up solr.log with attempts to connect without a password has been
> >> Leader/Follower replication (formerly known as Master/Slave
> replication).
> >>
> >>
> >> -----Original Message-----
> >> From: Shawn Heisey <ap...@elyograg.org>
> >> Sent: Saturday, August 5, 2023 4:24 PM
> >> To: users@solr.apache.org
> >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
> >>
> >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> >>> We are having problems transitioning Leader/Follower replication to
> >> Solr9.2.1
> >>>
> >>> In Solr8.5 and below, what was then called Master/Slave replication had
> >> the annoying problem that, even though we specified httpBasicAuthUser
> and
> >> httpBasicAuthPassword, it would always attempt to connect first without
> a
> >> password before retrying with a password. This made solr.log noisy with
> >> lots of unnecessary login failures: but at least it worked.
> >>
> >> In general, this is how basic auth via http works. The client first
> >> attempts the request without any credentials, and receives a 401
> response.
> >>
> >> At this point, a browser would see the 401 response and display a popup
> >> asking for username/password.
> >>
> >> Then on future requests to that server in the current session, the
> >> browser sends the supplied credentials on every request to that server.
> >>
> >> If you are supplying credentials in the replication config, it should
> >> NOT follow that pattern ... the credentials should be always used.
> >>
> >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> >> with the following:
> >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Leader at: http://
> [REDACTED]/solr/sequence2_shard1_replica_n1
> >> is not available.
> >>
> >> The info above is valid for Solr running in standalone mode. But those
> >> core names indicate that you are running in SolrCloud mode.
> >>
> >> In cloud mode, Solr handles all replication. Don't attempt to actually
> >> configure the replication handler in cloud mode ... Solr will handle it
> >> all for you, and will even automatically take care of authenticating the
> >> requests. You don't need to configure the replication handler at all.
> >>
> >> If you are using the replication handler as a "back door" to copy
> >> indexes to a separate Solr install, a better option is to use the
> >> backup/restore functionality in the Collections API.
> >>
> >> Thanks,
> >> Shawn
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you recognize the sender and are
> >> confident the content is safe.
> >>
> >>
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
> >
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
Re: [EXTERNAL] security.json not working for V2 syntax
Posted by Eric Pugh <ep...@opensourceconnections.com>.
In https://github.com/apache/solr/pull/1875 I added a bats test for replication, and eventually closed it as “not generating enough value over unit tests”…. Would this test as written have caught your issues?
It would be interesting if you could write a bats test demonstrating the setup you have and the issues you ran into, and that might be worth resurrecting as “Here is a common deploy configuration and we should test it regularly”.
> On Sep 29, 2023, at 1:25 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID> wrote:
>
> Newer information:
>
> Indeed, getting Leader/Follower replication to work was my primary concern: and that now seems to have been solved.
>
> The reason so many different things were pulled into the thread was that there did not seem to be any way to know which details were relevant to whatever mystery was causing the Leader/Follower problem.
>
> I am still wondering about why security.json does not seem to work with V2 syntax (curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'), and I hope to find some time to start a new separate thread (as you suggest) specifically on that topic if you might be able to help: but indeed that was mostly included because I thought it might be a clue.
>
> Meanwhile, there were two things which were needed to get Leader/Follower replication to work between SolrClouds in Solr9.2 (as it had worked in Solr8.11 and earlier). One was to add -Dsolr.disable.allowUrls=true in the startup script (solr.log made it easy to learn that solution). The other was to add in the Zookeeper path for the Leader SolrCloud an entry in live_nodes corresponding to the Follower node ("zkcli.sh -cmd putfile /live_nodes/[host].[domain]:[port]_solr /dev/null"). Is there anywhere that this requirement was documented, and I just did not find it? Or is there somewhere that this should be added to the documentation?
>
> It would be nice to have an explanation of why this was necessary; but Leader/Follower replication now seems to be working for me in Solr9.2
>
> Thanks
>
> -----Original Message-----
> From: Jason Gerlowski <ge...@gmail.com>
> Sent: Monday, September 11, 2023 10:32 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: security.json not working for V2 syntax
>
> Hi Craig,
>
> To be honest, I'm having a little trouble following all of the
> messages in this thread.
>
> This is at least my mail client's fault, as copy/paste has left the
> thread looking very jumbled in GMail. But I think it's also harder
> for us all to understand and help because we've been using a single
> thread to describe a handful of loosely related issues: JVM
> SecurityManager "allowedPath" problems, some errors from the PKI
> plugin, some manual tests using Solr's v1 and v2 collection-reload
> APIs (which is esp. confusing given discussion of the v1 and v2 PKI
> auth headers), etc.
>
> I think I understand that the PKI errors are your primary concern at
> this point. Would you be willing to spin some of the other topics
> (e.g. the v2 reload-collection authz questions, which I'm very
> interested in) into their own mailing list thread, so they don't
> distract from the PKI questions here.
>
>> Could someone clarify whether it the case that
>
> AFAIK most of what you have here looks correct. The
> "-Dsolr.pki.sendVersion" sysprop looks like the correct way to specify
> those values, PKI Auth _should_ be agnostic of the actual API endpoint
> (and so shouldn't be "ignored" by leader/follower replication), etc.
> Bugs do creep in periodically where the client used in a particular
> part of Solr isn't set up to do PKI auth (and therefore) doesn't send
> either the v1 or v2 PKI header), so that's a possibility. But I
> haven't seen anything about that yet, and leader/follower is pretty
> heavily used as it's a building block for SolrCloud replication as
> well.
>
> Maybe someone would be able to spot something if you provided more
> information: how are you starting and running Solr, what does the
> relevant replicationHandler config look like in your solrconfig.xml,
> is there any interesting DEBUG logging for the PKIAuthenticationPlugin
> class on the sending or receiving node?
>
> Good luck and hope we can help!
>
> Jason
>
> On Thu, Sep 7, 2023 at 1:30 PM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>>
>> I am still hoping to hear some suggestion of how to implement solr.apache.org/guide/solr/latest/deployment-guide/authentication-and-authorization-plugins.html#pkiauthenticationplugin
>>
>> Could someone clarify whether it the case that
>> 1> the suggestion "setting the system properties solr.pki.sendVersion=v1 and solr.pki.acceptVersions=v1,v2" is accomplished by some means other than adding "-Dsolr.pki.sendVersion=v1 -Dsolr.pki.acceptVersions=v1,v2" to the invocation of ${solr_root}/bin/solr
>> OR
>> 2> Follower replication is known to ignore "solr.pki.sendVersion=v1"
>> OR
>> 3> Follower replication does ignore "solr.pki.sendVersion=v1" and I happen to have been the first person to have encountered this fact
>> OR
>> 4> Leader replication is known to ignore "solr.pki.acceptVersions=v1,v2"
>> OR
>> 5> Leader replication does ignore "solr.pki.acceptVersions=v1,v2" and I happen to have been the first person to have encountered this fact
>> OR
>> 6> the error message "o.a.s.s.PKIAuthenticationPlugin Could not load principal from SolrAuthV2 header." Is referring to something else
>>
>> Is there any evidence of anyone having gotten Leader/Follower replication to work with Solr9.2?
>>
>> -----Original Message-----
>> From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> Sent: Wednesday, August 30, 2023 2:25 PM
>> To: users@solr.apache.org
>> Subject: security.json not working for V2 syntax
>>
>> Having not seen any response, I thought I would see if it would help to ask the question again with a different subject
>>
>> -----Original Message-----
>> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
>> Sent: Wednesday, August 23, 2023 11:34 PM
>> To: users@solr.apache.org
>> Subject: RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
>>
>> We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
>>
>> In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
>>
>> The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
>>
>> As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
>>
>>
>> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":255},
>> "success":{
>> "nosqltest21.be-md:9852_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":176}},
>> "nosqltest22.be-md:9852_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":219}}}}
>> [11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":237},
>> "success":{
>> "nosqltest21.be-md:9852_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":176}},
>> "nosqltest22.be-md:9852_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":216}}}}
>> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
>> <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>> <title>Error 401 Authentication failed, Response code: 401</title>
>> </head>
>> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
>> <table>
>> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
>> <tr><th>STATUS:</th><td>401</td></tr>
>> <tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
>> <tr><th>SERVLET:</th><td>default</td></tr>
>> </table>
>>
>> </body>
>> </html>
>> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
>> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
>> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
>> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
>> [11:51 dbh19850s 1155]$ less solr.log
>> ...
>> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
>> "name":"openreload8",
>> "path":"/admin/collections",
>> "index":9,
>> "collection":null,
>> "role":null,
>> "params":{
>> "action":["REGEX:(?i)RELOAD"],
>> "name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
>> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
>> "name":"openreload8",
>> "path":"/admin/collections",
>> "index":9,
>> "collection":null,
>> "role":null,
>> "params":{
>> "action":["REGEX:(?i)RELOAD"],
>> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
>> ...
>> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
>> "name":"catch-all-nocollection",
>> "path":"/*",
>> "index":24,
>> "collection":null,
>> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
>> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
>> "name":"catch-all-nocollection",
>> "path":"/*",
>> "index":24,
>> "collection":null,
>> "role":"allgen"}] has role, but request principal cannot be identified; forbidding access
>>
>> -----Original Message-----
>> From: Kevin Risden <kr...@apache.org>
>> Sent: Monday, August 14, 2023 10:31 AM
>> To: users@solr.apache.org
>> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>>
>> The issue w/ allowPaths was reported here:
>> https://issues.apache.org/jira/browse/SOLR-16905
>>
>> Basically allowPaths was not being reflected in the security manager
>> policy. You can update security.policy yourself or disable security manager
>> as you found.
>>
>> Kevin Risden
>>
>>
>> On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
>> <cr...@nih.gov.invalid> wrote:
>>
>>> Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
>>> SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
>>> and incremental=true)
>>>
>>>
>>> -----Original Message-----
>>> From: Jan Høydahl <ja...@cominvent.com>
>>> Sent: Monday, August 14, 2023 3:12 AM
>>> To: users@solr.apache.org
>>> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>>>
>>> This is probably due to SecurityManager rejecting an unknown path. Try
>>> starting Solr with:
>>>
>>> -Dsolr.allowPaths=/netmnt/
>>>
>>> or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
>>>
>>> Then try again. Alternatively you can try disabling SecMgr withenv.var
>>> SOLR_SECURITY_MANAGER_ENABLED=false
>>>
>>>
>>> https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
>>>
>>> Jan
>>>
>>>> 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
>>> <cr...@nih.gov.INVALID>:
>>>>
>>>> Thanks: I was forgetting the fact that (in order to get backups in the
>>> format we expected) I added "incremental=false" to our backup script when
>>> we upgraded from Solr8.5 to Solr8.11.
>>>>
>>>> It trying to test this functionality, however, I now find that backups
>>> do not seem to be working for me *at all* in Solr9.2, with or without
>>> "incremental=false"
>>>>
>>>> I keep getting a stack trace such as:
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <response>
>>>>
>>>> <lst name="responseHeader">
>>>> <int name="status">500</int>
>>>> <int name="QTime">9</int>
>>>> </lst>
>>>> <lst name="error">
>>>> <str name="msg">access denied ("java.io.FilePermission"
>>> "/netmnt/[redacted-directory]" "read")</str>
>>>> <str name="trace">java.security.AccessControlException: access denied
>>> ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
>>>> at
>>> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
>>>> at
>>> java.base/java.security.AccessController.checkPermission(AccessController.java:897)
>>>> at
>>> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
>>>> at
>>> java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
>>>> at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
>>>> at
>>> java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
>>>> at java.base/java.nio.file.Files.exists(Files.java:2435)
>>>> at
>>> org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
>>>> at
>>> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
>>>> at
>>> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
>>>> at
>>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
>>>> at
>>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
>>>> at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
>>>> at
>>> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
>>>> at
>>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
>>>> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
>>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
>>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
>>>> at
>>> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
>>>> at
>>> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
>>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
>>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>>>> at
>>> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
>>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
>>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
>>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
>>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
>>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
>>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
>>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
>>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
>>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
>>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
>>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
>>>> at
>>> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
>>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
>>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
>>>> at
>>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
>>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
>>>> at
>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
>>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:563)
>>>> at
>>> org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
>>>> at
>>> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
>>>> at
>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
>>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
>>>> at org.eclipse.jetty.io
>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
>>>> at org.eclipse.jetty.io
>>> .FillInterest.fillable(FillInterest.java:100)
>>>> at org.eclipse.jetty.io
>>> .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
>>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
>>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>> </str>
>>>> <int name="code">500</int>
>>>> </lst>
>>>> </response>
>>>>
>>>>
>>>> It should be noted that the Linux account which runs Solr has read and
>>> write access to the specified location, which is a subdirectory of what is
>>> listed in allowPaths in all the solr.xml files of this SolrCloud, and which
>>> is mounted on all hosts of the SolrCloud.
>>>>
>>>> I have tried both V1 API and V2 API as found at
>>> solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
>>>>
>>>> Not being familiar with V2 API, I tried various permutations of
>>> "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
>>> the same stack trace
>>>>
>>>> Does anyone have any example of a successful backup from Solr9.2?
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Dominique Bejean <do...@eolya.fr>
>>>> Sent: Friday, August 11, 2023 9:32 AM
>>>> To: users@solr.apache.org
>>>> Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
>>> replication
>>>>
>>>> Hi Craig,
>>>>
>>>> Yes, starting with the 8.9 and 9.0 versions, Collection API allows
>>>> incremental backup and much more (corruption check, backup to Amazon S3
>>> or
>>>> Google Cloud Storage) .
>>>> Take a look at this umbrella JIRA
>>>> https://issues.apache.org/jira/browse/SOLR-15086
>>>>
>>>> Regards
>>>>
>>>> Dominique
>>>>
>>>>
>>>>
>>>> Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
>>>> <cr...@nih.gov.invalid> a écrit :
>>>>
>>>>> RE: a better option is to use the backup/restore functionality in the
>>>>> Collections API.
>>>>>
>>>>> My impression was that there is no facility for _incremental_ backup and
>>>>> retore in the collections API: is there? To do backup and scp and
>>> restore
>>>>> of terabytes of data every few minutes does not sound practical.
>>>>>
>>>>> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
>>> as
>>>>> well as Solr) is redundancy (for failover) in our main Data Center, and
>>>>> redundancy in read-only reasonably-concurrent copies in our second Data
>>>>> Center. For Solr8.11 (and earlier), we have had SolrCloud for
>>> redundancy in
>>>>> our main Data Center, and Leader/Follower replication to the read-only
>>>>> SolrClouds in the second Data Center. At one time, we were hoping that
>>> CDCR
>>>>> would work better: but we never managed to get CDCR to work reliably
>>> (and
>>>>> perhaps others also found it unreliable, leading to it being deprecated
>>>>> rather than being fixed). We have found that SolrCloud does not work
>>>>> reliably when spread across Data Centers, so Leader/Follower replication
>>>>> (formerly known as Master/Slave replication) has been the only way we
>>> have
>>>>> found to keep our (read-only) copies in the second Data Center only a
>>> few
>>>>> minutes behind the data in the main Data Center (the only way to provide
>>>>> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
>>> and
>>>>> MySQL). My supervisor was asking for clarification whether you are
>>> implying
>>>>> that Leader/Follower replication is being deprecated.
>>>>>
>>>>>
>>>>> In my continuing attempts to resolve these issues, I have come across a
>>>>> related question. The error message about SolrAuthV2 prompted me to
>>> wonder
>>>>> about the V1 and V2 syntax options (such as shown at
>>>>>
>>> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
>>> );
>>>>> and I was wondering whether Leader/Follower replication changed from
>>> using
>>>>> syntax V1 to using syntax V2, and if that might contribute to the
>>> breaking
>>>>> of permissions. As an experiment in our test environment, I setup a
>>>>> permission in security.json to allow RELOAD collection without a
>>> password.
>>>>> After confirming that my V2 syntax does work _with_ a password, I then
>>>>> attempted RELOAD collection without a password using both syntax V1 and
>>>>> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
>>> several
>>>>> permutations in security.json to allow RELOAD without password in syntax
>>>>> V2, but have not yet found a successful permutation. Are there any
>>>>> clarifications what security.json changes are needed for syntax V2? Can
>>> it
>>>>> be confirmed whether Leader/Follower replication is using V2 (in other
>>>>> words, whether that may be contributing to the permission problem)?
>>>>>
>>>>>
>>>>> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
>>>>> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
>>> 'Content-Type:
>>>>> application/json' -d '{"reload":{}}'
>>>>> {
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":255},
>>>>> "success":{
>>>>> "nosqltest21.be-md:9852_solr":{
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":176}},
>>>>> "nosqltest22.be-md:9852_solr":{
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":219}}}}
>>>>> [11:51 dbh19850s 1152]$ curl "http://localhost:
>>> $p/solr/admin/collections?action=RELOAD&name=helpdocs"
>>>>>
>>>>> {
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":237},
>>>>> "success":{
>>>>> "nosqltest21.be-md:9852_solr":{
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":176}},
>>>>> "nosqltest22.be-md:9852_solr":{
>>>>> "responseHeader":{
>>>>> "status":0,
>>>>> "QTime":216}}}}
>>>>> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
>>> $p/api/collections/helpdocs"
>>>>> -H 'Content-Type: application/json' -d '{"reload":{}}'
>>>>> <html>
>>>>> <head>
>>>>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>>>>> <title>Error 401 Authentication failed, Response code: 401</title>
>>>>> </head>
>>>>> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
>>>>> <table>
>>>>> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
>>>>> <tr><th>STATUS:</th><td>401</td></tr>
>>>>> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
>>>>> 401</td></tr>
>>>>> <tr><th>SERVLET:</th><td>default</td></tr>
>>>>> </table>
>>>>>
>>>>> </body>
>>>>> </html>
>>>>> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
>>>>>
>>>>> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
>>> /api/collections/helpdocs
>>>>> HTTP/1.1" 200 280
>>>>> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
>>>>> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
>>>>> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
>>> /api/collections/helpdocs
>>>>> HTTP/1.1" 401 491
>>>>> [11:51 dbh19850s 1155]$ less solr.log
>>>>> ...
>>>>> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
>>>>> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
>>>>> "name":"openreload8",
>>>>> "path":"/admin/collections",
>>>>> "index":9,
>>>>> "collection":null,
>>>>> "role":null,
>>>>> "params":{
>>>>> "action":["REGEX:(?i)RELOAD"],
>>>>> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
>>>>> [/admin/collections]
>>>>> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
>>>>> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
>>>>> "name":"openreload8",
>>>>> "path":"/admin/collections",
>>>>> "index":9,
>>>>> "collection":null,
>>>>> "role":null,
>>>>> "params":{
>>>>> "action":["REGEX:(?i)RELOAD"],
>>>>> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
>>>>> ...
>>>>> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
>>>>> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
>>>>> "name":"catch-all-nocollection",
>>>>> "path":"/*",
>>>>> "index":24,
>>>>> "collection":null,
>>>>> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
>>>>> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
>>>>> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
>>>>> "name":"catch-all-nocollection",
>>>>> "path":"/*",
>>>>> "index":24,
>>>>> "collection":null,
>>>>> "role":"allgen"}] has role, but request principal cannot be identified;
>>>>> forbidding access
>>>>>
>>>>>
>>>>> As a side note, in our experience, the only thing that has been
>>> cluttering
>>>>> up solr.log with attempts to connect without a password has been
>>>>> Leader/Follower replication (formerly known as Master/Slave
>>> replication).
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Shawn Heisey <ap...@elyograg.org>
>>>>> Sent: Saturday, August 5, 2023 4:24 PM
>>>>> To: users@solr.apache.org
>>>>> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
>>>>>
>>>>> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
>>>>>> We are having problems transitioning Leader/Follower replication to
>>>>> Solr9.2.1
>>>>>>
>>>>>> In Solr8.5 and below, what was then called Master/Slave replication had
>>>>> the annoying problem that, even though we specified httpBasicAuthUser
>>> and
>>>>> httpBasicAuthPassword, it would always attempt to connect first without
>>> a
>>>>> password before retrying with a password. This made solr.log noisy with
>>>>> lots of unnecessary login failures: but at least it worked.
>>>>>
>>>>> In general, this is how basic auth via http works. The client first
>>>>> attempts the request without any credentials, and receives a 401
>>> response.
>>>>>
>>>>> At this point, a browser would see the 401 response and display a popup
>>>>> asking for username/password.
>>>>>
>>>>> Then on future requests to that server in the current session, the
>>>>> browser sends the supplied credentials on every request to that server.
>>>>>
>>>>> If you are supplying credentials in the replication config, it should
>>>>> NOT follow that pattern ... the credentials should be always used.
>>>>>
>>>>>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
>>>>> with the following:
>>>>>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
>>>>> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
>>>>>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
>>>>> o.a.s.h.IndexFetcher Leader at: http://
>>> [REDACTED]/solr/sequence2_shard1_replica_n1
>>>>> is not available.
>>>>>
>>>>> The info above is valid for Solr running in standalone mode. But those
>>>>> core names indicate that you are running in SolrCloud mode.
>>>>>
>>>>> In cloud mode, Solr handles all replication. Don't attempt to actually
>>>>> configure the replication handler in cloud mode ... Solr will handle it
>>>>> all for you, and will even automatically take care of authenticating the
>>>>> requests. You don't need to configure the replication handler at all.
>>>>>
>>>>> If you are using the replication handler as a "back door" to copy
>>>>> indexes to a separate Solr install, a better option is to use the
>>>>> backup/restore functionality in the Collections API.
>>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>> CAUTION: This email originated from outside of the organization. Do not
>>>>> click links or open attachments unless you recognize the sender and are
>>>>> confident the content is safe.
>>>>>
>>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you recognize the sender and are
>>> confident the content is safe.
>>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you recognize the sender and are
>>> confident the content is safe.
>>>
>>>
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
>>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
>
_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
RE: [EXTERNAL] Re: security.json not working for V2 syntax
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
Newer information:
Indeed, getting Leader/Follower replication to work was my primary concern: and that now seems to have been solved.
The reason so many different things were pulled into the thread was that there did not seem to be any way to know which details were relevant to whatever mystery was causing the Leader/Follower problem.
I am still wondering about why security.json does not seem to work with V2 syntax (curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'), and I hope to find some time to start a new separate thread (as you suggest) specifically on that topic if you might be able to help: but indeed that was mostly included because I thought it might be a clue.
Meanwhile, there were two things which were needed to get Leader/Follower replication to work between SolrClouds in Solr9.2 (as it had worked in Solr8.11 and earlier). One was to add -Dsolr.disable.allowUrls=true in the startup script (solr.log made it easy to learn that solution). The other was to add in the Zookeeper path for the Leader SolrCloud an entry in live_nodes corresponding to the Follower node ("zkcli.sh -cmd putfile /live_nodes/[host].[domain]:[port]_solr /dev/null"). Is there anywhere that this requirement was documented, and I just did not find it? Or is there somewhere that this should be added to the documentation?
It would be nice to have an explanation of why this was necessary; but Leader/Follower replication now seems to be working for me in Solr9.2
Thanks
-----Original Message-----
From: Jason Gerlowski <ge...@gmail.com>
Sent: Monday, September 11, 2023 10:32 AM
To: users@solr.apache.org
Subject: [EXTERNAL] Re: security.json not working for V2 syntax
Hi Craig,
To be honest, I'm having a little trouble following all of the
messages in this thread.
This is at least my mail client's fault, as copy/paste has left the
thread looking very jumbled in GMail. But I think it's also harder
for us all to understand and help because we've been using a single
thread to describe a handful of loosely related issues: JVM
SecurityManager "allowedPath" problems, some errors from the PKI
plugin, some manual tests using Solr's v1 and v2 collection-reload
APIs (which is esp. confusing given discussion of the v1 and v2 PKI
auth headers), etc.
I think I understand that the PKI errors are your primary concern at
this point. Would you be willing to spin some of the other topics
(e.g. the v2 reload-collection authz questions, which I'm very
interested in) into their own mailing list thread, so they don't
distract from the PKI questions here.
> Could someone clarify whether it the case that
AFAIK most of what you have here looks correct. The
"-Dsolr.pki.sendVersion" sysprop looks like the correct way to specify
those values, PKI Auth _should_ be agnostic of the actual API endpoint
(and so shouldn't be "ignored" by leader/follower replication), etc.
Bugs do creep in periodically where the client used in a particular
part of Solr isn't set up to do PKI auth (and therefore) doesn't send
either the v1 or v2 PKI header), so that's a possibility. But I
haven't seen anything about that yet, and leader/follower is pretty
heavily used as it's a building block for SolrCloud replication as
well.
Maybe someone would be able to spot something if you provided more
information: how are you starting and running Solr, what does the
relevant replicationHandler config look like in your solrconfig.xml,
is there any interesting DEBUG logging for the PKIAuthenticationPlugin
class on the sending or receiving node?
Good luck and hope we can help!
Jason
On Thu, Sep 7, 2023 at 1:30 PM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
>
> I am still hoping to hear some suggestion of how to implement solr.apache.org/guide/solr/latest/deployment-guide/authentication-and-authorization-plugins.html#pkiauthenticationplugin
>
> Could someone clarify whether it the case that
> 1> the suggestion "setting the system properties solr.pki.sendVersion=v1 and solr.pki.acceptVersions=v1,v2" is accomplished by some means other than adding "-Dsolr.pki.sendVersion=v1 -Dsolr.pki.acceptVersions=v1,v2" to the invocation of ${solr_root}/bin/solr
> OR
> 2> Follower replication is known to ignore "solr.pki.sendVersion=v1"
> OR
> 3> Follower replication does ignore "solr.pki.sendVersion=v1" and I happen to have been the first person to have encountered this fact
> OR
> 4> Leader replication is known to ignore "solr.pki.acceptVersions=v1,v2"
> OR
> 5> Leader replication does ignore "solr.pki.acceptVersions=v1,v2" and I happen to have been the first person to have encountered this fact
> OR
> 6> the error message "o.a.s.s.PKIAuthenticationPlugin Could not load principal from SolrAuthV2 header." Is referring to something else
>
> Is there any evidence of anyone having gotten Leader/Follower replication to work with Solr9.2?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Wednesday, August 30, 2023 2:25 PM
> To: users@solr.apache.org
> Subject: security.json not working for V2 syntax
>
> Having not seen any response, I thought I would see if it would help to ask the question again with a different subject
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
> Sent: Wednesday, August 23, 2023 11:34 PM
> To: users@solr.apache.org
> Subject: RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
>
> We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
>
> In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
>
> The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
>
> As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
>
>
> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
> {
> "responseHeader":{
> "status":0,
> "QTime":255},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":219}}}}
> [11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
> {
> "responseHeader":{
> "status":0,
> "QTime":237},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":216}}}}
> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 401 Authentication failed, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> <table>
> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
> [11:51 dbh19850s 1155]$ less solr.log
> ...
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> ...
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] has role, but request principal cannot be identified; forbidding access
>
> -----Original Message-----
> From: Kevin Risden <kr...@apache.org>
> Sent: Monday, August 14, 2023 10:31 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>
> The issue w/ allowPaths was reported here:
> https://issues.apache.org/jira/browse/SOLR-16905
>
> Basically allowPaths was not being reflected in the security manager
> policy. You can update security.policy yourself or disable security manager
> as you found.
>
> Kevin Risden
>
>
> On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
> > SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
> > and incremental=true)
> >
> >
> > -----Original Message-----
> > From: Jan Høydahl <ja...@cominvent.com>
> > Sent: Monday, August 14, 2023 3:12 AM
> > To: users@solr.apache.org
> > Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
> >
> > This is probably due to SecurityManager rejecting an unknown path. Try
> > starting Solr with:
> >
> > -Dsolr.allowPaths=/netmnt/
> >
> > or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
> >
> > Then try again. Alternatively you can try disabling SecMgr withenv.var
> > SOLR_SECURITY_MANAGER_ENABLED=false
> >
> >
> > https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
> >
> > Jan
> >
> > > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.INVALID>:
> > >
> > > Thanks: I was forgetting the fact that (in order to get backups in the
> > format we expected) I added "incremental=false" to our backup script when
> > we upgraded from Solr8.5 to Solr8.11.
> > >
> > > It trying to test this functionality, however, I now find that backups
> > do not seem to be working for me *at all* in Solr9.2, with or without
> > "incremental=false"
> > >
> > > I keep getting a stack trace such as:
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <response>
> > >
> > > <lst name="responseHeader">
> > > <int name="status">500</int>
> > > <int name="QTime">9</int>
> > > </lst>
> > > <lst name="error">
> > > <str name="msg">access denied ("java.io.FilePermission"
> > "/netmnt/[redacted-directory]" "read")</str>
> > > <str name="trace">java.security.AccessControlException: access denied
> > ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
> > > at
> > java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > > at
> > java.base/java.security.AccessController.checkPermission(AccessController.java:897)
> > > at
> > java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
> > > at
> > java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
> > > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
> > > at
> > java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
> > > at java.base/java.nio.file.Files.exists(Files.java:2435)
> > > at
> > org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
> > > at
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
> > > at
> > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> > > at
> > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> > > at
> > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > > at
> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > > at
> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > > at
> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > > at
> > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> > > at
> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > > at
> > org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > > at
> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:100)
> > > at org.eclipse.jetty.io
> > .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > > at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
> > > at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > </str>
> > > <int name="code">500</int>
> > > </lst>
> > > </response>
> > >
> > >
> > > It should be noted that the Linux account which runs Solr has read and
> > write access to the specified location, which is a subdirectory of what is
> > listed in allowPaths in all the solr.xml files of this SolrCloud, and which
> > is mounted on all hosts of the SolrCloud.
> > >
> > > I have tried both V1 API and V2 API as found at
> > solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
> > >
> > > Not being familiar with V2 API, I tried various permutations of
> > "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
> > the same stack trace
> > >
> > > Does anyone have any example of a successful backup from Solr9.2?
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dominique Bejean <do...@eolya.fr>
> > > Sent: Friday, August 11, 2023 9:32 AM
> > > To: users@solr.apache.org
> > > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
> > replication
> > >
> > > Hi Craig,
> > >
> > > Yes, starting with the 8.9 and 9.0 versions, Collection API allows
> > > incremental backup and much more (corruption check, backup to Amazon S3
> > or
> > > Google Cloud Storage) .
> > > Take a look at this umbrella JIRA
> > > https://issues.apache.org/jira/browse/SOLR-15086
> > >
> > > Regards
> > >
> > > Dominique
> > >
> > >
> > >
> > > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
> > > <cr...@nih.gov.invalid> a écrit :
> > >
> > >> RE: a better option is to use the backup/restore functionality in the
> > >> Collections API.
> > >>
> > >> My impression was that there is no facility for _incremental_ backup and
> > >> retore in the collections API: is there? To do backup and scp and
> > restore
> > >> of terabytes of data every few minutes does not sound practical.
> > >>
> > >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
> > as
> > >> well as Solr) is redundancy (for failover) in our main Data Center, and
> > >> redundancy in read-only reasonably-concurrent copies in our second Data
> > >> Center. For Solr8.11 (and earlier), we have had SolrCloud for
> > redundancy in
> > >> our main Data Center, and Leader/Follower replication to the read-only
> > >> SolrClouds in the second Data Center. At one time, we were hoping that
> > CDCR
> > >> would work better: but we never managed to get CDCR to work reliably
> > (and
> > >> perhaps others also found it unreliable, leading to it being deprecated
> > >> rather than being fixed). We have found that SolrCloud does not work
> > >> reliably when spread across Data Centers, so Leader/Follower replication
> > >> (formerly known as Master/Slave replication) has been the only way we
> > have
> > >> found to keep our (read-only) copies in the second Data Center only a
> > few
> > >> minutes behind the data in the main Data Center (the only way to provide
> > >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
> > and
> > >> MySQL). My supervisor was asking for clarification whether you are
> > implying
> > >> that Leader/Follower replication is being deprecated.
> > >>
> > >>
> > >> In my continuing attempts to resolve these issues, I have come across a
> > >> related question. The error message about SolrAuthV2 prompted me to
> > wonder
> > >> about the V1 and V2 syntax options (such as shown at
> > >>
> > https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
> > );
> > >> and I was wondering whether Leader/Follower replication changed from
> > using
> > >> syntax V1 to using syntax V2, and if that might contribute to the
> > breaking
> > >> of permissions. As an experiment in our test environment, I setup a
> > >> permission in security.json to allow RELOAD collection without a
> > password.
> > >> After confirming that my V2 syntax does work _with_ a password, I then
> > >> attempted RELOAD collection without a password using both syntax V1 and
> > >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
> > several
> > >> permutations in security.json to allow RELOAD without password in syntax
> > >> V2, but have not yet found a successful permutation. Are there any
> > >> clarifications what security.json changes are needed for syntax V2? Can
> > it
> > >> be confirmed whether Leader/Follower replication is using V2 (in other
> > >> words, whether that may be contributing to the permission problem)?
> > >>
> > >>
> > >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> > >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
> > 'Content-Type:
> > >> application/json' -d '{"reload":{}}'
> > >> {
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":255},
> > >> "success":{
> > >> "nosqltest21.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":176}},
> > >> "nosqltest22.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":219}}}}
> > >> [11:51 dbh19850s 1152]$ curl "http://localhost:
> > $p/solr/admin/collections?action=RELOAD&name=helpdocs"
> > >>
> > >> {
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":237},
> > >> "success":{
> > >> "nosqltest21.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":176}},
> > >> "nosqltest22.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":216}}}}
> > >> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
> > $p/api/collections/helpdocs"
> > >> -H 'Content-Type: application/json' -d '{"reload":{}}'
> > >> <html>
> > >> <head>
> > >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> > >> <title>Error 401 Authentication failed, Response code: 401</title>
> > >> </head>
> > >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> > >> <table>
> > >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> > >> <tr><th>STATUS:</th><td>401</td></tr>
> > >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> > >> 401</td></tr>
> > >> <tr><th>SERVLET:</th><td>default</td></tr>
> > >> </table>
> > >>
> > >> </body>
> > >> </html>
> > >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> > >>
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
> > /api/collections/helpdocs
> > >> HTTP/1.1" 200 280
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> > >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
> > /api/collections/helpdocs
> > >> HTTP/1.1" 401 491
> > >> [11:51 dbh19850s 1155]$ less solr.log
> > >> ...
> > >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> > >> "name":"openreload8",
> > >> "path":"/admin/collections",
> > >> "index":9,
> > >> "collection":null,
> > >> "role":null,
> > >> "params":{
> > >> "action":["REGEX:(?i)RELOAD"],
> > >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> > >> [/admin/collections]
> > >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> > >> "name":"openreload8",
> > >> "path":"/admin/collections",
> > >> "index":9,
> > >> "collection":null,
> > >> "role":null,
> > >> "params":{
> > >> "action":["REGEX:(?i)RELOAD"],
> > >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> > >> ...
> > >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> > >> "name":"catch-all-nocollection",
> > >> "path":"/*",
> > >> "index":24,
> > >> "collection":null,
> > >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> > >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> > >> "name":"catch-all-nocollection",
> > >> "path":"/*",
> > >> "index":24,
> > >> "collection":null,
> > >> "role":"allgen"}] has role, but request principal cannot be identified;
> > >> forbidding access
> > >>
> > >>
> > >> As a side note, in our experience, the only thing that has been
> > cluttering
> > >> up solr.log with attempts to connect without a password has been
> > >> Leader/Follower replication (formerly known as Master/Slave
> > replication).
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Shawn Heisey <ap...@elyograg.org>
> > >> Sent: Saturday, August 5, 2023 4:24 PM
> > >> To: users@solr.apache.org
> > >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
> > >>
> > >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> > >>> We are having problems transitioning Leader/Follower replication to
> > >> Solr9.2.1
> > >>>
> > >>> In Solr8.5 and below, what was then called Master/Slave replication had
> > >> the annoying problem that, even though we specified httpBasicAuthUser
> > and
> > >> httpBasicAuthPassword, it would always attempt to connect first without
> > a
> > >> password before retrying with a password. This made solr.log noisy with
> > >> lots of unnecessary login failures: but at least it worked.
> > >>
> > >> In general, this is how basic auth via http works. The client first
> > >> attempts the request without any credentials, and receives a 401
> > response.
> > >>
> > >> At this point, a browser would see the 401 response and display a popup
> > >> asking for username/password.
> > >>
> > >> Then on future requests to that server in the current session, the
> > >> browser sends the supplied credentials on every request to that server.
> > >>
> > >> If you are supplying credentials in the replication config, it should
> > >> NOT follow that pattern ... the credentials should be always used.
> > >>
> > >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> > >> with the following:
> > >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> > >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> > >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> > >> o.a.s.h.IndexFetcher Leader at: http://
> > [REDACTED]/solr/sequence2_shard1_replica_n1
> > >> is not available.
> > >>
> > >> The info above is valid for Solr running in standalone mode. But those
> > >> core names indicate that you are running in SolrCloud mode.
> > >>
> > >> In cloud mode, Solr handles all replication. Don't attempt to actually
> > >> configure the replication handler in cloud mode ... Solr will handle it
> > >> all for you, and will even automatically take care of authenticating the
> > >> requests. You don't need to configure the replication handler at all.
> > >>
> > >> If you are using the replication handler as a "back door" to copy
> > >> indexes to a separate Solr install, a better option is to use the
> > >> backup/restore functionality in the Collections API.
> > >>
> > >> Thanks,
> > >> Shawn
> > >> CAUTION: This email originated from outside of the organization. Do not
> > >> click links or open attachments unless you recognize the sender and are
> > >> confident the content is safe.
> > >>
> > >>
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> > >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> >
> >
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
Re: security.json not working for V2 syntax
Posted by Jason Gerlowski <ge...@gmail.com>.
Hi Craig,
To be honest, I'm having a little trouble following all of the
messages in this thread.
This is at least my mail client's fault, as copy/paste has left the
thread looking very jumbled in GMail. But I think it's also harder
for us all to understand and help because we've been using a single
thread to describe a handful of loosely related issues: JVM
SecurityManager "allowedPath" problems, some errors from the PKI
plugin, some manual tests using Solr's v1 and v2 collection-reload
APIs (which is esp. confusing given discussion of the v1 and v2 PKI
auth headers), etc.
I think I understand that the PKI errors are your primary concern at
this point. Would you be willing to spin some of the other topics
(e.g. the v2 reload-collection authz questions, which I'm very
interested in) into their own mailing list thread, so they don't
distract from the PKI questions here.
> Could someone clarify whether it the case that
AFAIK most of what you have here looks correct. The
"-Dsolr.pki.sendVersion" sysprop looks like the correct way to specify
those values, PKI Auth _should_ be agnostic of the actual API endpoint
(and so shouldn't be "ignored" by leader/follower replication), etc.
Bugs do creep in periodically where the client used in a particular
part of Solr isn't set up to do PKI auth (and therefore) doesn't send
either the v1 or v2 PKI header), so that's a possibility. But I
haven't seen anything about that yet, and leader/follower is pretty
heavily used as it's a building block for SolrCloud replication as
well.
Maybe someone would be able to spot something if you provided more
information: how are you starting and running Solr, what does the
relevant replicationHandler config look like in your solrconfig.xml,
is there any interesting DEBUG logging for the PKIAuthenticationPlugin
class on the sending or receiving node?
Good luck and hope we can help!
Jason
On Thu, Sep 7, 2023 at 1:30 PM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
>
> I am still hoping to hear some suggestion of how to implement solr.apache.org/guide/solr/latest/deployment-guide/authentication-and-authorization-plugins.html#pkiauthenticationplugin
>
> Could someone clarify whether it the case that
> 1> the suggestion "setting the system properties solr.pki.sendVersion=v1 and solr.pki.acceptVersions=v1,v2" is accomplished by some means other than adding "-Dsolr.pki.sendVersion=v1 -Dsolr.pki.acceptVersions=v1,v2" to the invocation of ${solr_root}/bin/solr
> OR
> 2> Follower replication is known to ignore "solr.pki.sendVersion=v1"
> OR
> 3> Follower replication does ignore "solr.pki.sendVersion=v1" and I happen to have been the first person to have encountered this fact
> OR
> 4> Leader replication is known to ignore "solr.pki.acceptVersions=v1,v2"
> OR
> 5> Leader replication does ignore "solr.pki.acceptVersions=v1,v2" and I happen to have been the first person to have encountered this fact
> OR
> 6> the error message "o.a.s.s.PKIAuthenticationPlugin Could not load principal from SolrAuthV2 header." Is referring to something else
>
> Is there any evidence of anyone having gotten Leader/Follower replication to work with Solr9.2?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Wednesday, August 30, 2023 2:25 PM
> To: users@solr.apache.org
> Subject: security.json not working for V2 syntax
>
> Having not seen any response, I thought I would see if it would help to ask the question again with a different subject
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
> Sent: Wednesday, August 23, 2023 11:34 PM
> To: users@solr.apache.org
> Subject: RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
>
> We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
>
> In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
>
> The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
>
> As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
>
>
> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
> {
> "responseHeader":{
> "status":0,
> "QTime":255},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":219}}}}
> [11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
> {
> "responseHeader":{
> "status":0,
> "QTime":237},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":216}}}}
> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 401 Authentication failed, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> <table>
> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
> [11:51 dbh19850s 1155]$ less solr.log
> ...
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> ...
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] has role, but request principal cannot be identified; forbidding access
>
> -----Original Message-----
> From: Kevin Risden <kr...@apache.org>
> Sent: Monday, August 14, 2023 10:31 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>
> The issue w/ allowPaths was reported here:
> https://issues.apache.org/jira/browse/SOLR-16905
>
> Basically allowPaths was not being reflected in the security manager
> policy. You can update security.policy yourself or disable security manager
> as you found.
>
> Kevin Risden
>
>
> On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
> > SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
> > and incremental=true)
> >
> >
> > -----Original Message-----
> > From: Jan Høydahl <ja...@cominvent.com>
> > Sent: Monday, August 14, 2023 3:12 AM
> > To: users@solr.apache.org
> > Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
> >
> > This is probably due to SecurityManager rejecting an unknown path. Try
> > starting Solr with:
> >
> > -Dsolr.allowPaths=/netmnt/
> >
> > or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
> >
> > Then try again. Alternatively you can try disabling SecMgr withenv.var
> > SOLR_SECURITY_MANAGER_ENABLED=false
> >
> >
> > https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
> >
> > Jan
> >
> > > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.INVALID>:
> > >
> > > Thanks: I was forgetting the fact that (in order to get backups in the
> > format we expected) I added "incremental=false" to our backup script when
> > we upgraded from Solr8.5 to Solr8.11.
> > >
> > > It trying to test this functionality, however, I now find that backups
> > do not seem to be working for me *at all* in Solr9.2, with or without
> > "incremental=false"
> > >
> > > I keep getting a stack trace such as:
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <response>
> > >
> > > <lst name="responseHeader">
> > > <int name="status">500</int>
> > > <int name="QTime">9</int>
> > > </lst>
> > > <lst name="error">
> > > <str name="msg">access denied ("java.io.FilePermission"
> > "/netmnt/[redacted-directory]" "read")</str>
> > > <str name="trace">java.security.AccessControlException: access denied
> > ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
> > > at
> > java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > > at
> > java.base/java.security.AccessController.checkPermission(AccessController.java:897)
> > > at
> > java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
> > > at
> > java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
> > > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
> > > at
> > java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
> > > at java.base/java.nio.file.Files.exists(Files.java:2435)
> > > at
> > org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
> > > at
> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
> > > at
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
> > > at
> > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> > > at
> > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
> > > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> > > at
> > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > > at
> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > > at
> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > > at
> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
> > > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > > at
> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > > at
> > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> > > at
> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at
> > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> > > at
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > > at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > > at
> > org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > > at
> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:100)
> > > at org.eclipse.jetty.io
> > .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > > at
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
> > > at
> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > </str>
> > > <int name="code">500</int>
> > > </lst>
> > > </response>
> > >
> > >
> > > It should be noted that the Linux account which runs Solr has read and
> > write access to the specified location, which is a subdirectory of what is
> > listed in allowPaths in all the solr.xml files of this SolrCloud, and which
> > is mounted on all hosts of the SolrCloud.
> > >
> > > I have tried both V1 API and V2 API as found at
> > solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
> > >
> > > Not being familiar with V2 API, I tried various permutations of
> > "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
> > the same stack trace
> > >
> > > Does anyone have any example of a successful backup from Solr9.2?
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dominique Bejean <do...@eolya.fr>
> > > Sent: Friday, August 11, 2023 9:32 AM
> > > To: users@solr.apache.org
> > > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
> > replication
> > >
> > > Hi Craig,
> > >
> > > Yes, starting with the 8.9 and 9.0 versions, Collection API allows
> > > incremental backup and much more (corruption check, backup to Amazon S3
> > or
> > > Google Cloud Storage) .
> > > Take a look at this umbrella JIRA
> > > https://issues.apache.org/jira/browse/SOLR-15086
> > >
> > > Regards
> > >
> > > Dominique
> > >
> > >
> > >
> > > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
> > > <cr...@nih.gov.invalid> a écrit :
> > >
> > >> RE: a better option is to use the backup/restore functionality in the
> > >> Collections API.
> > >>
> > >> My impression was that there is no facility for _incremental_ backup and
> > >> retore in the collections API: is there? To do backup and scp and
> > restore
> > >> of terabytes of data every few minutes does not sound practical.
> > >>
> > >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
> > as
> > >> well as Solr) is redundancy (for failover) in our main Data Center, and
> > >> redundancy in read-only reasonably-concurrent copies in our second Data
> > >> Center. For Solr8.11 (and earlier), we have had SolrCloud for
> > redundancy in
> > >> our main Data Center, and Leader/Follower replication to the read-only
> > >> SolrClouds in the second Data Center. At one time, we were hoping that
> > CDCR
> > >> would work better: but we never managed to get CDCR to work reliably
> > (and
> > >> perhaps others also found it unreliable, leading to it being deprecated
> > >> rather than being fixed). We have found that SolrCloud does not work
> > >> reliably when spread across Data Centers, so Leader/Follower replication
> > >> (formerly known as Master/Slave replication) has been the only way we
> > have
> > >> found to keep our (read-only) copies in the second Data Center only a
> > few
> > >> minutes behind the data in the main Data Center (the only way to provide
> > >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
> > and
> > >> MySQL). My supervisor was asking for clarification whether you are
> > implying
> > >> that Leader/Follower replication is being deprecated.
> > >>
> > >>
> > >> In my continuing attempts to resolve these issues, I have come across a
> > >> related question. The error message about SolrAuthV2 prompted me to
> > wonder
> > >> about the V1 and V2 syntax options (such as shown at
> > >>
> > https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
> > );
> > >> and I was wondering whether Leader/Follower replication changed from
> > using
> > >> syntax V1 to using syntax V2, and if that might contribute to the
> > breaking
> > >> of permissions. As an experiment in our test environment, I setup a
> > >> permission in security.json to allow RELOAD collection without a
> > password.
> > >> After confirming that my V2 syntax does work _with_ a password, I then
> > >> attempted RELOAD collection without a password using both syntax V1 and
> > >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
> > several
> > >> permutations in security.json to allow RELOAD without password in syntax
> > >> V2, but have not yet found a successful permutation. Are there any
> > >> clarifications what security.json changes are needed for syntax V2? Can
> > it
> > >> be confirmed whether Leader/Follower replication is using V2 (in other
> > >> words, whether that may be contributing to the permission problem)?
> > >>
> > >>
> > >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> > >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
> > 'Content-Type:
> > >> application/json' -d '{"reload":{}}'
> > >> {
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":255},
> > >> "success":{
> > >> "nosqltest21.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":176}},
> > >> "nosqltest22.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":219}}}}
> > >> [11:51 dbh19850s 1152]$ curl "http://localhost:
> > $p/solr/admin/collections?action=RELOAD&name=helpdocs"
> > >>
> > >> {
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":237},
> > >> "success":{
> > >> "nosqltest21.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":176}},
> > >> "nosqltest22.be-md:9852_solr":{
> > >> "responseHeader":{
> > >> "status":0,
> > >> "QTime":216}}}}
> > >> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
> > $p/api/collections/helpdocs"
> > >> -H 'Content-Type: application/json' -d '{"reload":{}}'
> > >> <html>
> > >> <head>
> > >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> > >> <title>Error 401 Authentication failed, Response code: 401</title>
> > >> </head>
> > >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> > >> <table>
> > >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> > >> <tr><th>STATUS:</th><td>401</td></tr>
> > >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> > >> 401</td></tr>
> > >> <tr><th>SERVLET:</th><td>default</td></tr>
> > >> </table>
> > >>
> > >> </body>
> > >> </html>
> > >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> > >>
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
> > /api/collections/helpdocs
> > >> HTTP/1.1" 200 280
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> > >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> > >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
> > /api/collections/helpdocs
> > >> HTTP/1.1" 401 491
> > >> [11:51 dbh19850s 1155]$ less solr.log
> > >> ...
> > >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> > >> "name":"openreload8",
> > >> "path":"/admin/collections",
> > >> "index":9,
> > >> "collection":null,
> > >> "role":null,
> > >> "params":{
> > >> "action":["REGEX:(?i)RELOAD"],
> > >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> > >> [/admin/collections]
> > >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> > >> "name":"openreload8",
> > >> "path":"/admin/collections",
> > >> "index":9,
> > >> "collection":null,
> > >> "role":null,
> > >> "params":{
> > >> "action":["REGEX:(?i)RELOAD"],
> > >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> > >> ...
> > >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> > >> "name":"catch-all-nocollection",
> > >> "path":"/*",
> > >> "index":24,
> > >> "collection":null,
> > >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> > >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> > >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> > >> "name":"catch-all-nocollection",
> > >> "path":"/*",
> > >> "index":24,
> > >> "collection":null,
> > >> "role":"allgen"}] has role, but request principal cannot be identified;
> > >> forbidding access
> > >>
> > >>
> > >> As a side note, in our experience, the only thing that has been
> > cluttering
> > >> up solr.log with attempts to connect without a password has been
> > >> Leader/Follower replication (formerly known as Master/Slave
> > replication).
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Shawn Heisey <ap...@elyograg.org>
> > >> Sent: Saturday, August 5, 2023 4:24 PM
> > >> To: users@solr.apache.org
> > >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
> > >>
> > >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> > >>> We are having problems transitioning Leader/Follower replication to
> > >> Solr9.2.1
> > >>>
> > >>> In Solr8.5 and below, what was then called Master/Slave replication had
> > >> the annoying problem that, even though we specified httpBasicAuthUser
> > and
> > >> httpBasicAuthPassword, it would always attempt to connect first without
> > a
> > >> password before retrying with a password. This made solr.log noisy with
> > >> lots of unnecessary login failures: but at least it worked.
> > >>
> > >> In general, this is how basic auth via http works. The client first
> > >> attempts the request without any credentials, and receives a 401
> > response.
> > >>
> > >> At this point, a browser would see the 401 response and display a popup
> > >> asking for username/password.
> > >>
> > >> Then on future requests to that server in the current session, the
> > >> browser sends the supplied credentials on every request to that server.
> > >>
> > >> If you are supplying credentials in the replication config, it should
> > >> NOT follow that pattern ... the credentials should be always used.
> > >>
> > >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> > >> with the following:
> > >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> > >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> > >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> > >> o.a.s.h.IndexFetcher Leader at: http://
> > [REDACTED]/solr/sequence2_shard1_replica_n1
> > >> is not available.
> > >>
> > >> The info above is valid for Solr running in standalone mode. But those
> > >> core names indicate that you are running in SolrCloud mode.
> > >>
> > >> In cloud mode, Solr handles all replication. Don't attempt to actually
> > >> configure the replication handler in cloud mode ... Solr will handle it
> > >> all for you, and will even automatically take care of authenticating the
> > >> requests. You don't need to configure the replication handler at all.
> > >>
> > >> If you are using the replication handler as a "back door" to copy
> > >> indexes to a separate Solr install, a better option is to use the
> > >> backup/restore functionality in the Collections API.
> > >>
> > >> Thanks,
> > >> Shawn
> > >> CAUTION: This email originated from outside of the organization. Do not
> > >> click links or open attachments unless you recognize the sender and are
> > >> confident the content is safe.
> > >>
> > >>
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> > >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> >
> >
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
>
RE: security.json not working for V2 syntax
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
I am still hoping to hear some suggestion of how to implement solr.apache.org/guide/solr/latest/deployment-guide/authentication-and-authorization-plugins.html#pkiauthenticationplugin
Could someone clarify whether it the case that
1> the suggestion "setting the system properties solr.pki.sendVersion=v1 and solr.pki.acceptVersions=v1,v2" is accomplished by some means other than adding "-Dsolr.pki.sendVersion=v1 -Dsolr.pki.acceptVersions=v1,v2" to the invocation of ${solr_root}/bin/solr
OR
2> Follower replication is known to ignore "solr.pki.sendVersion=v1"
OR
3> Follower replication does ignore "solr.pki.sendVersion=v1" and I happen to have been the first person to have encountered this fact
OR
4> Leader replication is known to ignore "solr.pki.acceptVersions=v1,v2"
OR
5> Leader replication does ignore "solr.pki.acceptVersions=v1,v2" and I happen to have been the first person to have encountered this fact
OR
6> the error message "o.a.s.s.PKIAuthenticationPlugin Could not load principal from SolrAuthV2 header." Is referring to something else
Is there any evidence of anyone having gotten Leader/Follower replication to work with Solr9.2?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Wednesday, August 30, 2023 2:25 PM
To: users@solr.apache.org
Subject: security.json not working for V2 syntax
Having not seen any response, I thought I would see if it would help to ask the question again with a different subject
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
Sent: Wednesday, August 23, 2023 11:34 PM
To: users@solr.apache.org
Subject: RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
[11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
{
"responseHeader":{
"status":0,
"QTime":255},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":219}}}}
[11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
{
"responseHeader":{
"status":0,
"QTime":237},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":216}}}}
[11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Authentication failed, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
<table>
<tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
*[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
[11:51 dbh19850s 1155]$ less solr.log
...
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
...
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] has role, but request principal cannot be identified; forbidding access
-----Original Message-----
From: Kevin Risden <kr...@apache.org>
Sent: Monday, August 14, 2023 10:31 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
The issue w/ allowPaths was reported here:
https://issues.apache.org/jira/browse/SOLR-16905
Basically allowPaths was not being reflected in the security manager
policy. You can update security.policy yourself or disable security manager
as you found.
Kevin Risden
On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
> SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
> and incremental=true)
>
>
> -----Original Message-----
> From: Jan Høydahl <ja...@cominvent.com>
> Sent: Monday, August 14, 2023 3:12 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>
> This is probably due to SecurityManager rejecting an unknown path. Try
> starting Solr with:
>
> -Dsolr.allowPaths=/netmnt/
>
> or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
>
> Then try again. Alternatively you can try disabling SecMgr withenv.var
> SOLR_SECURITY_MANAGER_ENABLED=false
>
>
> https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
>
> Jan
>
> > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.INVALID>:
> >
> > Thanks: I was forgetting the fact that (in order to get backups in the
> format we expected) I added "incremental=false" to our backup script when
> we upgraded from Solr8.5 to Solr8.11.
> >
> > It trying to test this functionality, however, I now find that backups
> do not seem to be working for me *at all* in Solr9.2, with or without
> "incremental=false"
> >
> > I keep getting a stack trace such as:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <response>
> >
> > <lst name="responseHeader">
> > <int name="status">500</int>
> > <int name="QTime">9</int>
> > </lst>
> > <lst name="error">
> > <str name="msg">access denied ("java.io.FilePermission"
> "/netmnt/[redacted-directory]" "read")</str>
> > <str name="trace">java.security.AccessControlException: access denied
> ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
> > at
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > at
> java.base/java.security.AccessController.checkPermission(AccessController.java:897)
> > at
> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
> > at
> java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
> > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
> > at
> java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
> > at java.base/java.nio.file.Files.exists(Files.java:2435)
> > at
> org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
> > at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
> > at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> > at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> > at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> > at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:100)
> > at org.eclipse.jetty.io
> .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > </str>
> > <int name="code">500</int>
> > </lst>
> > </response>
> >
> >
> > It should be noted that the Linux account which runs Solr has read and
> write access to the specified location, which is a subdirectory of what is
> listed in allowPaths in all the solr.xml files of this SolrCloud, and which
> is mounted on all hosts of the SolrCloud.
> >
> > I have tried both V1 API and V2 API as found at
> solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
> >
> > Not being familiar with V2 API, I tried various permutations of
> "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
> the same stack trace
> >
> > Does anyone have any example of a successful backup from Solr9.2?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Dominique Bejean <do...@eolya.fr>
> > Sent: Friday, August 11, 2023 9:32 AM
> > To: users@solr.apache.org
> > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
> replication
> >
> > Hi Craig,
> >
> > Yes, starting with the 8.9 and 9.0 versions, Collection API allows
> > incremental backup and much more (corruption check, backup to Amazon S3
> or
> > Google Cloud Storage) .
> > Take a look at this umbrella JIRA
> > https://issues.apache.org/jira/browse/SOLR-15086
> >
> > Regards
> >
> > Dominique
> >
> >
> >
> > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.invalid> a écrit :
> >
> >> RE: a better option is to use the backup/restore functionality in the
> >> Collections API.
> >>
> >> My impression was that there is no facility for _incremental_ backup and
> >> retore in the collections API: is there? To do backup and scp and
> restore
> >> of terabytes of data every few minutes does not sound practical.
> >>
> >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
> as
> >> well as Solr) is redundancy (for failover) in our main Data Center, and
> >> redundancy in read-only reasonably-concurrent copies in our second Data
> >> Center. For Solr8.11 (and earlier), we have had SolrCloud for
> redundancy in
> >> our main Data Center, and Leader/Follower replication to the read-only
> >> SolrClouds in the second Data Center. At one time, we were hoping that
> CDCR
> >> would work better: but we never managed to get CDCR to work reliably
> (and
> >> perhaps others also found it unreliable, leading to it being deprecated
> >> rather than being fixed). We have found that SolrCloud does not work
> >> reliably when spread across Data Centers, so Leader/Follower replication
> >> (formerly known as Master/Slave replication) has been the only way we
> have
> >> found to keep our (read-only) copies in the second Data Center only a
> few
> >> minutes behind the data in the main Data Center (the only way to provide
> >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
> and
> >> MySQL). My supervisor was asking for clarification whether you are
> implying
> >> that Leader/Follower replication is being deprecated.
> >>
> >>
> >> In my continuing attempts to resolve these issues, I have come across a
> >> related question. The error message about SolrAuthV2 prompted me to
> wonder
> >> about the V1 and V2 syntax options (such as shown at
> >>
> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
> );
> >> and I was wondering whether Leader/Follower replication changed from
> using
> >> syntax V1 to using syntax V2, and if that might contribute to the
> breaking
> >> of permissions. As an experiment in our test environment, I setup a
> >> permission in security.json to allow RELOAD collection without a
> password.
> >> After confirming that my V2 syntax does work _with_ a password, I then
> >> attempted RELOAD collection without a password using both syntax V1 and
> >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
> several
> >> permutations in security.json to allow RELOAD without password in syntax
> >> V2, but have not yet found a successful permutation. Are there any
> >> clarifications what security.json changes are needed for syntax V2? Can
> it
> >> be confirmed whether Leader/Follower replication is using V2 (in other
> >> words, whether that may be contributing to the permission problem)?
> >>
> >>
> >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
> 'Content-Type:
> >> application/json' -d '{"reload":{}}'
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":255},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":219}}}}
> >> [11:51 dbh19850s 1152]$ curl "http://localhost:
> $p/solr/admin/collections?action=RELOAD&name=helpdocs"
> >>
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":237},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":216}}}}
> >> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
> $p/api/collections/helpdocs"
> >> -H 'Content-Type: application/json' -d '{"reload":{}}'
> >> <html>
> >> <head>
> >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> >> <title>Error 401 Authentication failed, Response code: 401</title>
> >> </head>
> >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> >> <table>
> >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> >> <tr><th>STATUS:</th><td>401</td></tr>
> >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> >> 401</td></tr>
> >> <tr><th>SERVLET:</th><td>default</td></tr>
> >> </table>
> >>
> >> </body>
> >> </html>
> >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> >>
> >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 401 491
> >> [11:51 dbh19850s 1155]$ less solr.log
> >> ...
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> >> [/admin/collections]
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> >> ...
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] has role, but request principal cannot be identified;
> >> forbidding access
> >>
> >>
> >> As a side note, in our experience, the only thing that has been
> cluttering
> >> up solr.log with attempts to connect without a password has been
> >> Leader/Follower replication (formerly known as Master/Slave
> replication).
> >>
> >>
> >> -----Original Message-----
> >> From: Shawn Heisey <ap...@elyograg.org>
> >> Sent: Saturday, August 5, 2023 4:24 PM
> >> To: users@solr.apache.org
> >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
> >>
> >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> >>> We are having problems transitioning Leader/Follower replication to
> >> Solr9.2.1
> >>>
> >>> In Solr8.5 and below, what was then called Master/Slave replication had
> >> the annoying problem that, even though we specified httpBasicAuthUser
> and
> >> httpBasicAuthPassword, it would always attempt to connect first without
> a
> >> password before retrying with a password. This made solr.log noisy with
> >> lots of unnecessary login failures: but at least it worked.
> >>
> >> In general, this is how basic auth via http works. The client first
> >> attempts the request without any credentials, and receives a 401
> response.
> >>
> >> At this point, a browser would see the 401 response and display a popup
> >> asking for username/password.
> >>
> >> Then on future requests to that server in the current session, the
> >> browser sends the supplied credentials on every request to that server.
> >>
> >> If you are supplying credentials in the replication config, it should
> >> NOT follow that pattern ... the credentials should be always used.
> >>
> >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> >> with the following:
> >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Leader at: http://
> [REDACTED]/solr/sequence2_shard1_replica_n1
> >> is not available.
> >>
> >> The info above is valid for Solr running in standalone mode. But those
> >> core names indicate that you are running in SolrCloud mode.
> >>
> >> In cloud mode, Solr handles all replication. Don't attempt to actually
> >> configure the replication handler in cloud mode ... Solr will handle it
> >> all for you, and will even automatically take care of authenticating the
> >> requests. You don't need to configure the replication handler at all.
> >>
> >> If you are using the replication handler as a "back door" to copy
> >> indexes to a separate Solr install, a better option is to use the
> >> backup/restore functionality in the Collections API.
> >>
> >> Thanks,
> >> Shawn
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you recognize the sender and are
> >> confident the content is safe.
> >>
> >>
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
> >
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
security.json not working for V2 syntax
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
Having not seen any response, I thought I would see if it would help to ask the question again with a different subject
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
Sent: Wednesday, August 23, 2023 11:34 PM
To: users@solr.apache.org
Subject: RE: [EXTERNAL] authorization differences between V1 (works) and V2 (fails)
We may implement something with incremental backup and restore; but in the meanwhile, if we could resolve the permission problem with Leader/Follower replication, that would expedite our upgrade to Solr9
In hopes that the discrepancy mentioned below is a clue, I would like to ask it again (copying and pasting to put it back at the top)
The error message about SolrAuthV2 prompted me to wonder about the V1 and V2 syntax options (such as shown at solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); and I was wondering whether Leader/Follower replication changed from using syntax V1 to using syntax V2, and if that might contribute to the breaking of permissions.
As an experiment in our test environment, I setup a permission in security.json to allow RELOAD collection without a password. After confirming that my V2 syntax does work _with_ a password, I then attempted RELOAD collection without a password using both syntax V1 and syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several permutations in security.json to allow RELOAD without password in syntax V2, but have not yet found a successful permutation. Are there any clarifications what security.json changes are needed for syntax V2? Can it be confirmed whether Leader/Follower replication is using V2 (in other words, whether that may be contributing to the permission problem)?
[11:51 dbh19850s 1152]$ curl -X POST "http://`cat /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
{
"responseHeader":{
"status":0,
"QTime":255},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":219}}}}
[11:51 dbh19850s 1152]$ curl "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"
{
"responseHeader":{
"status":0,
"QTime":237},
"success":{
"nosqltest21.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":176}},
"nosqltest22.be-md:9852_solr":{
"responseHeader":{
"status":0,
"QTime":216}}}}
[11:51 dbh19850s 1153]$ curl -X POST "http://localhost:$p/api/collections/helpdocs" -H 'Content-Type: application/json' -d '{"reload":{}}'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Authentication failed, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
<table>
<tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Authentication failed, Response code: 401</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
*[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs HTTP/1.1" 401 491
[11:51 dbh19850s 1155]$ less solr.log
...
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] to govern resource [/admin/collections]
2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"openreload8",
"path":"/admin/collections",
"index":9,
"collection":null,
"role":null,
"params":{
"action":["REGEX:(?i)RELOAD"],
"name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
...
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
"name":"catch-all-nocollection",
"path":"/*",
"index":24,
"collection":null,
"role":"allgen"}] has role, but request principal cannot be identified; forbidding access
-----Original Message-----
From: Kevin Risden <kr...@apache.org>
Sent: Monday, August 14, 2023 10:31 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
The issue w/ allowPaths was reported here:
https://issues.apache.org/jira/browse/SOLR-16905
Basically allowPaths was not being reflected in the security manager
policy. You can update security.policy yourself or disable security manager
as you found.
Kevin Risden
On Mon, Aug 14, 2023 at 10:24 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks: the "-Dsolr.allowPaths=/netmnt/" did not suffice, but after
> SOLR_SECURITY_MANAGER_ENABLED=false it worked (both with incremental=false
> and incremental=true)
>
>
> -----Original Message-----
> From: Jan Høydahl <ja...@cominvent.com>
> Sent: Monday, August 14, 2023 3:12 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] authentication for Leader/Follower replication
>
> This is probably due to SecurityManager rejecting an unknown path. Try
> starting Solr with:
>
> -Dsolr.allowPaths=/netmnt/
>
> or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/"
>
> Then try again. Alternatively you can try disabling SecMgr withenv.var
> SOLR_SECURITY_MANAGER_ENABLED=false
>
>
> https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html
>
> Jan
>
> > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.INVALID>:
> >
> > Thanks: I was forgetting the fact that (in order to get backups in the
> format we expected) I added "incremental=false" to our backup script when
> we upgraded from Solr8.5 to Solr8.11.
> >
> > It trying to test this functionality, however, I now find that backups
> do not seem to be working for me *at all* in Solr9.2, with or without
> "incremental=false"
> >
> > I keep getting a stack trace such as:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <response>
> >
> > <lst name="responseHeader">
> > <int name="status">500</int>
> > <int name="QTime">9</int>
> > </lst>
> > <lst name="error">
> > <str name="msg">access denied ("java.io.FilePermission"
> "/netmnt/[redacted-directory]" "read")</str>
> > <str name="trace">java.security.AccessControlException: access denied
> ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
> > at
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > at
> java.base/java.security.AccessController.checkPermission(AccessController.java:897)
> > at
> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
> > at
> java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
> > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
> > at
> java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
> > at java.base/java.nio.file.Files.exists(Files.java:2435)
> > at
> org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
> > at
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
> > at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
> > at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
> > at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
> > at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> > at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> > at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> > at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> > at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:100)
> > at org.eclipse.jetty.io
> .SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
> > at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > </str>
> > <int name="code">500</int>
> > </lst>
> > </response>
> >
> >
> > It should be noted that the Linux account which runs Solr has read and
> write access to the specified location, which is a subdirectory of what is
> listed in allowPaths in all the solr.xml files of this SolrCloud, and which
> is mounted on all hosts of the SolrCloud.
> >
> > I have tried both V1 API and V2 API as found at
> solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
> >
> > Not being familiar with V2 API, I tried various permutations of
> "file:///netmnt/" (various numbers of slashes, etc): they all seem to get
> the same stack trace
> >
> > Does anyone have any example of a successful backup from Solr9.2?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Dominique Bejean <do...@eolya.fr>
> > Sent: Friday, August 11, 2023 9:32 AM
> > To: users@solr.apache.org
> > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower
> replication
> >
> > Hi Craig,
> >
> > Yes, starting with the 8.9 and 9.0 versions, Collection API allows
> > incremental backup and much more (corruption check, backup to Amazon S3
> or
> > Google Cloud Storage) .
> > Take a look at this umbrella JIRA
> > https://issues.apache.org/jira/browse/SOLR-15086
> >
> > Regards
> >
> > Dominique
> >
> >
> >
> > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.invalid> a écrit :
> >
> >> RE: a better option is to use the backup/restore functionality in the
> >> Collections API.
> >>
> >> My impression was that there is no facility for _incremental_ backup and
> >> retore in the collections API: is there? To do backup and scp and
> restore
> >> of terabytes of data every few minutes does not sound practical.
> >>
> >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL
> as
> >> well as Solr) is redundancy (for failover) in our main Data Center, and
> >> redundancy in read-only reasonably-concurrent copies in our second Data
> >> Center. For Solr8.11 (and earlier), we have had SolrCloud for
> redundancy in
> >> our main Data Center, and Leader/Follower replication to the read-only
> >> SolrClouds in the second Data Center. At one time, we were hoping that
> CDCR
> >> would work better: but we never managed to get CDCR to work reliably
> (and
> >> perhaps others also found it unreliable, leading to it being deprecated
> >> rather than being fixed). We have found that SolrCloud does not work
> >> reliably when spread across Data Centers, so Leader/Follower replication
> >> (formerly known as Master/Slave replication) has been the only way we
> have
> >> found to keep our (read-only) copies in the second Data Center only a
> few
> >> minutes behind the data in the main Data Center (the only way to provide
> >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres
> and
> >> MySQL). My supervisor was asking for clarification whether you are
> implying
> >> that Leader/Follower replication is being deprecated.
> >>
> >>
> >> In my continuing attempts to resolve these issues, I have come across a
> >> related question. The error message about SolrAuthV2 prompted me to
> wonder
> >> about the V1 and V2 syntax options (such as shown at
> >>
> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload
> );
> >> and I was wondering whether Leader/Follower replication changed from
> using
> >> syntax V1 to using syntax V2, and if that might contribute to the
> breaking
> >> of permissions. As an experiment in our test environment, I setup a
> >> permission in security.json to allow RELOAD collection without a
> password.
> >> After confirming that my V2 syntax does work _with_ a password, I then
> >> attempted RELOAD collection without a password using both syntax V1 and
> >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried
> several
> >> permutations in security.json to allow RELOAD without password in syntax
> >> V2, but have not yet found a successful permutation. Are there any
> >> clarifications what security.json changes are needed for syntax V2? Can
> it
> >> be confirmed whether Leader/Follower replication is using V2 (in other
> >> words, whether that may be contributing to the permission problem)?
> >>
> >>
> >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H
> 'Content-Type:
> >> application/json' -d '{"reload":{}}'
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":255},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":219}}}}
> >> [11:51 dbh19850s 1152]$ curl "http://localhost:
> $p/solr/admin/collections?action=RELOAD&name=helpdocs"
> >>
> >> {
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":237},
> >> "success":{
> >> "nosqltest21.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":176}},
> >> "nosqltest22.be-md:9852_solr":{
> >> "responseHeader":{
> >> "status":0,
> >> "QTime":216}}}}
> >> [11:51 dbh19850s 1153]$ curl -X POST "http://localhost:
> $p/api/collections/helpdocs"
> >> -H 'Content-Type: application/json' -d '{"reload":{}}'
> >> <html>
> >> <head>
> >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> >> <title>Error 401 Authentication failed, Response code: 401</title>
> >> </head>
> >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> >> <table>
> >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> >> <tr><th>STATUS:</th><td>401</td></tr>
> >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> >> 401</td></tr>
> >> <tr><th>SERVLET:</th><td>default</td></tr>
> >> </table>
> >>
> >> </body>
> >> </html>
> >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
> >>
> >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST
> /api/collections/helpdocs
> >> HTTP/1.1" 401 491
> >> [11:51 dbh19850s 1155]$ less solr.log
> >> ...
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> >> [/admin/collections]
> >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"openreload8",
> >> "path":"/admin/collections",
> >> "index":9,
> >> "collection":null,
> >> "role":null,
> >> "params":{
> >> "action":["REGEX:(?i)RELOAD"],
> >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> >> ...
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> >> "name":"catch-all-nocollection",
> >> "path":"/*",
> >> "index":24,
> >> "collection":null,
> >> "role":"allgen"}] has role, but request principal cannot be identified;
> >> forbidding access
> >>
> >>
> >> As a side note, in our experience, the only thing that has been
> cluttering
> >> up solr.log with attempts to connect without a password has been
> >> Leader/Follower replication (formerly known as Master/Slave
> replication).
> >>
> >>
> >> -----Original Message-----
> >> From: Shawn Heisey <ap...@elyograg.org>
> >> Sent: Saturday, August 5, 2023 4:24 PM
> >> To: users@solr.apache.org
> >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
> >>
> >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> >>> We are having problems transitioning Leader/Follower replication to
> >> Solr9.2.1
> >>>
> >>> In Solr8.5 and below, what was then called Master/Slave replication had
> >> the annoying problem that, even though we specified httpBasicAuthUser
> and
> >> httpBasicAuthPassword, it would always attempt to connect first without
> a
> >> password before retrying with a password. This made solr.log noisy with
> >> lots of unnecessary login failures: but at least it worked.
> >>
> >> In general, this is how basic auth via http works. The client first
> >> attempts the request without any credentials, and receives a 401
> response.
> >>
> >> At this point, a browser would see the 401 response and display a popup
> >> asking for username/password.
> >>
> >> Then on future requests to that server in the current session, the
> >> browser sends the supplied credentials on every request to that server.
> >>
> >> If you are supplying credentials in the replication config, it should
> >> NOT follow that pattern ... the credentials should be always used.
> >>
> >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> >> with the following:
> >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> >> o.a.s.h.IndexFetcher Leader at: http://
> [REDACTED]/solr/sequence2_shard1_replica_n1
> >> is not available.
> >>
> >> The info above is valid for Solr running in standalone mode. But those
> >> core names indicate that you are running in SolrCloud mode.
> >>
> >> In cloud mode, Solr handles all replication. Don't attempt to actually
> >> configure the replication handler in cloud mode ... Solr will handle it
> >> all for you, and will even automatically take care of authenticating the
> >> requests. You don't need to configure the replication handler at all.
> >>
> >> If you are using the replication handler as a "back door" to copy
> >> indexes to a separate Solr install, a better option is to use the
> >> backup/restore functionality in the Collections API.
> >>
> >> Thanks,
> >> Shawn
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you recognize the sender and are
> >> confident the content is safe.
> >>
> >>
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
> >
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.