solr 8x -> 9.0.0 upgrade; BasicAuth security FAILs @ "o.a.s.s.BasicAuthPlugin Bad auth credentials supplied in Authorization header" ?

[PGNet Dev <pg...@gmail.com>]

  on

	lsb_release -rd
		Description:    Fedora release 36 (Thirty Six)
		Release:        36

	java -version
		Picked up JAVA_TOOL_OPTIONS: -Xmx512M
		openjdk version "18.0.1" 2022-04-19
		OpenJDK Runtime Environment 22.3 (build 18.0.1+10)
		OpenJDK 64-Bit Server VM 22.3 (build 18.0.1+10, mixed mode, sharing)

i've clean-installed solr 9.0.0

	sudo -u solr /srv/webapps/solr/solr/bin/solr version
		9.0.0

it's up/running

	systemctl status solr
	● solr.service - LSB: Controls Apache Solr as a Service
	     Loaded: loaded (/etc/rc.d/init.d/solr; generated)
	     Active: active (exited) since Fri 2022-05-13 06:22:40 EDT; 2min 54s ago
	       Docs: man:systemd-sysv-generator(8)
	    Process: 56877 ExecStart=/etc/rc.d/init.d/solr start (code=exited, status=0/SUCCESS)
	        CPU: 43ms

with no user/auth security,

	ls -al /data/solr/data/security.json
		ls: cannot access '/data/solr/data/security.json': No such file or directory

nav to & admin @,

	https:///solr.example.com:8983/solr

works as expected.

deploying user BasicAuth security

	https://solr.apache.org/guide/solr/latest/deployment-guide/basic-authentication-plugin.html

with

	MY_USER_PASS="aaaaaaaaaaaaaa_bbbbbbbbbbbb_111111111+ccccccc_22"
	MY_USER_HASH=$( echo -n $MY_USER_PASS | shasum -a 256 | awk '{print $1}' | tr -d ' ')
	echo $MY_USER_HASH
		79a054509e27e20b16fb85caf221ac8d488168afa6715f2543d761269a72d832
and


	egrep "Dbasicauth|SOLR_LOG_LEVEL" /etc/default/solr.in.sh
		SOLR_LOG_LEVEL=DEBUG
		SOLR_AUTHENTICATION_OPTS="-Dbasicauth=testuser:aaaaaaaaaaaaaa_bbbbbbbbbbbb_111111111+ccccccc_22"

and

	cat /data/solr/data/security.json
		{
		"authentication":{
		   "blockUnknown": true,
		   "class":"solr.BasicAuthPlugin",
		   "credentials":{"testuser":"79a054509e27e20b16fb85caf221ac8d488168afa6715f2543d761269a72d832"},
		   "realm":"MyRealm Solr",
		   "forwardCredentials": false
		},
		"authorization":{
		   "class":"solr.RuleBasedAuthorizationPlugin",
		   "permissions":[{"name":"security-edit",
		      "role":"admin"}],
		   "user-role":{"solr":"admin"}
		}}

nav to:

	https:///solr.example.com:8983/solr

returns the expected

	Basic Authentication

form.

entering credentials

	username: testuser
	password: aaaaaaaaaaaaaa_bbbbbbbbbbbb_111111111+ccccccc_22

fails with

	Basic Authentication
	Unauthorized

	Solr requires authentication for resource Dashboard.
	Please log in with your username and password for realm MyRealm Solr.

and DEBUG logs,

	--> https://pastebin.com/raw/aHVCgGKF

there, this looks possibly suspect,

		...
		2022-05-13 06:33:00.651 DEBUG (qtp1777443462-23) [] o.a.s.s.SolrDispatchFilter Request to authenticate: org.apache.solr.servlet.ServletUtils$1@3acaf4f, domain: 10.1.1.27, port: 8983
		2022-05-13 06:33:00.656 DEBUG (qtp1777443462-22) [] o.a.s.s.SolrDispatchFilter Request to authenticate: org.apache.solr.servlet.ServletUtils$1@540dbd19, domain: 10.1.1.27, port: 8983
		2022-05-13 06:33:00.660 DEBUG (qtp1777443462-23) [] o.a.s.s.BasicAuthPlugin Bad auth credentials supplied in Authorization header
		2022-05-13 06:33:00.650 DEBUG (qtp1777443462-20) [] o.a.s.s.SolrDispatchFilter Request to authenticate: org.apache.solr.servlet.ServletUtils$1@7e6b57df, domain: 10.1.1.27, port: 8983
		2022-05-13 06:33:00.661 DEBUG (qtp1777443462-20) [] o.a.s.s.BasicAuthPlugin Bad auth credentials supplied in Authorization header
		2022-05-13 06:33:00.662 DEBUG (qtp1777443462-20) [] o.a.s.s.BasicAuthPlugin Prefixing WWW-Authenticate header for Basic Auth with 'x' to prevent browser basic auth popup
??		2022-05-13 06:33:00.663 DEBUG (qtp1777443462-22) [] o.a.s.s.BasicAuthPlugin Bad auth credentials supplied in Authorization header
??		2022-05-13 06:33:00.663 DEBUG (qtp1777443462-22) [] o.a.s.s.BasicAuthPlugin Prefixing WWW-Authenticate header for Basic Auth with 'x' to prevent browser basic auth popup
		2022-05-13 06:33:00.667 DEBUG (qtp1777443462-22) [] o.e.j.s.HttpChannelState sendError HttpChannelState@191ce1ad{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0}
		...

dropping back to solr 8x, i've no issues with basicauth.

is my config wrong for 9x?  known issue? other?