You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "epugh (via GitHub)" <gi...@apache.org> on 2024/03/29 15:15:56 UTC
[PR] SOLR-17194: Export Tool with Basic Auth parameters doesn't proeprly build the CloudHttp2SolrClient [solr]
epugh opened a new pull request, #2378:
URL: https://github.com/apache/solr/pull/2378
https://issues.apache.org/jira/browse/SOLR-17194
# Description
We pass in a httpClient builder to CloudHttp2SolrClient, but in at least this code path we never use, so we have a null httpClient, which means we build a default one, which means if you passed one in then you don't get your Basic Auth params!
# Solution
A test and a small fix. I did not try to reasoon about all the builder stuff.. I find the combination of httpClient, and the solrClient and their various builders VERY hard to understand... I wrote at test, I fixed the code, the test passes...
# Tests
Added a test to the Export Tool
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1561775883
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 2, 1)
+ .setBasicAuthCredentials(USER, PASS)
+ .process(cluster.getSolrClient());
+ cluster.waitForActiveCollection(COLLECTION_NAME, 8, 8);
Review Comment:
You won't get 8 replicas anymore; you changed this. The test probably fails now. I hate `waitForActiveCollection`; the collection is already active when createCollection finishes. The risk this method mitigates is that the SolrClient returned by `cluster.getSolrClient` doesn't know it yet, but this test doesn't use the client henceforth (unlike most tests).
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 2, 1)
Review Comment:
Two shards doesn't seem pertinent to what is being tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1557527811
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
Oh, wait... I found it.. The test is `testVeryLargeCluster` and it's a nightly... so I think the is why....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569181378
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
I think there might be an issue here with the `httpClient` instance created here is not being closed correctly. not 100% sure I will need to check more closely. before this change I agree there was a default client being created lacking the correct params but it also closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClients_.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. I added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1570859249
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
Actually I have 2 PRs that I am pretty sure would help a lot on this.
One that fixes this bug.
And anther one that I talked about [here](https://github.com/apache/solr/pull/1994#discussion_r1535512663) where I refactored ZK related SolrCli tools to separate _SolrZkClient_ and _CloudHttp2SolrClient_ instantiation/usage.
Will see if I can speed up the internal review and push them upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1559563492
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
I made the change to reduce the number...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2378:
URL: https://github.com/apache/solr/pull/2378#issuecomment-2067066181
Closing in favour of #2412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1557525041
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
total copy paste ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1561769753
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
Review Comment:
Can the behavior be tested with one node instead of two? Authentication seems unrelated to the size of the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1559411299
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
argh... you are right! somehow I was conflating my test with an existing one... (maybe looking at the wrong branch of code)... changing now....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClient_s.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. I added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClients_.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. The idea is to first assign/build the _myClient_ and then pass it to _Http2ClusterStateProvider_. This guaranties that one and unique httClient instance is used (and released).
I added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1570775062
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
Thanks all for looking at this. I have definitly hit the "end" of my rope on this one, just can't figure it out... My Java-fu isn't up to this one.
I am inclined to close this PR in favour of what you can whip up.. However, since I really really really really really really want basic auth to work on all the tools, I am hesitating to actually do that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: Export Tool with Basic Auth parameters doesn't proeprly build the CloudHttp2SolrClient [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2378:
URL: https://github.com/apache/solr/pull/2378#issuecomment-2027374009
@risdenk @HoustonPutman I tagged you guys based on git blame suggesting you had worked on CloudHttp2SolrClient in the areas that I changed ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClients_.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. The idea is to first create the myClient and then pass it to _Http2ClusterStateProvider_. This guaranties that one and unique httClient instance is used (and released).
I added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2378:
URL: https://github.com/apache/solr/pull/2378#issuecomment-2045001521
@dsmiley do the rest of the changes look okay to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2378:
URL: https://github.com/apache/solr/pull/2378#issuecomment-2037299366
Pinged community for feedback, I'd like to merge this week...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1558355768
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
I'm looking at your PR and don't see that it's a nightly test nor see why it deserves to be such even if it did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClients_.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. The idea is to first assign/build the _myClient_ and then pass it to _Http2ClusterStateProvider_. This guaranties that one and unique httClient instance is used (and released).
I also added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569571691
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
That's funny, I pushed 2 weeks ago an internal PR (at my work) for this exact same bug, as I faced it myself.
@stillalex is correct, the resource won't be released if an exception is raised when instantiating the _Http2ClusterStateProvider_. It also interferes with the _clientIsInternal_ logic and prevents _myClient_ from being released.
Plus we end up creating two _httpCient_ instances. CloudHttp2SolrClient and stateProvider would use 2 different _httClients_.
In my PR, I pushed most of the code into the CloudHttp2SolrClient constructor. The idea is to first assign/build the _myClient_ and then pass it to _Http2ClusterStateProvider_. This guaranties that one and unique httClient instance is used (and released).
I also added a test case to test if the same _httpClient_ is preserved in both _CloudHttp2SolrClient_ and _Http2ClusterStateProvider_.
All test cases passed, including _TestExportTool_.
Will see if I can push the PR here (upstream).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1569186974
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
thanks for chiming in... so, this change is making my TestExportTool blow up with unclosed httpclients.. I've banged my head on this for a LONG time, and not making headway. What I can say is that the test with the BasicAuth demonstrates the problem, but I can't grok the soluiton... @stillalex I would love your ideas if you wanted to take a stab at it.. it felt like such an easy fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh closed pull request #2378: SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths
URL: https://github.com/apache/solr/pull/2378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "laminelam (via GitHub)" <gi...@apache.org>.
laminelam commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1572519870
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudHttp2SolrClient.java:
##########
@@ -422,7 +422,10 @@ public CloudHttp2SolrClient build() {
}
} else if (!solrUrls.isEmpty()) {
try {
- stateProvider = new Http2ClusterStateProvider(solrUrls, httpClient);
+ if (this.internalClientBuilder != null) {
+ this.httpClient = internalClientBuilder.build();
Review Comment:
@epugh
Here's the [PR](https://github.com/apache/solr/pull/2412)
Tested (locally) your testWithBasicAuth() code and the test passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1551972946
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
8 shards?!
To tame our tests, tests should have a comment justifying >1 nodes or >1 shards. If there is a reason; justify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17194: CloudHttp2SolrClient doesn't properly use httpClient Builder on certain paths [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2378:
URL: https://github.com/apache/solr/pull/2378#discussion_r1557528153
##########
solr/core/src/test/org/apache/solr/cli/TestExportTool.java:
##########
@@ -218,6 +231,66 @@ public void testVeryLargeCluster() throws Exception {
}
}
+ @Test
+ public void testWithBasicAuth() throws Exception {
+ String COLLECTION_NAME = "secureCollection";
+ String USER = "solr";
+ String PASS = "SolrRocksAgain";
+ final String SECURITY_JSON =
+ Utils.toJSONString(
+ Map.of(
+ "authorization",
+ Map.of(
+ "class",
+ RuleBasedAuthorizationPlugin.class.getName(),
+ "user-role",
+ singletonMap(USER, "admin"),
+ "permissions",
+ singletonList(Map.of("name", "all", "role", "admin"))),
+ "authentication",
+ Map.of(
+ "class",
+ BasicAuthPlugin.class.getName(),
+ "blockUnknown",
+ true,
+ "credentials",
+ singletonMap(USER, getSaltedHashedValue(PASS)))));
+
+ configureCluster(2)
+ .addConfig("conf", configset("cloud-minimal"))
+ .withSecurityJson(SECURITY_JSON)
+ .configure();
+
+ CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf", 8, 1)
Review Comment:
do you want another specific comment on the test....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org