You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org> on 2026/07/15 16:19:48 UTC

[PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

serhiy-bzhezytskyy opened a new pull request, #4640:
URL: https://github.com/apache/solr/pull/4640

   https://issues.apache.org/jira/browse/SOLR-17316
   
   SolrJ's response classes assume binary-parser types, so reading a response parsed by a non-binary parser (the JSON parser) throws ClassCastException. Two commits so the small safe fix can go in on its own if you'd rather.
   
   Commit 1 is the narrow fix: getStatus/getQTime cast to Integer, which CCEs when JSON gives Long. Widened via Number. That's it — resolves the getters the issue names.
   
   Commit 2 goes further. It's not just the header — getResults(), facets, grouping all break under JSON too, because the parser hands back raw Map/List where the code wants NamedList/SolrDocumentList. So there's a ResponseNormalizer that converts a parsed response to the canonical shape at the client boundary (no-op for binary/XML), gated by a new ResponseParser.producesCanonicalForm() so only the JSON map parser triggers it. Both transports covered, binary pays nothing. Plus the remaining Integer/Float casts widened.
   
   Heads up: commit 2 basically does what SOLR-3451 asked for in 2012, which was closed Won't Fix ("solr does not have a way to write a JSON response and read the same value"). Still true for json.nl=flat since it's lossy, but json.nl=map round-trips and the normalizer does the rest. So I'd treat commit 2 as reopening that discussion — fine to split it out or take it to dev@ if you'd prefer, commit 1 stands alone either way.
   
   Tests: the normalizer + edge cases, binary/xml/json-map parity, the affected sections (grouping, facets, stats, spellcheck, highlighting, terms, moreLikeThis, analysis, Luke, schema), and an end-to-end JSON query on both the Jetty and JDK clients.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5088820099

   Are there any objections to its merger? I am ready to resolve them, just let me know. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3708683324


##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/json/JsonMapResponseParser.java:
##########
@@ -64,8 +68,23 @@ public Set<String> getContentTypes() {
     return CONTENT_TYPES;
   }
 
+  private static final SolrParams REQUEST_PARAMS =
+      new MapSolrParams(Map.of(JsonTextWriter.JSON_NL_STYLE, JsonTextWriter.JSON_NL_MAP));

Review Comment:
   see `SolrParams.of(...`



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/ResponseParserCanonicalResponseTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import java.io.ByteArrayInputStream;
+import java.io.InputStream;
+import java.util.Map;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.JsonTextWriter;
+import org.apache.solr.common.util.NamedList;
+import org.junit.Test;
+
+/**
+ * Pins the {@link ResponseParser#processCanonicalResponse} contract: whatever a parser's natural
+ * output looks like, this method returns the canonical shape the SolrJ response classes read — a
+ * NamedList tree with SolrDocumentList for document sections. The conversion belongs to the parser,
+ * so a client does not need to know which parsers require it.
+ */
+public class ResponseParserCanonicalResponseTest extends SolrTestCase {
+
+  private static final String JSON =
+      """
+      {"responseHeader":{"status":0,"QTime":1},\
+      "response":{"numFound":1,"start":0,"numFoundExact":true,"docs":[{"id":"1"}]}}""";
+
+  private static InputStream json() {
+    return new ByteArrayInputStream(JSON.getBytes(UTF_8));
+  }
+
+  /** The JSON map parser's own output is raw: Maps where the response classes expect NamedLists. */
+  @Test
+  public void testJsonMapParserRawOutputIsNotCanonical() throws Exception {
+    NamedList<Object> raw = new JsonMapResponseParser().processResponse(json(), null);
+    assertTrue("raw header should be a Map", raw.get("responseHeader") instanceof Map);
+    assertFalse(
+        "raw header should not be a NamedList", raw.get("responseHeader") instanceof NamedList);
+    assertFalse(
+        "raw response should not be a SolrDocumentList",
+        raw.get("response") instanceof SolrDocumentList);
+  }
+
+  /** ... and processCanonicalResponse converts it, without the caller asking. */
+  @Test
+  public void testJsonMapParserCanonicalResponseIsConverted() throws Exception {
+    NamedList<Object> out = new JsonMapResponseParser().processCanonicalResponse(json(), null);
+    assertTrue("header must be a NamedList", out.get("responseHeader") instanceof NamedList);
+    assertTrue(
+        "response must be a SolrDocumentList", out.get("response") instanceof SolrDocumentList);
+    assertEquals(1, ((SolrDocumentList) out.get("response")).getNumFound());
+  }
+
+  /** Parsers that are canonical already inherit the default and are unchanged by it. */
+  @Test
+  public void testCanonicalParsersPassThrough() throws Exception {
+    String xml =
+        """
+        <?xml version="1.0" encoding="UTF-8"?>
+        <response><lst name="responseHeader"><int name="status">0</int></lst></response>""";
+    NamedList<Object> out =
+        new XMLResponseParser()
+            .processCanonicalResponse(new ByteArrayInputStream(xml.getBytes(UTF_8)), null);
+    assertTrue("header must be a NamedList", out.get("responseHeader") instanceof NamedList);
+  }
+
+  /**
+   * A parser that needs the response written a particular way supplies that param itself, rather
+   * than relying on every caller to know it. The JSON map parser needs {@code json.nl=map}: under
+   * the default {@code flat} a NamedList arrives as an array of alternating names and values, whose
+   * structure cannot be recovered.
+   */
+  @Test
+  public void testJsonMapParserRequestsNlMap() {
+    SolrParams params = new JsonMapResponseParser().getRequestParams();
+    assertNotNull("the JSON map parser must ask for a recoverable NamedList form", params);
+    assertEquals(JsonTextWriter.JSON_NL_MAP, params.get(JsonTextWriter.JSON_NL_STYLE));
+  }

Review Comment:
   IMO this test doesn't add value.  It unit tests a fine detail using more code than the non-test.  Instead we want to see an integration test showing the correct behavior that this underlying detail here helps arrange for -- i.e. the circumstance that led to the need to have this.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseParser.java:
##########
@@ -50,6 +51,17 @@ private boolean validateContentTypes() {
   /** The writer type placed onto the request as the {@code wt} param. */
   public abstract String getWriterType(); // for example: wt=XML, JSON, etc
 
+  /**
+   * Params this parser needs on the request for its own reads, applied alongside {@code wt}.
+   *
+   * <p>A parser that needs the response written a particular way returns those params here rather
+   * than relying on callers to set them. Anything the caller set explicitly wins, so this only
+   * supplies defaults. Returns null when the parser needs nothing beyond {@code wt}.
+   */
+  public SolrParams getRequestParams() {

Review Comment:
   Lets name this `getAdditionalRequestParams`.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -139,6 +140,16 @@ protected ModifiableSolrParams initializeSolrParams(
     // The parser 'wt=' param is used instead of the original params
     ModifiableSolrParams wparams = new ModifiableSolrParams(solrRequest.getParams());
     wparams.set(CommonParams.WT, parserToUse.getWriterType());
+    // Params the parser needs for its own reads, without overriding what the request already set.
+    SolrParams parserParams = parserToUse.getRequestParams();
+    if (parserParams != null) {
+      for (Iterator<String> it = parserParams.getParameterNamesIterator(); it.hasNext(); ) {
+        String name = it.next();
+        if (wparams.get(name) == null) {
+          wparams.set(name, parserParams.getParams(name));
+        }
+      }

Review Comment:
   why loop the names when you could just call `parserParams.get(CommonParams.WT)` ?



##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -3,7 +3,9 @@
 title: >
   SolrJ's QueryResponse and other response objects now work when the client is configured with a
   non-binary response parser (such as the JSON parser); previously their accessors could throw a
-  ClassCastException.
+  ClassCastException, and nested documents were unreachable. A ResponseParser can now declare the
+  request params it needs via getRequestParams(); JsonMapResponseParser uses this to ask for

Review Comment:
   Not worth putting here; it's an implementation detail.  What you wrote originally was very good.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/TestSuggesterResponse.java:
##########


Review Comment:
   thanks for this.  It's good.  
   
   However, note that most of Lucene & Solr's randomized testing is done at a deeper level such that an individual test generally doesn't even have to do anything to get the randomization -- it just happens at a deeper test framework/infra level.  For example... imagine if the default was a settable static supplier... and imagine if SolrTestCase were to set it.  Then the useful test coverage would go through the roof (thousands of Solr tests) and we'd probably toss aside more of your tests as redundant.  I'm not sure I want to say we should do precisely this... but I'm at least informing you of the rather unique randomized testing philosophy that permeates the Lucene & Solr projects.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleJsonMapTest.java:
##########
@@ -14,28 +14,19 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.solr.client.solrj.response;
+package org.apache.solr.client.solrj;
 
-import org.apache.solr.SolrTestCase;
+import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
 import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
-import org.junit.Test;
 
-/**
- * Pins the producesCanonicalForm() contract that gates response normalization: only the JSON map
- * parser (which yields raw Maps/Lists) needs normalizing; the parsers that already produce the
- * canonical NamedList/SolrDocumentList shape must report true so they pass through untouched.
- */
-public class ResponseParserCanonicalFormTest extends SolrTestCase {
-
-  @Test
-  public void testCanonicalParsersReportTrue() {
-    assertTrue(new JavaBinResponseParser().producesCanonicalForm());
-    assertTrue(new XMLResponseParser().producesCanonicalForm());
-    assertTrue(new InputStreamResponseParser("json").producesCanonicalForm());
-  }
-
-  @Test
-  public void testJsonMapParserReportsFalse() {
-    assertFalse(new JsonMapResponseParser().producesCanonicalForm());
+/** Runs the example tests over {@link JsonMapResponseParser}. */

Review Comment:
   Niiiiice.... :-) thank you



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/ResponseParserCanonicalResponseTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import java.io.ByteArrayInputStream;
+import java.io.InputStream;
+import java.util.Map;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.JsonTextWriter;
+import org.apache.solr.common.util.NamedList;
+import org.junit.Test;
+
+/**
+ * Pins the {@link ResponseParser#processCanonicalResponse} contract: whatever a parser's natural
+ * output looks like, this method returns the canonical shape the SolrJ response classes read — a
+ * NamedList tree with SolrDocumentList for document sections. The conversion belongs to the parser,
+ * so a client does not need to know which parsers require it.
+ */
+public class ResponseParserCanonicalResponseTest extends SolrTestCase {
+
+  private static final String JSON =
+      """
+      {"responseHeader":{"status":0,"QTime":1},\
+      "response":{"numFound":1,"start":0,"numFoundExact":true,"docs":[{"id":"1"}]}}""";
+
+  private static InputStream json() {
+    return new ByteArrayInputStream(JSON.getBytes(UTF_8));
+  }
+
+  /** The JSON map parser's own output is raw: Maps where the response classes expect NamedLists. */
+  @Test
+  public void testJsonMapParserRawOutputIsNotCanonical() throws Exception {
+    NamedList<Object> raw = new JsonMapResponseParser().processResponse(json(), null);
+    assertTrue("raw header should be a Map", raw.get("responseHeader") instanceof Map);
+    assertFalse(
+        "raw header should not be a NamedList", raw.get("responseHeader") instanceof NamedList);
+    assertFalse(
+        "raw response should not be a SolrDocumentList",
+        raw.get("response") instanceof SolrDocumentList);
+  }
+
+  /** ... and processCanonicalResponse converts it, without the caller asking. */
+  @Test
+  public void testJsonMapParserCanonicalResponseIsConverted() throws Exception {
+    NamedList<Object> out = new JsonMapResponseParser().processCanonicalResponse(json(), null);
+    assertTrue("header must be a NamedList", out.get("responseHeader") instanceof NamedList);
+    assertTrue(
+        "response must be a SolrDocumentList", out.get("response") instanceof SolrDocumentList);
+    assertEquals(1, ((SolrDocumentList) out.get("response")).getNumFound());
+  }
+
+  /** Parsers that are canonical already inherit the default and are unchanged by it. */
+  @Test
+  public void testCanonicalParsersPassThrough() throws Exception {
+    String xml =
+        """
+        <?xml version="1.0" encoding="UTF-8"?>
+        <response><lst name="responseHeader"><int name="status">0</int></lst></response>""";
+    NamedList<Object> out =
+        new XMLResponseParser()
+            .processCanonicalResponse(new ByteArrayInputStream(xml.getBytes(UTF_8)), null);
+    assertTrue("header must be a NamedList", out.get("responseHeader") instanceof NamedList);
+  }
+
+  /**
+   * A parser that needs the response written a particular way supplies that param itself, rather
+   * than relying on every caller to know it. The JSON map parser needs {@code json.nl=map}: under
+   * the default {@code flat} a NamedList arrives as an array of alternating names and values, whose
+   * structure cannot be recovered.
+   */
+  @Test
+  public void testJsonMapParserRequestsNlMap() {
+    SolrParams params = new JsonMapResponseParser().getRequestParams();
+    assertNotNull("the JSON map parser must ask for a recoverable NamedList form", params);
+    assertEquals(JsonTextWriter.JSON_NL_MAP, params.get(JsonTextWriter.JSON_NL_STYLE));
+  }
+
+  /** Parsers that need nothing beyond wt contribute no params. */

Review Comment:
   again; no value



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/ResponseNormalizerTest.java:
##########


Review Comment:
   I see you are supporting anonymous child documents -- which is the original thing and I've been meaning to deprecate it.  Nowadays, we do "nested documents", which have named relationships from parent to child.  Neither is reflected in the schema, but anyway you can fetch children (named **or** anonymous) via `fl=*,[child fl=*]` if I recall off the top of my head.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -139,6 +140,16 @@ protected ModifiableSolrParams initializeSolrParams(
     // The parser 'wt=' param is used instead of the original params
     ModifiableSolrParams wparams = new ModifiableSolrParams(solrRequest.getParams());
     wparams.set(CommonParams.WT, parserToUse.getWriterType());
+    // Params the parser needs for its own reads, without overriding what the request already set.
+    SolrParams parserParams = parserToUse.getRequestParams();
+    if (parserParams != null) {
+      for (Iterator<String> it = parserParams.getParameterNamesIterator(); it.hasNext(); ) {
+        String name = it.next();
+        if (wparams.get(name) == null) {
+          wparams.set(name, parserParams.getParams(name));
+        }
+      }

Review Comment:
   I can see you (really the LLMs you use :-) ) are unfamiliar with `SolrParams.wrapDefaults` use that.  Even has the null check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3966577188


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,12 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects now work when the client is configured with a
+  non-binary response parser (such as the JSON parser); previously their accessors could throw a
+  ClassCastException.

Review Comment:
   Applied, and then reworded once more: after the default change the entry names `canonical()` as the new thing and says the plain constructor still returns the raw Maps and Lists, since that is what a reader upgrading needs to know.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3978775553


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,12 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects now work when the client is configured with a
+  non-binary response parser (such as the JSON parser); previously their accessors could throw a
+  ClassCastException.
+type: fixed

Review Comment:
   Applied.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3699950662


##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -229,6 +230,9 @@ protected NamedList<Object> processErrorsAndResponse(
       NamedList<Object> rsp;
       try {
         rsp = processor.processResponse(is, encoding);
+        if (!processor.producesCanonicalForm()) {

Review Comment:
   Agreed, and done as you described:
   
   ```java
   // ResponseParser — the default, inherited by every parser that is canonical already
   public NamedList<Object> processCanonicalResponse(InputStream body, String encoding)
       throws IOException {
     return processResponse(body, encoding);
   }
   
   // JsonMapResponseParser — the only override
   @Override
   public NamedList<Object> processCanonicalResponse(InputStream body, String encoding)
       throws IOException {
     return ResponseNormalizer.normalize(processResponse(body, encoding));
   }
   ```
   
   `HttpSolrClient` is now one line with no predicate, and `producesCanonicalForm()` is gone entirely.
   
   One thing I deliberately did not do: `processResponse` is left as it is. It has 20 call sites, and at least one of them wants the raw form — the error path in `ConcurrentUpdateBaseSolrClient` reads only `resp.get("error")`, so converting there would be work for nothing.
   
   The test that pinned the predicate is now `ResponseParserCanonicalResponseTest` and pins behaviour instead: the JSON parser's `processResponse` yields `Map`s, its `processCanonicalResponse` yields `NamedList`s and a `SolrDocumentList`, and a canonical parser passes through unchanged. Mutation-checked — dropping the override fails both that test and the integration test.



##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.util;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+
+/**
+ * Converts a parsed response into the canonical shape the SolrJ response classes expect (the shape
+ * the binary and XML parsers produce): nested JSON objects become {@link NamedList}s and a {@code
+ * {numFound, docs}} object becomes a {@link SolrDocumentList}.
+ *
+ * <p>Only unambiguous, self-describing conversions are performed. It is a no-op for values already
+ * in canonical form (so binary/XML responses pass through unchanged). It does not attempt to
+ * interpret the ambiguous flat arrays produced by {@code json.nl=flat}; a typed JSON parser should
+ * request {@code json.nl=map} for its own reads.
+ */
+public final class ResponseNormalizer {
+
+  private ResponseNormalizer() {}
+
+  /** Returns a normalized copy of the given response NamedList. */
+  public static NamedList<Object> normalize(NamedList<Object> response) {
+    if (response == null) {
+      return null;
+    }
+    SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(response.size());
+    for (Map.Entry<String, Object> e : response) {
+      out.add(e.getKey(), normalizeValue(e.getValue()));
+    }
+    return out;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static Object normalizeValue(Object val) {
+    if (val instanceof SolrDocumentList || val instanceof SolrDocument) {
+      // Already canonical (binary/XML produce these directly); leave untouched. Must precede the
+      // List/Map branches since SolrDocumentList is a List and SolrDocument is a Map.
+      return val;
+    } else if (val instanceof NamedList) {

Review Comment:
   Fixed — the concrete type is preserved now:
   
   ```java
   NamedList<Object> out =
       in instanceof SimpleOrderedMap<?>
           ? new SimpleOrderedMap<>(in.size())
           : new NamedList<>(in.size());
   ```
   
   A JSON object still becomes a `SimpleOrderedMap`, since its keys are unique by construction, but nothing widens a plain `NamedList` into one any more.
   
   Two things I checked while fixing it, in case either is useful. `SimpleOrderedMap` does not enforce uniqueness — its javadoc says "It's normally not a good idea to repeat keys… but this is not enforced" — so the concrete harm is elsewhere: it `implements Map`, and the response writers render the two differently ("a JSON response writer may choose to write a SimpleOrderedMap as {"foo":10,"bar":20} and may choose to write a NamedList as ["foo",10,"bar",20]"). Widening the type changes both of those.
   
   And a mutation check was worth running: with "always promote" restored, all 27 existing tests still passed. So the distinction had no coverage at all. There are two tests for it now — one asserting a plain `NamedList` with a repeated key survives as a `NamedList`, one asserting a `SimpleOrderedMap` stays one — and they do fail on the mutation.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");

Review Comment:
   It is necessary, though not for a good reason — I measured it rather than guess. Removing the line fails the test at collection creation:
   
   ```
   Path .../server/solr/configsets/sample_techproducts_configs must be relative to
   SOLR_HOME, SOLR_DATA_HOME coreRootDirectory.
   Set system property 'solr.security.allow.paths' to add other allowed paths.
   ```
   
   So it is `TECHPRODUCTS_CONFIGSET` that requires it, being outside the test SOLR_HOME.
   
   I tried to avoid it and could not, at least not cheaply. `SolrPingTest` uses the `solrj/solr` test home with no such property, but it gets away with it because it uses `EmbeddedSolrServerTestRule` — this test needs a real HTTP round trip, so `SolrJettyTestRule`, and Jetty scans the whole home and picks up `solrj/solr/shared/core0/core.properties`, which has an unsubstituted `dataDir=${dataDir1}`. The seven configsets under `solrj/src/test-files/.../configsets` each carry 4–15 unsubstituted placeholders, and only three have a `*_s` dynamic field.
   
   If you know a configset that works for this without the property, I will switch to it. Otherwise, when SOLR-18123 lands and the property goes away, this test will need whatever replaces it — I'd rather it be adjusted then than have me invent a workaround now.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/AdminResponseNumericTypeTest.java:
##########


Review Comment:
   Taking this one separately — it is the largest of your notes and I have not acted on it yet.
   
   `QueryResponseJsonParserIntegrationTest` (in this PR) is the integration test: a real Jetty round trip over `wt=json` that reads documents, facets and grouping through `QueryResponse`. `AdminResponseNumericTypeTest` and `QueryResponseSectionParityTest` are the unit tests you are reacting to.
   
   On randomizing `wt` in existing tests: that is the better shape, and it would cover far more than these tests do. Before I rewrite it that way I want to check what it costs — a good number of tests assert on the binary form specifically, so the randomization would have to be scoped rather than global. I'll report back with what I find rather than guessing at it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3986397520


##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/json/JsonMapResponseParser.java:
##########
@@ -78,12 +103,13 @@ public Set<String> getContentTypes() {
    */
   @Override
   public SolrParams getAdditionalRequestParams() {
-    return REQUEST_PARAMS;
+    return canonical ? REQUEST_PARAMS : null;

Review Comment:
   Renamed to `CANONICAL_PARAMS`.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseCanonicalizer.java:
##########


Review Comment:
   Renamed to `ResponseCanonicalizer`, and `normalize`/`normalizeValue` with it -- otherwise `ResponseCanonicalizer.normalize` keeps the one mismatch the rename removes, with `processCanonicalResponse`, `canonical()` and the `canonical` field already next to it. The class is new in this PR, so nothing outside it is reached.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3704065183


##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.util;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+
+/**
+ * Converts a parsed response into the canonical shape the SolrJ response classes expect (the shape
+ * the binary and XML parsers produce): nested JSON objects become {@link NamedList}s and a {@code
+ * {numFound, docs}} object becomes a {@link SolrDocumentList}.
+ *
+ * <p>Only unambiguous, self-describing conversions are performed. It is a no-op for values already
+ * in canonical form (so binary/XML responses pass through unchanged). It does not attempt to
+ * interpret the ambiguous flat arrays produced by {@code json.nl=flat}; a typed JSON parser should
+ * request {@code json.nl=map} for its own reads.
+ */
+public final class ResponseNormalizer {
+
+  private ResponseNormalizer() {}
+
+  /** Returns a normalized copy of the given response NamedList. */
+  public static NamedList<Object> normalize(NamedList<Object> response) {
+    if (response == null) {
+      return null;
+    }
+    SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(response.size());
+    for (Map.Entry<String, Object> e : response) {
+      out.add(e.getKey(), normalizeValue(e.getValue()));
+    }
+    return out;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static Object normalizeValue(Object val) {
+    if (val instanceof SolrDocumentList || val instanceof SolrDocument) {
+      // Already canonical (binary/XML produce these directly); leave untouched. Must precede the
+      // List/Map branches since SolrDocumentList is a List and SolrDocument is a Map.
+      return val;
+    } else if (val instanceof NamedList) {

Review Comment:
   >Two things I checked ...
   
   Yes this is by design.  It's why SOM exists.  It's for efficient response formulation -- no safety check, and conveys a Map and thus it renders different with JSON.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5638165651

   Or CanonicalJsonResponseParser as a separate class.  Subclassing JsonMap suggests it *is a* Json Map ResponseParser but actually it's NamedList etc. oriented.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3700109843


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/AdminResponseNumericTypeTest.java:
##########


Review Comment:
   Randomized `wt` in `TestSuggesterResponse` — it already randomized javabin/xml for SOLR-15070, so this adds the JSON parser to that. Removed `QueryResponseCrossFormatTest`.
   
   `SolrExampleTests` has a subclass per parser and JSON was missing, so `SolrExampleJsonMapTest` now runs all 42 against `JsonMapResponseParser`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3704476129


##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -229,6 +230,9 @@ protected NamedList<Object> processErrorsAndResponse(
       NamedList<Object> rsp;
       try {
         rsp = processor.processResponse(is, encoding);
+        if (!processor.producesCanonicalForm()) {

Review Comment:
   hey David, deal :) (not bot)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3696530420


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");

Review Comment:
   is this necessary?
   FYI I'm hoping for this to go away basically... there's a JIRA and a PR albeit presently it's stalled:  https://github.com/apache/solr/pull/4215#issuecomment-4329198060



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/AdminResponseNumericTypeTest.java:
##########


Review Comment:
   TBH I'd rather see an integration test, showing something working realisitcally that previously didn't.  I'm not a fan of unit tests like this.
   
   I can't tell yet as I'm coming to understand this PR still but it's possible there's existting tests that just need to randomly pick the "wt".  We use randomization a lot, and it can reduce the testing maintenance  & increase net tested exposure IMO.



##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########


Review Comment:
   IMO this goes in the org.apache.solr.client.solrj.response package.  It's not a common utility.



##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.util;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+
+/**
+ * Converts a parsed response into the canonical shape the SolrJ response classes expect (the shape
+ * the binary and XML parsers produce): nested JSON objects become {@link NamedList}s and a {@code
+ * {numFound, docs}} object becomes a {@link SolrDocumentList}.
+ *
+ * <p>Only unambiguous, self-describing conversions are performed. It is a no-op for values already
+ * in canonical form (so binary/XML responses pass through unchanged). It does not attempt to
+ * interpret the ambiguous flat arrays produced by {@code json.nl=flat}; a typed JSON parser should
+ * request {@code json.nl=map} for its own reads.
+ */
+public final class ResponseNormalizer {
+
+  private ResponseNormalizer() {}
+
+  /** Returns a normalized copy of the given response NamedList. */
+  public static NamedList<Object> normalize(NamedList<Object> response) {
+    if (response == null) {
+      return null;
+    }
+    SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(response.size());
+    for (Map.Entry<String, Object> e : response) {
+      out.add(e.getKey(), normalizeValue(e.getValue()));
+    }
+    return out;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static Object normalizeValue(Object val) {
+    if (val instanceof SolrDocumentList || val instanceof SolrDocument) {
+      // Already canonical (binary/XML produce these directly); leave untouched. Must precede the
+      // List/Map branches since SolrDocumentList is a List and SolrDocument is a Map.
+      return val;
+    } else if (val instanceof NamedList) {

Review Comment:
   We should not (on principle) replace every NamedList with a SimpleOrderedMap.  Some NL's are a SOM; some are not.  A SOM conveys unique keys that may not be true with a general NL.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");
+    solrTestRule.startSolr();
+    solrTestRule.newCollection().withConfigSet(ExternalPaths.TECHPRODUCTS_CONFIGSET).create();
+
+    try (SolrClient client = solrTestRule.getSolrClient()) {
+      client.add(
+          java.util.List.of(

Review Comment:
   I believe we have linting rules prohibiting FQNs



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseSectionParityTest.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.ResponseNormalizer;
+import org.junit.Test;
+
+/**
+ * Each test feeds a JSON (json.nl=map) response for one QueryResponse section through the
+ * normalizer and asserts the typed accessor works. Sections with a numeric cast (grouping, facets,
+ * spellcheck) also guard the Number widening; the rest guard the structural Map -&gt; NamedList /
+ * SolrDocumentList reconstruction the section relies on.
+ */
+public class QueryResponseSectionParityTest extends SolrTestCase {
+
+  private static QueryResponse parse(String json) throws Exception {
+    NamedList<Object> parsed =
+        new JsonMapResponseParser()
+            .processResponse(
+                new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8)), "UTF-8");
+    QueryResponse r = new QueryResponse();
+    r.setResponse(ResponseNormalizer.normalize(parsed));
+    return r;
+  }
+
+  private static final String HEADER = "\"responseHeader\":{\"status\":0,\"QTime\":1},";
+
+  /** pivot facets: count is an Integer cast (QueryResponse readPivots). */
+  @Test
+  public void testPivotFacets() throws Exception {
+    String json =
+        "{"

Review Comment:
   We're using Java 17 here; please use multi-line strings



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");
+    solrTestRule.startSolr();
+    solrTestRule.newCollection().withConfigSet(ExternalPaths.TECHPRODUCTS_CONFIGSET).create();
+
+    try (SolrClient client = solrTestRule.getSolrClient()) {

Review Comment:
   Doesn't need to be closed; the rule manages that.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -229,6 +230,9 @@ protected NamedList<Object> processErrorsAndResponse(
       NamedList<Object> rsp;
       try {
         rsp = processor.processResponse(is, encoding);
+        if (!processor.producesCanonicalForm()) {

Review Comment:
   hmm; this seems bolted on.  Could we instead have processor.processCanonicalResponse that internally has this similar logic?  This puts the processor in charge.  No `processor.producesCanonicalForm` predicate would be needed.



##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.util;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+
+/**
+ * Converts a parsed response into the canonical shape the SolrJ response classes expect (the shape
+ * the binary and XML parsers produce): nested JSON objects become {@link NamedList}s and a {@code
+ * {numFound, docs}} object becomes a {@link SolrDocumentList}.
+ *
+ * <p>Only unambiguous, self-describing conversions are performed. It is a no-op for values already
+ * in canonical form (so binary/XML responses pass through unchanged). It does not attempt to
+ * interpret the ambiguous flat arrays produced by {@code json.nl=flat}; a typed JSON parser should
+ * request {@code json.nl=map} for its own reads.
+ */
+public final class ResponseNormalizer {
+
+  private ResponseNormalizer() {}
+
+  /** Returns a normalized copy of the given response NamedList. */
+  public static NamedList<Object> normalize(NamedList<Object> response) {
+    if (response == null) {
+      return null;
+    }
+    SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(response.size());
+    for (Map.Entry<String, Object> e : response) {
+      out.add(e.getKey(), normalizeValue(e.getValue()));
+    }
+    return out;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static Object normalizeValue(Object val) {
+    if (val instanceof SolrDocumentList || val instanceof SolrDocument) {
+      // Already canonical (binary/XML produce these directly); leave untouched. Must precede the
+      // List/Map branches since SolrDocumentList is a List and SolrDocument is a Map.
+      return val;
+    } else if (val instanceof NamedList) {
+      // Already canonical (binary/XML), but its children may still need normalizing.
+      NamedList<Object> in = (NamedList<Object>) val;
+      SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(in.size());
+      for (Map.Entry<String, Object> e : in) {
+        out.add(e.getKey(), normalizeValue(e.getValue()));
+      }
+      return out;
+    } else if (val instanceof Map) {
+      Map<String, Object> m = (Map<String, Object>) val;

Review Comment:
   We're using Java 17 which supports cast-less instanceof checks.  Broadly your PR would benefit from updated Java.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "sergeybzhezitskiy (via GitHub)" <gi...@apache.org>.
sergeybzhezitskiy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3716773320


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/TestSuggesterResponse.java:
##########


Review Comment:
   David, thanks a lot for sharing it. i will dive deeper into it, it's very interesting approach...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3708821229


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/TestSuggesterResponse.java:
##########


Review Comment:
   thanks for this.  It's good.  
   
   However, note that most of Lucene & Solr's randomized testing is done at a deeper level such that an individual test generally doesn't even have to do anything to get the randomization -- it just happens at a deeper test framework/infra level.  For example... imagine if the default was a settable static supplier... and imagine if SolrTestCase were to set it.  Then the useful test coverage would go through the roof (thousands of Solr tests) and we'd probably toss aside more of your tests as redundant.  I'm hesitant to truly recommend precisely this... but I'm at least taking the educational opportunity of sharing the rather unique randomized testing philosophy that permeates the Lucene & Solr projects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3998847779


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,11 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects can now read a JSON response, via the new
+  `CanonicalJsonResponseParser`; previously their accessors could throw ClassCastException.

Review Comment:
   Applied.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleJsonTest.java:
##########


Review Comment:
   Renamed to SolrExampleJsonTest.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5686825922

   Ugh; should not have merged:
   
   ----
   
   The integration test failure ([job](https://github.com/apache/solr/actions/runs/34974725157/job/104401142043?pr=4640)) isn't flaky; it's caused by this PR.
   
   **What failed:** `test_ssl.bats:124` ("use different hostname when not checking peer-name"). With `solr.jetty.ssl.sni.host.check.enabled=true`, the test runs `solr api` against `https://localhost`. Jetty rejects that with HTTP **400** "Invalid SNI". The test expects `solr api` to exit nonzero, but it exited 0.
   
   **Why:** `ApiTool.callGet` changed from `JsonMapResponseParser` to `InputStreamResponseParser("json")`. In `HttpSolrClient.processErrorsAndResponse`:
   - A 400 status doesn't throw by itself.
   - The old parser declared `application/json`, so Jetty's HTML error page failed `checkContentType`, which threw `RemoteSolrException` with the body ("Invalid SNI"). The CLI then exited nonzero.
   - `InputStreamResponseParser.getContentTypes()` returns `Set.of()` ("don't enforce"), and `wantStream()` returns the raw stream. The HTML error page comes back as normal output and the exit code is 0.
   
   The earlier peer-name check (line 116) still passes because that one fails during the TLS handshake, before any HTTP response exists.
   
   **Possible fix:** in `ApiTool`, read the `responseStatus` from the NamedList that `createInputStreamNamedList(httpStatus, is)` returns, and throw if it isn't 2xx. That keeps the raw pass-through and still fails on error responses. It's also worth checking whether other `InputStreamResponseParser` callers in this PR (e.g. `PackageUtils`) now let 400s through silently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3966573592


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/ResponseNormalizerTest.java:
##########


Review Comment:
   Both forms are covered. Named children arrive stamped with `_nest_path_`/`_nest_parent_` and are rebuilt as child documents -- `testNamedNestedDocumentsAreReconstructed`, with paths like `/toppings#0` and a grandchild. Anonymous ones arrive as a `_childDocuments_` list -- `testChildDocumentsAreReconstructed` and `testChildDocumentsNest`. If the anonymous form is deprecated, that branch and its two tests go with it.
   



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/TestSuggesterResponse.java:
##########


Review Comment:
   That philosophy is worth the detour, thanks. The scale is also the reason I did not reach for it here: 592 classes extend SolrTestCaseJ4 and 264 SolrCloudTestCase, and nothing in the test framework randomizes the response parser today, so making the default a settable supplier flips ~850 classes in one step. That deserves its own change and its own failure list. Randomizing `wt` in TestSuggesterResponse is as far as I took it in this PR.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3941995362


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,12 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects now work when the client is configured with a
+  non-binary response parser (such as the JSON parser); previously their accessors could throw a
+  ClassCastException.
+type: fixed

Review Comment:
   ```suggestion
   type: added
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3998848307


##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseParser.java:
##########
@@ -67,4 +82,19 @@ public abstract NamedList<Object> processResponse(InputStream body, String encod
    * @return the MIME types that this parser is capable of parsing. Never null.
    */
   public abstract Set<String> getContentTypes();
+
+  /**
+   * Parses the response and returns it in the canonical shape the SolrJ response classes expect: a
+   * {@link NamedList} tree with {@link org.apache.solr.common.SolrDocumentList} for document
+   * sections.
+   *
+   * <p>Most parsers produce that shape directly and inherit this method unchanged. A parser whose
+   * natural output is a raw structure of {@code Map}s and {@code List}s — such as the JSON map
+   * parser — overrides it to convert, so that the conversion is the parser's own responsibility
+   * rather than something a client has to know to apply.
+   */
+  public NamedList<Object> processCanonicalResponse(InputStream body, String encoding)

Review Comment:
   Agreed, removed. CanonicalJsonResponseParser now overrides processResponse directly; HttpSolrClient and EmbeddedSolrServer call processResponse instead of processCanonicalResponse. Checked the other direct processResponse callers first: the two tests that call it on a JSON parser construct JsonMapResponseParser, not the canonical subclass, so unaffected; ConcurrentUpdateBaseSolrClient's error-parsing path already branches on NamedList vs List for exactly this raw/canonical ambiguity, so a canonical result reaching it now is not a behaviour change worth guarding against.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "sergeybzhezitskiy (via GitHub)" <gi...@apache.org>.
sergeybzhezitskiy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3716773320


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/TestSuggesterResponse.java:
##########


Review Comment:
   David, thanks a lot for sharing it. i will dive deeper into it, it's very interesting approach...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3704053118


##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java:
##########
@@ -229,6 +230,9 @@ protected NamedList<Object> processErrorsAndResponse(
       NamedList<Object> rsp;
       try {
         rsp = processor.processResponse(is, encoding);
+        if (!processor.producesCanonicalForm()) {

Review Comment:
   I'd like to make a claim/observation, at the risk of being wrong and a little embarrassed.  Your PR comment *responses* are AI generated, at least this and some others were.  Yes?  I think it's strongly preferable for AI generated responses to humans to be somehow marked as such.  FWIW at work we have a footer and/or marker emoji.  I went to know the difference between Serhiy, in the flesh whom I'm getting to know, and a bot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3985857998


##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseNormalizer.java:
##########


Review Comment:
   if "canonicalize" is the name choice we're going with on JsonMapResponseParser, then I suppose this class here should be renamed to ResponseCanonicalizer



##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/json/JsonMapResponseParser.java:
##########
@@ -78,12 +103,13 @@ public Set<String> getContentTypes() {
    */
   @Override
   public SolrParams getAdditionalRequestParams() {
-    return REQUEST_PARAMS;
+    return canonical ? REQUEST_PARAMS : null;

Review Comment:
   renaming that constant to CANONICAL_PARAMS would be good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3997900965


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,11 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects can now read a JSON response, via the new
+  `CanonicalJsonResponseParser`; previously their accessors could throw ClassCastException.

Review Comment:
   ```suggestion
     `CanonicalJsonResponseParser`..
   ```



##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseParser.java:
##########
@@ -67,4 +82,19 @@ public abstract NamedList<Object> processResponse(InputStream body, String encod
    * @return the MIME types that this parser is capable of parsing. Never null.
    */
   public abstract Set<String> getContentTypes();
+
+  /**
+   * Parses the response and returns it in the canonical shape the SolrJ response classes expect: a
+   * {@link NamedList} tree with {@link org.apache.solr.common.SolrDocumentList} for document
+   * sections.
+   *
+   * <p>Most parsers produce that shape directly and inherit this method unchanged. A parser whose
+   * natural output is a raw structure of {@code Map}s and {@code List}s — such as the JSON map
+   * parser — overrides it to convert, so that the conversion is the parser's own responsibility
+   * rather than something a client has to know to apply.
+   */
+  public NamedList<Object> processCanonicalResponse(InputStream body, String encoding)

Review Comment:
   This no longer needs to exist; right?  Most Response Parsers are "canonical" by definition; the "JsonMap" one is not.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleJsonMapTest.java:
##########


Review Comment:
   no longer named suitably.  Just ditch the "Map" part IMO.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5595312331

   There are a number of genuine test failures to be resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3708850909


##########
solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleJsonMapTest.java:
##########
@@ -14,28 +14,19 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.solr.client.solrj.response;
+package org.apache.solr.client.solrj;
 
-import org.apache.solr.SolrTestCase;
+import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
 import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
-import org.junit.Test;
 
-/**
- * Pins the producesCanonicalForm() contract that gates response normalization: only the JSON map
- * parser (which yields raw Maps/Lists) needs normalizing; the parsers that already produce the
- * canonical NamedList/SolrDocumentList shape must report true so they pass through untouched.
- */
-public class ResponseParserCanonicalFormTest extends SolrTestCase {
-
-  @Test
-  public void testCanonicalParsersReportTrue() {
-    assertTrue(new JavaBinResponseParser().producesCanonicalForm());
-    assertTrue(new XMLResponseParser().producesCanonicalForm());
-    assertTrue(new InputStreamResponseParser("json").producesCanonicalForm());
-  }
-
-  @Test
-  public void testJsonMapParserReportsFalse() {
-    assertFalse(new JsonMapResponseParser().producesCanonicalForm());
+/** Runs the example tests over {@link JsonMapResponseParser}. */

Review Comment:
   Niiiiice.... :-) thank you.  Should be getting lots more coverage here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3699950183


##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");
+    solrTestRule.startSolr();
+    solrTestRule.newCollection().withConfigSet(ExternalPaths.TECHPRODUCTS_CONFIGSET).create();
+
+    try (SolrClient client = solrTestRule.getSolrClient()) {
+      client.add(
+          java.util.List.of(

Review Comment:
   Fixed — `java.util.List` imported.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseJsonParserIntegrationTest.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import static org.apache.solr.SolrTestCaseJ4.sdoc;
+
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.util.ExternalPaths;
+import org.apache.solr.util.SolrJettyTestRule;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+/**
+ * End-to-end: a real HTTP query with the JSON map response parser must return a fully typed
+ * QueryResponse, proving SolrRequest.process() normalizes the non-canonical JSON response at the
+ * boundary before the response classes read it (SOLR-17316).
+ */
+public class QueryResponseJsonParserIntegrationTest extends SolrTestCase {
+
+  @ClassRule public static SolrJettyTestRule solrTestRule = new SolrJettyTestRule();
+
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+    System.setProperty("solr.security.allow.paths", "*");
+    solrTestRule.startSolr();
+    solrTestRule.newCollection().withConfigSet(ExternalPaths.TECHPRODUCTS_CONFIGSET).create();
+
+    try (SolrClient client = solrTestRule.getSolrClient()) {

Review Comment:
   Fixed. The javadoc on `getSolrClient(String)` says it outright — "The caller doesn't need to close it" — so the try-with-resources is gone.



##########
solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseSectionParityTest.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.response;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import org.apache.solr.SolrTestCase;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.ResponseNormalizer;
+import org.junit.Test;
+
+/**
+ * Each test feeds a JSON (json.nl=map) response for one QueryResponse section through the
+ * normalizer and asserts the typed accessor works. Sections with a numeric cast (grouping, facets,
+ * spellcheck) also guard the Number widening; the rest guard the structural Map -&gt; NamedList /
+ * SolrDocumentList reconstruction the section relies on.
+ */
+public class QueryResponseSectionParityTest extends SolrTestCase {
+
+  private static QueryResponse parse(String json) throws Exception {
+    NamedList<Object> parsed =
+        new JsonMapResponseParser()
+            .processResponse(
+                new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8)), "UTF-8");
+    QueryResponse r = new QueryResponse();
+    r.setResponse(ResponseNormalizer.normalize(parsed));
+    return r;
+  }
+
+  private static final String HEADER = "\"responseHeader\":{\"status\":0,\"QTime\":1},";
+
+  /** pivot facets: count is an Integer cast (QueryResponse readPivots). */
+  @Test
+  public void testPivotFacets() throws Exception {
+    String json =
+        "{"

Review Comment:
   Fixed — all eight JSON literals are text blocks now, which also removed every escaped quote in the file.



##########
solr/solrj/src/java/org/apache/solr/client/solrj/response/ResponseNormalizer.java:
##########


Review Comment:
   Moved to `org.apache.solr.client.solrj.response`, along with its test.



##########
solr/solrj/src/java/org/apache/solr/common/util/ResponseNormalizer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.util;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+
+/**
+ * Converts a parsed response into the canonical shape the SolrJ response classes expect (the shape
+ * the binary and XML parsers produce): nested JSON objects become {@link NamedList}s and a {@code
+ * {numFound, docs}} object becomes a {@link SolrDocumentList}.
+ *
+ * <p>Only unambiguous, self-describing conversions are performed. It is a no-op for values already
+ * in canonical form (so binary/XML responses pass through unchanged). It does not attempt to
+ * interpret the ambiguous flat arrays produced by {@code json.nl=flat}; a typed JSON parser should
+ * request {@code json.nl=map} for its own reads.
+ */
+public final class ResponseNormalizer {
+
+  private ResponseNormalizer() {}
+
+  /** Returns a normalized copy of the given response NamedList. */
+  public static NamedList<Object> normalize(NamedList<Object> response) {
+    if (response == null) {
+      return null;
+    }
+    SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(response.size());
+    for (Map.Entry<String, Object> e : response) {
+      out.add(e.getKey(), normalizeValue(e.getValue()));
+    }
+    return out;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static Object normalizeValue(Object val) {
+    if (val instanceof SolrDocumentList || val instanceof SolrDocument) {
+      // Already canonical (binary/XML produce these directly); leave untouched. Must precede the
+      // List/Map branches since SolrDocumentList is a List and SolrDocument is a Map.
+      return val;
+    } else if (val instanceof NamedList) {
+      // Already canonical (binary/XML), but its children may still need normalizing.
+      NamedList<Object> in = (NamedList<Object>) val;
+      SimpleOrderedMap<Object> out = new SimpleOrderedMap<>(in.size());
+      for (Map.Entry<String, Object> e : in) {
+        out.add(e.getKey(), normalizeValue(e.getValue()));
+      }
+      return out;
+    } else if (val instanceof Map) {
+      Map<String, Object> m = (Map<String, Object>) val;

Review Comment:
   Fixed throughout `ResponseNormalizer` — `instanceof NamedList<?> in`, `instanceof Map<?,?> raw`, `instanceof List<?> in`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #4640:
URL: https://github.com/apache/solr/pull/4640#discussion_r3942034667


##########
changelog/unreleased/SOLR-17316-response-parsers.yml:
##########
@@ -0,0 +1,12 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+
+title: >
+  SolrJ's QueryResponse and other response objects now work when the client is configured with a
+  non-binary response parser (such as the JSON parser); previously their accessors could throw a
+  ClassCastException.

Review Comment:
   ```suggestion
     JSON response parser (namely `JsonMapResponseParser`);
     previously their accessors could throw ClassCastException.
   ```
   
   I think we should just name JsonMapResponseParser specifically, as we know this works.  JacksonDataBindResponseParser probably doesn't work.  Albeit it'd be interesting to try experimentally.  I'm definitely keen on use of Jackson for CBOR support.  For Solr 11, I hope we can use Jackson inside JsonMapResponseParser instead of noggit, but that's a separate discussion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5599293955

   Five tests were failing, and they predate the main merge: main alone is green, this branch alone fails the same five.
   
   Four came from one decision in this PR -- canonicalising in the client for every request. `JsonMapResponseParser` is public, and twenty existing call sites read the Maps and Lists it has always returned, so the conversion changed their contract silently; the four that broke were the ones that happened to have tests. The parser now keeps its old behaviour and `canonical()` offers the conversion to the ten callers that pass the response to a response object. That is the same reasoning you gave against replacing every NamedList with a SimpleOrderedMap, applied one level down.
   
   The fifth was a plain bug: `SolrParams.of` rejects a null value, so a parser asking for no `wt` threw where `ModifiableSolrParams.set` had omitted the param.
   
   `solr:core` and `solrj` are green -- 7,103 tests, 0 failures -- and `check -x test` passes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #4640:
URL: https://github.com/apache/solr/pull/4640#issuecomment-5643803855

   Done: `CanonicalJsonResponseParser extends JsonMapResponseParser`, overriding `getAdditionalRequestParams` and `processCanonicalResponse`. The boolean and the `canonical()` factory are gone, and `JsonMapResponseParser` is back to what main has (0-line diff). Two tests had used `canonical()` with `processResponse`, where the flag changed nothing; they use the plain parser now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] SOLR-17316: make SolrJ response objects work with non-binary ResponseParsers [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley merged PR #4640:
URL: https://github.com/apache/solr/pull/4640


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org