[jira] [Updated] (SOLR-16751) ClusterState.collectionFromObjects() stores copy of shards in DocCollection.getProperties()

["Chris M. Hostetter (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris M. Hostetter updated SOLR-16751:
--------------------------------------
    Attachment: SOLR-16751.patch
        Status: Open  (was: Open)

I'm not sure why/how this could cause any errors in functionality, since all of the code I found for _consuming_ {{state.json}} seemed to be doing a "last key wins" when parsing the JSON, and once a {{DocCollection}} exists, I _think_ all the consumers are correctly dealing with {{DocCollection.getSlices()}} , and never looking at (or trying to mutate) a {{"shards"}} key in {{DocCollection.getProperties()}} ...

...and yet...

We have seen a big spike in recent jenkins failures (with seeds I haven't been able to reproduce) in {{SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull}} – the timing of which seems to line up with GIT:2ac7ed29563a33d9f9a31737996a1d4cfb0fca0d.

If i'm right, we should see a big drop in failures from {{SplitShardWithNodeRoleTest}} if we commit the attached patch.

> ClusterState.collectionFromObjects() stores copy of shards in DocCollection.getProperties()
> -------------------------------------------------------------------------------------------
>
>                 Key: SOLR-16751
>                 URL: https://issues.apache.org/jira/browse/SOLR-16751
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Chris M. Hostetter
>            Priority: Major
>         Attachments: SOLR-16751.patch
>
>
> The way a {{DocCollection}} is created by {{ClusterState.collectionFromObjects()}} ( ie: {{ClusterState}} is read from a JSON payload – such as {{state.json}} ) a copy of the original {{"shards"}} mappings is left in the {{props}} argument (where it's carried around in {{DocCollection.getProperties()}} ) in addition to the (correct) {{slices}} argument (which is used to build {{DocCollection.getSlices()}} )
> This is due to the following lines of code, which AFAICT should have been done in reverse order...
> {code:java}
>       props = new HashMap<>(objs);
>       objs.remove(CollectionStateProps.SHARDS);
> {code}
> This bug seems to have existed for a while, but seems to have slipped under the radar since nothing has any reason to be looking for a {{"shards"}} key in {{DocCollection.getProperties()}} .
> Recently however, noble committed 2ac7ed29563a33d9f9a31737996a1d4cfb0fca0d which changed the serialization logic for {{DocCollection}} to eliminate a temporary map...
> {noformat}
> -    LinkedHashMap<String, Object> all = CollectionUtil.newLinkedHashMap(slices.size() + 1);
> -    all.putAll(propMap);
> -    all.put(CollectionStateProps.SHARDS, slices);
> -    jsonWriter.write(all);
> +    jsonWriter.write(
> +        (MapWriter)
> +            ew -> {
> +              propMap.forEach(ew.getBiConsumer());
> +              ew.put(CollectionStateProps.SHARDS, slices);
> +            });
> {noformat}
> ... so now, instead of an {{all}} Map where the {{"shards"}} key from {{propMap}} being (correctly) overridden by {{slices}} and then serialized, now the JSON output contains two entries with the {{"shards"}} key.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org