You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/05/11 04:53:19 UTC
[GitHub] [solr] anshumg opened a new pull request, #851: Upgrade notes and major changes cleanup and update
anshumg opened a new pull request, #851:
URL: https://github.com/apache/solr/pull/851
This PR is against `main` but should be easy to just copy and paste for branch_9x and branch_9_0.
I was a little confused if we wanted to leave out some details, but based on the conversations over slack with a few other committers I felt folks preferred more over less with these pages. It's still a much summarized version of the change log and way more detailed than just the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870266176
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
Review Comment:
```suggestion
* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use `clusterstate.json` you will need to take some steps, described elsewhere in this document.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870279005
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
Review Comment:
@dsmiley please have a look at whether this description looks right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy merged pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy merged PR #851:
URL: https://github.com/apache/solr/pull/851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870645850
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
Review Comment:
Started https://github.com/apache/solr/pull/858 for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870211507
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
Review Comment:
```suggestion
* JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
Review Comment:
No need for JIRA issue numbers
```suggestion
* Facet count in Json Facet module always returns a `long` value, irrespective of number of shards.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
Review Comment:
```suggestion
* Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
Review Comment:
There was a change in [SOLR-14401](https://issues.apache.org/jira/browse/SOLR-14401) which ended up with using `internal=true` instead of `[shard]` syntax
```suggestion
* Only `SearchHandler` and subclasses have "local" metrics now. It's now tracked with a new `internal=true` tag, which can be filtered in e.g. Prometheus.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
Review Comment:
```suggestion
* The property `blockUnknown` in the `BasicAuthPlugin` and the `JWTAuthPlugin` now defaults to `true` instead of `false`. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
Review Comment:
```suggestion
This may require some more disk space for logs than was the case in version 8.x. See xref:deployment-guide:configuring-logging.adoc[Configuring Logging] for how to change this.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
Review Comment:
```suggestion
* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See `CloudLegacySolrClient` for the 8.x version of this class.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
Review Comment:
```suggestion
* xref:configuration-guide:solr-modules.adoc[Contrib modules] are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
Review Comment:
```suggestion
* The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
Review Comment:
```suggestion
Additionally, users are advised to use the `InjectionDefense` class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the `expr` parameter can be reinstated with `-DStreamingExpressionMacros=true` passed to the JVM at startup
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
Review Comment:
```suggestion
* Jetty Request log is now enabled by default, i.e. logging every request.
```
```suggestion
* xref:deployment-guide:configuring-logging.adoc#request-logging[Jetty Request log] is now enabled by default, i.e. logging every request.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
Review Comment:
```suggestion
* Logging is now asynchronous by default. There's a small window where log messages may be lost in the event of some hard crash.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
Review Comment:
```suggestion
* Highlighting: `hl.method=unified` is the new default. Use `hl.method=original`
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
+* Auto-scaling framework has been removed. Please refer to replica placement plugins for alternate options
Review Comment:
```suggestion
* Auto-scaling framework has been removed. Please refer to xref:configuration-guide:replica-placement-plugins.adoc[Replica Placement Plugins] for alternate options.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
Review Comment:
```suggestion
* `MacroExpander` will no longer will expand URL parameters inside of the `expr` parameter (used by streaming expressions).
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
Review Comment:
```suggestion
* The default port of "Prometheus exporter" has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
Review Comment:
```suggestion
* Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
Review Comment:
```suggestion
* Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
Review Comment:
```suggestion
* xref:deployment-guide:rate-limiters.adoc[Rate limiting] provides a way to throttle update and search requests based on usage metrics.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
Review Comment:
Should the "Other" section be at the end?
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
Review Comment:
```suggestion
* New xref:deployment-guide:cert-authentication-plugin.adoc[Certificate Authentication Plugin], enabling end-to-end use of x509 client certificates for Authentication and Authorization.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
Review Comment:
```suggestion
* xref:query-guide:dense-vector-search.adoc[Dense Vector "Neural" Search] through `DenseVectorField` fieldType and K-Nearest-Neighbor (KNN) Query Parser.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
Review Comment:
```suggestion
* All request handlers support security permissions. Users may have to adapt their `security.json`.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
Review Comment:
Too bold claim - we can instead say that our dependencies are now more secure:
```suggestion
* Dependency updates - A lot of dependency updates removes several security issues from dependencies, and thus make Solr more secure.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
Review Comment:
```suggestion
Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (`solr.in.sh`/`solr.in.cmd`) file.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
Review Comment:
```suggestion
* A new xref:deployment-guide:task-management.adoc[Task management] interface allows declaring tasks as cancellable and trackable.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
Review Comment:
```suggestion
* xref:deployment-guide:jwt-authentication-plugin.adoc[JWTAuthPlugin] has been moved to a module. Users need to add the module `jwt-auth` to classpath. The plugin has also
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
Review Comment:
```suggestion
* Ability to specify xref:deployment-guide:node-roles.adoc[node roles] in Solr. This release supports `overseer` and `data` roles out of the box.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
Review Comment:
Add link and use official name of the feature
```suggestion
* New API for pluggable xref:configuration-guide:replica-placement-plugins.adoc[Replica Placement Plugins] that replaces the auto-scaling framework.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
Review Comment:
```suggestion
* The `solr-extraction` module has been cleaned up to produce `solr-extraction-*` jar instead of `solr-cell-*` jars.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
+* Auto-scaling framework has been removed. Please refer to replica placement plugins for alternate options
+* LegacyBM25SimilarityFactory has been removed.
+* Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed. Users have to modify their existing configurations to use CaffeineCache instead.
+* `VelocityResponseWriter` is an independent project now; it is no longer a part of Solr. This encompasses all previously included /browse and wt=velocity examples
Review Comment:
```suggestion
* `VelocityResponseWriter` is an independent project now; it is no longer a part of Solr. This encompasses all previously included `/browse` and `wt=velocity` examples.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
+* Auto-scaling framework has been removed. Please refer to replica placement plugins for alternate options
+* LegacyBM25SimilarityFactory has been removed.
+* Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed. Users have to modify their existing configurations to use CaffeineCache instead.
+* `VelocityResponseWriter` is an independent project now; it is no longer a part of Solr. This encompasses all previously included /browse and wt=velocity examples
+* Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
Review Comment:
Duplicate entry
```suggestion
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
+* Auto-scaling framework has been removed. Please refer to replica placement plugins for alternate options
+* LegacyBM25SimilarityFactory has been removed.
Review Comment:
```suggestion
* `LegacyBM25SimilarityFactory` has been removed.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
Review Comment:
```suggestion
* `SolrMetricProducer` / `SolrInfoBean` APIs have changed and third-party components that implement these APIs need to be updated.
```
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
Review Comment:
```suggestion
* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use `clusterstate.json`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on PR #851:
URL: https://github.com/apache/solr/pull/851#issuecomment-1123772369
Thanks @anshumg . I committed a bunch of adjustments, and will now merge and backport so it is ready for release.
Please open further PRs for followup edits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870215743
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
Review Comment:
```suggestion
* Hadoop authentication support has been moved to the new `hadoop-auth` module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870625792
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
Review Comment:
Your change is from the viewpoint of our prometheus configuration that we ship with 9. But for Solr directly, you removed what I wrote which is accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870279766
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
+There are no longer ".distrib." named metrics; all metrics are assumed to be such except "[shard]". The default Prometheus exporter config splits that component to a new label named "internal". The sample Grafana dashboard now filters to include or exclude this.
+* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
+* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
+Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
+* Log4J configuration & Solr MDC values - link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, without the special single character prefixes that were included in past version. The default `log4j2.xml` configuration file for Solr has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`. Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
+* Jetty Request log is now enabled by default, i.e. logging every request.
+* The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
* SOLR-15914: Solr modules (formerly known as contribs) can now easily be enabled by an environment variable (e.g. in `solr.in.sh` or `solr.in.cmd`) or as a system property (e.g. in `SOLR_OPTS`). Example: `SOLR_MODULES=extraction,ltr`.
-== Deprecations & Removed Features
-
-The following list of features have been permanently removed from Solr:
-
-* SOLR-14656: Autoscaling framework removed.
-This includes:
-** Autoscaling, policy, triggers etc.
-** withCollection handling (SOLR-14964)
-** UTILIZENODE command
-** Sim framework
-** Suggestions tab in UI
-** Reference guide pages for autoscaling
-** autoAddReplicas feature
-
-* SOLR-14783: Data Import Handler (DIH) has been removed from Solr.
-The community package is available at: https://github.com/rohitbemax/dataimporthandler
-
-* SOLR-12823: Remove `/clusterstate.json` support, i.e., support for collections created with `stateFormat=1` as well as support for Collection API MIGRATESTATEFORMAT action.
-Also removes support for cluster property `legacyCloud` (as if always false now).
-
-* SOLR-14792: VelocityResponseWriter has been removed from Solr.
-This encompasses all previous included `/browse` and `wt=velocity` examples.
-This feature has been migrated to an installable package at https://github.com/erikhatcher/solr-velocity
-
-* SOLR-13817: Legacy SolrCache implementations (LRUCache, LFUCache, FastLRUCache) have been removed.
-Users have to modify their existing configurations to use CaffeineCache instead. (ab)
-
-* CDCR
-
-* Solr's blob store
-** SOLR-14654: plugins cannot be loaded using "runtimeLib=true" option. Use the package manager to use and load plugins
-
-* Metrics History
-
-* SOLR-15470: The binary distribution no longer contains test-framework jars.
-
-* SOLR-15203: Remove the deprecated `jwkUrl` in favour of `jwksUrl` when configuring JWT authentication.
-
-* SOLR-12847: maxShardsPerNode parameter has been removed because it was broken and inconsistent with other replica placement strategies.
+=== Deprecations and Removals
+* The Data Import Handler (DIH) is an independent project now; it is no longer a part of Solr.
+* No more support for `clusterstate.json` and `MIGRATESTATE` API has been removed. If your collections use clusterstate.json, please refer to the Upgrade Notes.
Review Comment:
Feels strange to refer to "upgrade notes" when already there :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #851: Upgrade notes and major changes cleanup and update
Posted by GitBox <gi...@apache.org>.
janhoy commented on code in PR #851:
URL: https://github.com/apache/solr/pull/851#discussion_r870641426
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc:
##########
@@ -61,333 +61,126 @@ A rolling upgrade from Solr 8 to Solr 9 requires the following multiple restart
It is always strongly recommended that you fully reindex your documents after a major version upgrade. For details, see the xref:indexing-guide:reindexing.adoc[] section, which covers several strategies for how to reindex.
-In Solr 8, it's possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
-Due to changes in Lucene 9, that isn't possible anymore; the component was removed.
-
-== Solr 9.0 Raw Notes (NOT YET EDITED)
-
-_(raw; not yet edited)_
-
-
-
-
-
-* SOLR-13854, SOLR-13858: SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
-
-* SOLR-14344: Remove Deprecated HttpSolrClient.RemoteSolrException and HttpSolrClient.RemoteExcecutionException.
-All the usages are replaced by BaseHttpSolrClient.RemoteSolrException and BaseHttpSolrClient.RemoteExcecutionException.
-
-* SOLR-15409: Zookeeper client libraries upgraded to 3.7.0, which may not be compatible with your existing server installations
-
-* SOLR-15809: Get rid of blacklist/whitelist terminology. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still
- work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
-
-* SOLR-11623: Every request handler in Solr now implements PermissionNameProvider. Any custom or 3rd party request handler must also do this
-
-* SOLR-14142: Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
- This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
-
-* SOLR-15944: The Tagger's JSON response format now always uses an object/map to represent each tag instead of an array.
-
-* SOLR-15842: Async responses for backups now correctly aggregate and return information.
-In previous versions there was a field returned in async backup status responses, `Response`. This has now been renamed to `msg`, to better fit other collections API responses.
-The `response` field is now a map, containing information about the backup (`startTime`, `indexSizeMB`, `indexFileCount`, etc.).
-
-* SOLR-15982: For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
-Also, both snapshot and incremental backup request responses will now contain `starTime` and `endTime`.
-Snapshot backup shard's response were updated to add fields `indexFileCount` and `endTime`, snapshot delete shard's response were updated to add fields `startTime` and `endTime`.
-Previous fields `fileCount`, `snapshotCompletedAt` and `snapshotDeletedAt` of backup and delete shard's responses are now deprecated and will be removed in future releases.
-All date/time fields of backup and delete related shard's responses have been updated to use `Instance` instead of `Date`, meaning the output will be in the standard ISO 8601 Format.
-
-* SOLR-15884: In Backup request responses, the `response` key now uses a map to return information instead of a list.
-This is only applicable for users returning information in JSON format, which is the default behavior.
-
-* SOLR-14660: HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
-changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
-
-* SOLR-16040: If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
-
-* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related
-changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
-
-* SOLR-15904: SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related
-changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
-
-* SOLR-15950: The folder $SOLR_HOME/userfiles, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
-
-* SOLR-15097: JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
- changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
-
-* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now.
-It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
-There are no longer ".distrib." named metrics; all metrics are assumed to be such except
-"[shard]". The default Prometheus exporter config splits that component to a new label
-named "internal". The sample Grafana dashboard now filters to include or exclude this.
-
-== New Features & Enhancements
-
-// Fill these sub headings
-
-=== Docker
-
-=== Security
-
-=== Scalability
-
-=== Modules
-
-* HDFS
-* Hadoop-auth
-* GCS-repository
-* JWT-auth
-* Scripting
-* SQL
-
-
-=== Gradle build
-
-=== Other
-
-* SOLR-13671: Allow 'var' keyword in Java sources
-
-// TBD
-
-* Replica placement plugins
-
-* Rate limiting and task management
-
-* Certificate Auth Plugin
-
-* SQL Query interface in UI
-
-== Configuration and Default Parameter Changes
-
-// TODO: Move into sub headings
-
-=== Schema Changes in 9.0
-
-=== Indexing Changes in 9.0
-
-=== Query Changes in 9.0
-
-=== Authentication & Security Changes in 9.0
-
-=== UI Changes in 9.0
-
-=== Dependency Updates in 9.0
-
-// RAW notes below
-
-* SOLR-7530: TermsComponent's JSON response format was changed so that "terms" property carries per field arrays by default regardless of distrib, terms.list, terms.ttf parameters.
-This affects JSON based response format but not others
-
-* SOLR-14036: Implicit /terms handler now returns terms across all shards in SolrCloud instead of only the local core.
-Users/apps may be assuming the old behavior.
-A request can be modified via the standard distrib=false param to only use the local core receiving the request.
-
-* SOLR-13783: In situations where a NamedList must be output as plain text, commas between key-value pairs will now be followed by a space (e.g., {shape=square, color=yellow} rather than {shape=square,color=yellow}) for consistency with other `java.util.Map` implementations based on `AbstractMap`.
-
-* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance.
-The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
-
-* SOLR-14012: unique and hll aggregations always returns long value irrespective of standalone or solcloud
-
-* SOLR-11775: Return long value for facet count in Json Facet module irrespective of number of shards
-
-* SOLR-15276: V2 API call to look up async request status restful style of "/cluster/command-status/1000" instead of "/cluster/command-status?requestid=1000".
-
-* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
-
-* SOLR-15471: The language identification "whitelist" configuration is now an "allowlist" to better convey the meaning of the property
-
+In Solr 8, it was possible to add docValues to a schema without re-indexing via `UninvertDocValuesMergePolicy`, an advanced/expert utility.
+Due to changes in Lucene 9, that isn't possible any more.
+
+== Solr 9.0
+=== Querying and Indexing
+* Dense Vector "Neural" Search through DenseVectorField fieldType and K-Nearest-Neighbor (KNN) Query Parser.
+* Admin UI support for SQL Querying.
+* New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, Tamil, and Yiddish.
+* New NorwegianNormalizationFilter
+* Implicit `/terms` handler now returns terms across all shards in SolrCloud instead of only the local core.
+Users/apps may be assuming the old behavior. A request can be modified via the standard `distrib=false` param to only use the local core receiving the request.
+* SQL support has been moved to the sql module. Existing Solr configurations do not need any SQL related changes, however the module needs to be installed - see the section xref:query-guide:sql-query.adoc[].
+* SOLR-11725: JSON aggregations uses corrected sample formula to compute standard deviation and variance. The computation of stdDev and variance in JSON aggregation is same as StatsComponent.
+* SOLR-11775: Facet count in Json Facet module always returns a long value for irrespective of number of shards.
* SOLR-12891: MacroExpander will no longer will expand URL parameters inside of the 'expr' parameter (used by streaming expressions).
-Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection.
-The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
-
-* SOLR-13324: URLClassifyProcessor#getCanonicalUrl now throws MalformedURLException rather than hiding it.
-Although the present code is unlikely to produce such an exception it may be possible in future changes or in subclasses.
-Currently this change should only effect compatibility of custom code overriding this method.
-
-* SOLR-14510: The `writeStartDocumentList` in `TextResponseWriter` now receives an extra boolean parameter representing the "exactness" of the `numFound` value (exact vs approximation).
-Any custom response writer extending `TextResponseWriter` will need to implement this abstract method now (instead previous with the same name but without the new boolean parameter).
-
-* SOLR-15259: hl.fragAlignRatio now defaults to 0.33 to be faster and maybe looks nicer.
-
+Additionally, users are advised to use the 'InjectionDefense' class when constructing streaming expressions that include user supplied data to avoid risks similar to SQL injection. The legacy behavior of expanding the 'expr' parameter can be reinstated with -DStreamingExpressionMacros=true passed to the JVM at startup
* SOLR-9376: The response format for field values serialized as raw XML (via the `[xml]` raw value DocTransformer
and `wt=xml`) has changed. Previously, values were dropped in directly as top-level child elements of each `<doc>`,
obscuring associated field names and yielding inconsistent `<doc>` structure. As of version 9.0, raw values are
wrapped in a `<raw name="field_name">[...]</raw>` element at the top level of each `<doc>` (or within an enclosing
`<arr name="field_name"><raw>[...]</raw></arr>` element for multi-valued fields). Existing clients that parse field
values serialized in this way will need to be updated accordingly.
-
-* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
-
* SOLR-12901: Highlighting: hl.method=unified is the new default. Use hl.method=original
- to switch back if needed.
-
-* SOLR-12055 introduces async logging by default. There's a small window where log messages may be lost in the event of some hard crash.
-Switch back to synchronous logging if this is unacceptable, see comments in the log4j2 configuration files (log4j2.xml by default).
-
-=== solr.xml maxBooleanClauses now enforced recursively
-
-Lucene 9.0 has additional safety checks over previous versions that impact how the `solr.xml` global xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] option is enforced.
-
-In previous versions of Solr, this option was a hard limit on the number of clauses in any `BooleanQuery` object - but it was only enforced for the _direct_ clauses.
-Starting with Solr 9, this global limit is now also enforced against the total number of clauses in a _nested_ query structure.
-
-Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail.
-
-User's in this situation are advised to consider the complexity f their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
-
-=== Log4J configuration & Solr MDC values
-
-link:http://www.slf4j.org/apidocs/org/slf4j/MDC.html[MDC] values that Solr sets for use by Logging calls (such as the collection name, shard name, replica name, etc...) have been modified to now be "bare" values, with out the special single character prefixes that were included in past version.
-For example: In 8.x Log messages for a collection named "gettingstarted" would have an MDC value with a key `collection` mapped to a value of `c:gettingstarted`, in 9.x the value will simply be `gettingstarted`.
-
-Solr's default `log4j2.xml` configuration file has been modified to prepend these same prefixes to MDC values when included in Log messages as part of the `<PatternLayout/>`.
-Users who have custom logging configurations that wish to ensure Solr 9.x logs are consistently formatted after upgrading will need to make similar changes to their logging configuration files. See link:https://issues.apache.org/jira/browse/SOLR-15630[SOLR-15630] for more details.
-
-
-=== base_url removed from stored state
-
-If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
-However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper.
-For background, see: SOLR-12182 and SOLR-15145.
-
-Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
-
-* Solr's distributed tracing no longer incorporates a special `samplePercentage` SolrCloud cluster property.
-Instead, consult the documentation for the tracing system you use on how to sample the traces.
-Consequently, if you use a Tracer at all, you will always have traces and thus trace IDs in logs.
-What percentage of them get reported to a tracing server is up to you.
-
-* JaegerTracerConfigurator no longer recognizes any configuration in solr.xml.
- It is now completely configured via System properties and/or Environment variables as documented by Jaeger.
-
-=== Schema Changes
-
-* `LegacyBM25SimilarityFactory` has been removed.
-
-* SOLR-13593 SOLR-13690 SOLR-13691: Allow to look up analyzer components by their SPI names in field type configuration.
-
-=== Authentication & Security Changes
-
-* The property `blockUnknown` in the BasicAuthPlugin and the JWTAuthPlugin now defaults to `true`.
-This change is backward incompatible.
-If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+to switch back if needed.
+* solr.xml `maxBooleanClauses` is now enforced recursively. Users who upgrade from prior versions of Solr may find that some requests involving complex internal query structures (Example: long query strings using `edismax` with many `qf` and `pf` fields that include query time synonym expansion) which worked in the past now hit this limit and fail. Users in this situation are advised to consider the complexity of their queries/configuration, and increase the value of xref:configuration-guide:configuring-solr-xml#global-maxbooleanclauses[`maxBooleanClauses`] if warranted.
+* Atomic/partial updates to nested documents now _require_ the `\_root_` field to clearly show the document isn't a root document. Solr 8 would fallback on the `\_route_` param but no longer.
+=== Security
+* Certificate Authentication Plugin, enabling end-to-end use of x509 client certificates for Authentication and Authorization.
+* Improved security when using PKI Authentication plugin.
+* Upgrade to Zookeeper 3.7, allowing for TLS protected ZK communication.
+* All request handlers support security permissions for access.
+* Ability to disable admin UI through a system property.
+* The property blockUnknown in the BasicAuthPlugin and the JWTAuthPlugin now defaults to true instead of false. This change is backward incompatible. If you need the pre-9.0 default behavior, you need to explicitly set `blockUnknown:false` in `security.json`.
+* Solr now runs with the Java security manager enabled by default. Hadoop users may need to disable this.
+* Solr now binds to localhost network interface by default for better out of the box security.
+Administrators that need Solr exposed more broadly can change the `SOLR_JETTY_HOST` property in their Solr include (solr.in.sh/solr.in.cmd) file.
+* Solr embedded zookeeper only binds to localhost by default. This embedded zookeeper should not be used in production.
+If you rely upon the previous behavior, then you can change the `clientPortAddress` in `solr/server/solr/zoo.cfg`
+* Jetty low level request-logging in NCSA format is now enabled by default, with a retention of 3 days worth of logs.
+This may require some more disk space for logs than was the case in 8.x. See Reference Guide chapter "Configuring Logging" for how to change this.
+* SOLR-13989: Hadoop authentication support has been moved to the hadoop-auth module. Existing Solr configurations do not need any Hadoop authentication related changes, however the module needs to be installed - see the section xref:deployment-guide:hadoop-authentication-plugin.adoc[].
+* JWTAuthPlugin has been moved to a module. Users need to add the module to classpath. The plugin has also
+changed package name to `org.apache.solr.security.jwt`, but can still be loaded as shortform `class="solr.JWTAuthPlugin"`.
+* Dependency updates - A lot of dependency updates make Solr overall much more secure.
* The allow-list defining allowed URLs for the `shards` parameter is not in the `shardHandler` configuration anymore. It is defined by the `allowUrls` top-level property of the `solr.xml` file. For more information, see xref:configuration-guide:configuring-solr-xml.adoc#allow-urls[Format of solr.allowUrls] documentation.
+* To improve security, `StatelessScriptUpdateProcessorFactory` has been renamed as `ScriptUpdateProcessorFactory` and moved to `modules/scripting` package instead of shipping as part of Solr core.
+* To improve security, `XSLTResponseWriter` has been moved to `modules/scripting` package instead of shipping as part of Solr core.
+
+=== Stability and Scalability
+* Rate limiting provides a way to throttle update and search requests based on usage metrics.
+* Task management interface allows declaring tasks as cancellable and trackable.
+* Ability to specify node roles in Solr. This release supports 'Overseer' and 'Data' roles out of the box.
+* New API for pluggable replica assignment implementations that replaces the auto-scaling framework.
+* Support for distributed processing of cluster state updates and collection API calls, without relying on the Overseer.
+
+=== Build and Docker
+* Solr is now built and released independently of Lucene (separate Apache projects).
+* Build system switched to Gradle, no longer uses Ant + Ivy.
+* Docker image creation is now a part of the Apache Solr GitHub repo.
+* Docker image documentation is now a part of the reference guide.
+* Official Docker image upgraded to use JDK17 (by Eclipse Temurin) and ability to create functionally identical local image.
-* SOLR-13985: Solr's Jetty now binds to localhost network interface by default for better out of the box security.
-Administrators that need Solr exposed more broadly can change the SOLR_JETTY_HOST property in their Solr include (solr.in.sh/solr.in.cmd) file.
-
-* SOLR-14147: Solr now runs with the java security manager enabled by default. Administrators that need to run Solr with Hadoop will need to disable this feature by setting SOLR_SECURITY_MANAGER_ENABLED=false in the environment or in one of the Solr init scripts. Other features in Solr could also break. (Robert Muir, marcussorealheis)
-
-* SOLR-14118: Solr embedded zookeeper only binds to localhost by default.
-This embedded zookeeper should not be used in production.
-If you rely upon the previous behavior, then you can change the clientPortAddress in solr/server/solr/zoo.cfg
-
-=== Module Changes
-
-* **SOLR-15917: "Contrib modules" have been renamed to "Modules", and have been moved from the `contrib/` to `modules/`.**
-Use of these modules remains the same, except for the changes listed below.
-
-* SOLR-15916: `dist/` is no longer provided in the binary release.
-All module jars are now provided under `modules/<name>/lib`, including the module jar and all dependency jars.
-Please update your `<lib>` entries in your `solrconfig.xml` to use this new location.
-More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
-
-* SOLR-14067: `StatelessScriptUpdateProcessorFactory` moved to `modules/scripting` package instead of shipping as part of Solr, due to security concerns.
-Renamed to ScriptUpdateProcessorFactory for simpler name.
-
-* SOLR-15121: `XSLTResponseWriter` moved to `modules/scripting` package instead
-of shipping as part of Solr, due to security concerns.
-
-* SOLR-14926: `modules/clustering` back and rewritten
-
-* SOLR-14912: Cleaned up solr-extraction module to produce solr-extraction-* jar (instead of solr-cell-*). (Dawid Weiss)
-
-* SOLR-15924: Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
+=== Other
+* Contrib modules are now just "modules". You can easily enable module(s) through environment variable `SOLR_MODULES`.
+* Features lifted out as separate modules are: HDFS, Hadoop-Auth, SQL, Scripting, and JWT-Auth.
+* The "dist" folder in the release has been removed. Please update your `<lib>` entries in your `solrconfig.xml` to use the new location.
+** The `solr-core` and `solr-solrj` jars can be found under `server/solr-webapp/webapp/WEB-INF/lib/`.
+** The Solr module jars and their dependencies can be found in `modules/<module-name>/lib`, packaged individually for each module.
+** The `solrj-deps` (SolrJ Dependencies) are no longer separated out from the other Server jars.
+** Please refer to the SolrJ Maven artifact to see the exact dependencies you need to include from `server/solr-webapp/webapp/WEB-INF/lib/` and `server/lib/ext/` if you are loading in SolrJ manually.
+If you plan on using SolrJ as a JDBC driver, please refer to the xref:query-guide:sql-query.adoc#generic-clients[JDBC documentation]
+** More information can be found in the xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[Libs documentation].
+
+* SolrJ class `CloudSolrClient` now supports HTTP2. It has a new Builder. See CloudLegacySolrClient for the 8.x version of this class.
+* In Backup request responses, the `response` key now uses a map to return information instead of a list. This is only applicable for users returning information in JSON format, which is the default behavior.
+* SolrMetricProducer / SolrInfoBean APIs have changed and third-party components that implement these APIs need to be updated.
+* Use of blacklist/whitelist terminology has been completely removed. JWTAuthPlugin parameter `algWhitelist` is now `algAllowlist`. The old parameter will still work in 9.x. Environment variables `SOLR_IP_WHITELIST` and `SOLR_IP_BLACKLIST` are no longer supported, but replaced with `SOLR_IP_ALLOWLIST` and `SOLR_IP_DENYLIST`.
+* Solr Backups - Async responses for backups now correctly aggregate and return information. For collection's snapshot backup request responses additional fields `indexVersion`, `indexFileCount`, etc. were added similar to incremental backup request responses.
+* If you are using the HDFS backup repository, you need to change the repository class to `org.apache.solr.hdfs.backup.repository.HdfsBackupRepository` - see the xref:deployment-guide:backup-restore.adoc#hdfsbackuprepository[HDFS Backup Repository] section.
+* HDFS storage support has been moved to a module. Existing Solr configurations do not need any HDFS-related
+changes, however the module needs to be installed - see the section xref:deployment-guide:solr-on-hdfs.adoc[].
+* The folder `$SOLR_HOME/userfiles`, used by the "cat" streaming expression, is no longer created automatically on startup. The user must create this folder.
+* SOLR-9575: Solr no longer requires a `solr.xml` in `$SOLR_HOME`. If one is not found, Solr will instead use the default one from `$SOLR_TIP/server/solr/solr.xml`. You can revert to the pre-9.0 behaviour by setting environment variable `SOLR_SOLRXML_REQUIRED=true` or system property `-Dsolr.solrxml.required=true`. Solr also does not require a `zoo.cfg` in `$SOLR_HOME` if started with embedded zookeeper.
+* `base_url` has been removed from stored cluster state. If you're able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state in Zookeeper with future versions of Solr; as of Solr 9.x, the `base_url` will no longer be persisted in stored state.
+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then you should set `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in Zookeeper. For background, see: link:https://issues.apache.org/jira/browse/SOLR-12182[SOLR-12182] and link:https://issues.apache.org/jira/browse/SOLR-15145[SOLR-15145]. Support for the `solr.storeBaseUrl` system property will be removed in Solr 10.x and `base_url` will no longer be stored.
+* Analyzer components can now be looked up by their SPI names based on the field type configuration.
+* solr-extraction module has been cleaned up to produce solr-extraction-* jar instead of solr-cell-* jars.
+* Extra lucene libraries used in modules are no longer packaged in `lucene-libs/` under module directories in the binary release.
Instead, these libraries will be included with all other module dependencies in `lib/`.
-* SOLR-15954: The prometheus-exporter is no longer packaged as a Solr module. It can be found under `solr/prometheus-exporter/`.
-
+=== Logging and Metrics
+* Metrics handler only depends on SolrJ instead of core and has its own `log4j2.xml` and no longer shares Solr’s logging config.
+* SOLR-14401: Metrics: Only SearchHandler and subclasses have "local" metrics now. It's now tracked as if it's another handler with a "[shard]" suffix, e.g. "/select[shard]".
Review Comment:
Bummer :) I moved fast to get the bulk of this in place for the release. Will prepare another PR which puts the old text back..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org