You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2020/07/08 17:55:16 UTC
[lucene-solr] branch branch_8x updated: Ref Guide: add more context
to HDFS, CDCR, DIH deprecations; put CDCR notice on all CDCR pages
This is an automated email from the ASF dual-hosted git repository.
ctargett pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
The following commit(s) were added to refs/heads/branch_8x by this push:
new e5233cd Ref Guide: add more context to HDFS, CDCR, DIH deprecations; put CDCR notice on all CDCR pages
e5233cd is described below
commit e5233cd259d8bde8db747dd4a0ee1bb9b06bebbd
Author: Cassandra Targett <ca...@lucidworks.com>
AuthorDate: Wed Jul 8 12:53:06 2020 -0500
Ref Guide: add more context to HDFS, CDCR, DIH deprecations; put CDCR notice on all CDCR pages
---
solr/solr-ref-guide/src/cdcr-api.adoc | 8 ++++
solr/solr-ref-guide/src/cdcr-architecture.adoc | 8 ++++
solr/solr-ref-guide/src/cdcr-config.adoc | 8 ++++
solr/solr-ref-guide/src/cdcr-operations.adoc | 8 ++++
.../src/cross-data-center-replication-cdcr.adoc | 16 ++++++-
.../src/making-and-restoring-backups.adoc | 2 +-
solr/solr-ref-guide/src/running-solr-on-hdfs.adoc | 2 +-
solr/solr-ref-guide/src/solr-upgrade-notes.adoc | 52 ++++++++++++++++------
...ta-store-data-with-the-data-import-handler.adoc | 2 +-
9 files changed, 87 insertions(+), 19 deletions(-)
diff --git a/solr/solr-ref-guide/src/cdcr-api.adoc b/solr/solr-ref-guide/src/cdcr-api.adoc
index 05464cd..f5c26dc 100644
--- a/solr/solr-ref-guide/src/cdcr-api.adoc
+++ b/solr/solr-ref-guide/src/cdcr-api.adoc
@@ -18,6 +18,14 @@
The CDCR API is used to control and monitor the replication process. Control actions are performed at a collection level, i.e., by using the following base URL for API calls: `\http://localhost:8983/solr/<collection>/cdcr`.
+[WARNING]
+.CDCR is deprecated
+====
+This feature (in its current form) is deprecated and will likely be removed in 9.0.
+
+See <<cross-data-center-replication-cdcr.adoc#cross-data-center-replication-cdcr,Cross Data Center Replication>> for more details.
+====
+
Monitor actions are performed at a core level, i.e., by using the following base URL for API calls: `\http://localhost:8983/solr/<core>/cdcr`.
Currently, none of the CDCR API calls have parameters.
diff --git a/solr/solr-ref-guide/src/cdcr-architecture.adoc b/solr/solr-ref-guide/src/cdcr-architecture.adoc
index 169d385..b479f3d 100644
--- a/solr/solr-ref-guide/src/cdcr-architecture.adoc
+++ b/solr/solr-ref-guide/src/cdcr-architecture.adoc
@@ -20,6 +20,14 @@
With CDCR, Source and Target data centers can each serve search queries when CDCR is operating. The Target data center will lag somewhat behind the Source cluster due to propagation delays.
+[WARNING]
+.CDCR is deprecated
+====
+This feature (in its current form) is deprecated and will likely be removed in 9.0.
+
+See <<cross-data-center-replication-cdcr.adoc#cross-data-center-replication-cdcr,Cross Data Center Replication>> for more details.
+====
+
Data changes on the Source data center are replicated to the Target data center only after they are persisted to disk. The data changes can be replicated in near real-time (with a small delay) or could be scheduled to be sent at longer intervals to the Target data center. CDCR can "bootstrap" the collection to the Target data center. Since this is a full copy of the entire index, network bandwidth should be considered. Of course both Source and Target collections may be empty to start.
Each shard leader in the Source data center will be responsible for replicating its updates to the corresponding leader in the Target data center. When receiving updates from the Source data center, shard leaders in the Target data center will replicate the changes to their own replicas as normal SolrCloud updates.
diff --git a/solr/solr-ref-guide/src/cdcr-config.adoc b/solr/solr-ref-guide/src/cdcr-config.adoc
index cb61747..25add10 100644
--- a/solr/solr-ref-guide/src/cdcr-config.adoc
+++ b/solr/solr-ref-guide/src/cdcr-config.adoc
@@ -18,6 +18,14 @@
The Source and Target configurations differ in the case of the data centers being in separate clusters. "Cluster" here means separate ZooKeeper ensembles controlling disjoint Solr instances. Whether these data centers are physically separated or not is immaterial for this discussion.
+[WARNING]
+.CDCR is deprecated
+====
+This feature (in its current form) is deprecated and will likely be removed in 9.0.
+
+See <<cross-data-center-replication-cdcr.adoc#cross-data-center-replication-cdcr,Cross Data Center Replication>> for more details.
+====
+
As described in the section <<cdcr-architecture.adoc#cdcr-architecture,CDCR Architecture>>, two approaches are supported: uni-directional updates and bi-directional updates.
All CDCR configuration is done in the `solrconfig.xml` file. Because this is a per-collection configuration file, all CDCR configuration is done for each collection.
diff --git a/solr/solr-ref-guide/src/cdcr-operations.adoc b/solr/solr-ref-guide/src/cdcr-operations.adoc
index c50aa6a..af4d0d9 100644
--- a/solr/solr-ref-guide/src/cdcr-operations.adoc
+++ b/solr/solr-ref-guide/src/cdcr-operations.adoc
@@ -16,6 +16,14 @@
// specific language governing permissions and limitations
// under the License.
+[WARNING]
+.CDCR is deprecated
+====
+This feature (in its current form) is deprecated and will likely be removed in 9.0.
+
+See <<cross-data-center-replication-cdcr.adoc#cross-data-center-replication-cdcr,Cross Data Center Replication>> for more details.
+====
+
== Monitoring
. Network and disk space monitoring are essential. Ensure that the system has plenty of available storage to queue up changes if there is a disconnect between the Source and Target. A network outage between the two data centers can cause your disk usage to grow. Some tips:
diff --git a/solr/solr-ref-guide/src/cross-data-center-replication-cdcr.adoc b/solr/solr-ref-guide/src/cross-data-center-replication-cdcr.adoc
index 67747fd..f19b2c4 100644
--- a/solr/solr-ref-guide/src/cross-data-center-replication-cdcr.adoc
+++ b/solr/solr-ref-guide/src/cross-data-center-replication-cdcr.adoc
@@ -17,10 +17,22 @@
// specific language governing permissions and limitations
// under the License.
-WARNING: This feature (in its current form) is deprecated as of 8.6 and shall be removed in 9.0. See SOLR-14022 for details.
-
Cross Data Center Replication (CDCR) allows you to create multiple SolrCloud data centers and keep them in sync.
+[WARNING]
+.CDCR is deprecated
+====
+This feature (in its current form) is deprecated and will likely be removed in 9.0.
+
+Anyone currently using CDCR should consider migrating away from it.
+There are several open issues which make CDCR complex to maintain and generally unstable.
+
+The simplest alternative to CDCR is to manually maintain two Solr implementations in separate data centers and manage them as completely separate installations.
+While this may require more initial setup, it will be more stable in the long run.
+
+There are other alternatives, and the Solr community is working to identify the best recommended replacement in time for 9.0.
+====
+
== What is CDCR?
The <<solrcloud.adoc#solrcloud,SolrCloud>> architecture is designed to support <<near-real-time-searching.adoc#near-real-time-searching,Near Real Time (NRT)>> searches on a Solr collection that usual consists of multiple nodes in a single data center. CDCR augments this model by forwarding updates from a Solr collection in one data center to a parallel Solr collection in another data center where the network latencies are greater than the SolrCloud model was designed to accommodate.
diff --git a/solr/solr-ref-guide/src/making-and-restoring-backups.adoc b/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
index 9e4afd0..b39a6af 100644
--- a/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
+++ b/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
@@ -29,7 +29,7 @@ Likewise, committing changes using `openSearcher=false` may result in changes co
== SolrCloud Backups
-Support for backups when running SolrCloud is provided with the <<collections-api.adoc#collections-api,Collections API>>. This allows the backups to be generated across multiple shards, and restored to the same number of shards and replicas as the original collection.
+Support for backups when running SolrCloud is provided with the <<collection-management.adoc#collection-management,Collections API>>. This allows the backups to be generated across multiple shards, and restored to the same number of shards and replicas as the original collection.
NOTE: SolrCloud Backup/Restore requires a shared file system mounted at the same path on all nodes, or HDFS.
diff --git a/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc b/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
index 44797fc..e74b7a5 100644
--- a/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
+++ b/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
@@ -16,7 +16,7 @@
// specific language governing permissions and limitations
// under the License.
-WARNING: HDFS support in Solr is deprecated as of 8.6 and maybe be removed in 9.0. See SOLR-14021 for details.
+WARNING: Storing indexes in HDFS is deprecated and may be be removed in 9.0. This functionality may be moved to a 3rd-party plugin in the future.
Solr has support for writing and reading its index and transaction log files to the HDFS distributed filesystem.
diff --git a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc b/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
index 8b3bb81..320f425 100644
--- a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
+++ b/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
@@ -36,7 +36,7 @@ If you are upgrading from 7.x, see the section <<Upgrading from 7.x Releases>> b
=== Solr 8.6
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote86[8.6 Release Notes]
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote86[8.6 Release Notes^]
for an overview of the main new features of Solr 8.6.
When upgrading to 8.6.x users should be aware of the following major changes from 8.5.
@@ -45,6 +45,8 @@ When upgrading to 8.6.x users should be aware of the following major changes fro
Lucene added support for Block-Max WAND in 8.0, and 8.6 makes this available for Solr also.
+This can provide significant performance enhancements by not calculating the score for results which are not likely to appear in the top set of results.
+
It is enabled when using a new query parameter `minExactCount`.
This parameter tells Solr to accurately count the number of hits accurately until at least this value.
Once this value is reached, Solr can skip over documents that don't have a score high enough to be in the top set of documents, which has the potential for greatly speeding up searches.
@@ -87,25 +89,47 @@ For the Unified Highlighter: The setting `hl.fragsizeIsMinimum` now defaults to
This will yield longer highlights on average compared to Solr 8.5 but relatively unchanged compared to previous releases.
Furthermore, if your application highlights lots of text, you may want to experiment with lowering `hl.fragAlignRatio` to trade ideal fragment alignment for better performance.
-*Restricting file paths*
+*Restricting File Paths*
+
+Several Solr APIs allow specifying a file system location.
+This includes core creation, backup, restore and others.
+Prior to Solr 8.6 these parameters were not validated, and Solr would allow any absolute or relative path.
+From 8.6 we by default allow only paths that are relative to `SOLR_HOME`, `SOLR_DATA_HOME` and `coreRootDir`.
+If you need to create a core or store a backup outside these pre-allowed paths, you now need to tell Solr about what paths to allow.
-Several Solr APIs allow specifying a file system location. This includes core creation, backup, restore and others. Prior to Solr 8.6 these parameters were not validated, and Solr would allow any absolute or relative path. From 8.6 we by default allow only paths that are relative to `SOLR_HOME`, `SOLR_DATA_HOME` and `coreRootDir`. If you need to create a core or store a backup outside these pre-allowed paths, you now need to tell Solr about what paths to allow.
+Solr can be configured with a comma separated list of paths to allow in `solr.xml` using the new `allowPaths` element.
+When using the `solr.xml` file that ships with 8.6, you can configure the list of paths to allow through the system property `solr.allowPaths`.
+Please see `bin/solr.in.sh` or `bin\solr.in.cmd` for example usage.
+Using the value `*` will allow any path as in earlier versions.
-Solr can be configured with a comma separated list of paths to allow in `solr.xml` using the new `allowPaths` element, see <<format-of-solr-xml.adoc#the-solr-element,Solr.xml Parameters>>. When using the `solr.xml` file shipping with 8.6, you can configure the list of paths to allow through the system property `solr.allowPaths`. Please see `bin/solr.in.sh` or `bin\solr.in.cmd` for example usage. Using the value `*` will allow any path as in erlier versions.
+For more on this, see the section <<format-of-solr-xml.adoc#the-solr-element,Solr.xml Parameters>>.
-Windows SMB shares on the UNC format, such as `\\myhost\myshare\mypath` are now always disallowed. Please use drive letter mounts instead, i.e. `S:\mypath`.
+Windows SMB shares on the UNC format, such as `\\myhost\myshare\mypath` are now always disallowed.
+Please use drive letter mounts instead, i.e. `S:\mypath`.
*Deprecations*
-* Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0. Please refer to SOLR-14022.
+A primary focus of the community is improving Solr's stability and supportability.
+With the addition of the package manager system in 8.4, we now have the ability to move some features into plugins maintained by third-parties with the expertise to properly develop and support them.
+Our goal is to make running Solr easier and less prone to outages and other headaches.
+In this spirit, the following features have been deprecated and are slated to be removed in Solr 9.0.
+
+* Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0.
+This feature is unlikely to be replaced by an identical plugin.
+However, the community is working on figuring out a replacement feature for disaster recovery and failover.
+
+* The Data Import Handler (DIH) is deprecated and is scheduled to be removed in 9.0.
+Work to replace DIH with a community-supported plugin is underway and may be available soon.
-* Data Import Handler is deprecated and is scheduled to be removed in 9.0. A community supported version of this may be available as an external package to be used with the package management system. For more details, please refer to SOLR-14066.
+* Support to store indexes and backups in HDFS is deprecated and is scheduled to be removed in 9.0.
+A community-supported version of this may be available as a plugin in the future.
+For more details, please see https://issues.apache.org/jira/browse/SOLR-14021[SOLR-14021^].
-* HDFS support in Solr is deprecated and is scheduled to be removed in 9.0. A community supported version of this may be available as an external package to be used with the package management system. For more details, please refer to SOLR-14021.
+Users interested in maintaining a feature as a plugin are encouraged to join the https://lucene.apache.org/solr/community.html#mailing-lists-irc[developer mailing list^] to find out more about how to help.
=== Solr 8.5
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote85[8.5 Release Notes]
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote85[8.5 Release Notes^]
for an overview of the main new features of Solr 8.5.
When upgrading to 8.5.x users should be aware of the following major changes from 8.4.
@@ -181,7 +205,7 @@ By default, the <<other-parsers.adoc#boolean-query-parser,Boolean Query Parser>>
Solr now includes a command line tool, `bin/postlogs` which will index Solr's log files into a collection.
This provides an easy way to use Solr or visualization tools (such as Zeppelin) to troubleshoot problems with the system.
-This tool is not yet officially documented in the Reference Guide, but draft documentation is available in a branch and can be accessed https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc[via GitHub].
+This tool is not yet officially documented in the Reference Guide, but draft documentation is available in a branch and can be accessed https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc[via GitHub^].
*Highlighting*
@@ -199,7 +223,7 @@ This property can now be defined in `solr.in.sh` or `solr.in.cmd` as a system pr
=== Solr 8.4
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote84[8.4 Release Notes]
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote84[8.4 Release Notes^]
for an overview of the main new features of Solr 8.4.
When upgrading to 8.4.x users should be aware of the following major changes from 8.3.
@@ -328,7 +352,7 @@ Users are encouraged to transition their cache configurations to use
=== Solr 8.3
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote83[8.3 Release Notes] for an overview of the main new features of Solr 8.3.
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote83[8.3 Release Notes^] for an overview of the main new features of Solr 8.3.
When upgrading to 8.3.x users should be aware of the following major changes from 8.2.
@@ -355,7 +379,7 @@ See the section <<query-settings-in-solrconfig.adoc#query-settings-in-solrconfig
=== Solr 8.2
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote82[8.2 Release Notes] for an overview of the main new features of Solr 8.2.
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote82[8.2 Release Notes^] for an overview of the main new features of Solr 8.2.
When upgrading to 8.2.x, users should be aware of the following major changes from v8.1.
@@ -393,7 +417,7 @@ This release adds support for tracing requests in Solr. Please review the sectio
=== Solr 8.1
-See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote810[8.1 Release Notes] for an overview of the main new features of Solr 8.1.
+See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote810[8.1 Release Notes^] for an overview of the main new features of Solr 8.1.
When upgrading to 8.1.x, users should be aware of the following major changes from v8.0.
diff --git a/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc b/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
index 5d925ff..98c315e 100644
--- a/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
+++ b/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
@@ -17,7 +17,7 @@
// specific language governing permissions and limitations
// under the License.
-WARNING: The Data Import Handler is deprecated as of v8.6 and is scheduled to be removed in 9.0.
+WARNING: The Data Import Handler is deprecated is scheduled to be removed in 9.0. This functionality will likely migrate to a 3rd-party plugin in the near future.
Many search applications store the content to be indexed in a structured data store, such as a relational database. The Data Import Handler (DIH) provides a mechanism for importing content from a data store and indexing it.