You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by dp...@apache.org on 2026/10/16 15:37:10 UTC
(stormcrawler) 04/07: drop the sniff and responseBufferSize settings
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch opensearch-3.x
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit dade9a7f6bea9c6b82974f40a37a2b558e525785
Author: Davide Polato <dp...@apache.org>
AuthorDate: Fri Oct 16 15:58:25 2026 +0200
drop the sniff and responseBufferSize settings
The opensearch-java client has no sniffer and no per-request response
buffer setting. The keys are removed from the module's
opensearch-conf.yaml, from the one of the archetype and from the
configuration reference. When they are set, OpenSearchConnection
still logs a WARN, which now points to external/opensearch/README.md.
The configuration reference and the Javadoc of DelegateRefresher no
longer mention a second OpenSearch module.
Signed-off-by: Davide Polato <dp...@apache.org>
---
docs/src/main/asciidoc/configuration.adoc | 7 +++----
.../resources/archetype-resources/opensearch-conf.yaml | 4 ----
external/opensearch/opensearch-conf.yaml | 13 -------------
.../apache/stormcrawler/opensearch/DelegateRefresher.java | 4 ----
.../stormcrawler/opensearch/OpenSearchConnection.java | 14 +++++++-------
5 files changed, 10 insertions(+), 32 deletions(-)
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index 7ec62c89..f628335b 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -355,8 +355,8 @@ Note: `indexer.md.docid` is never used raw: the matching value is passed through
(the same digest used for the default URL-based ID), except for CloudSearch, whose
`CloudSearchIndexerBolt` re-digests it with SHA-512 via `CloudSearchUtils.getID` to satisfy
CloudSearch's ID constraints. It is only honoured by backends whose bolts call
-`AbstractIndexerBolt#getDocumentID` under the hood: the OpenSearch and OpenSearch-Java
-`IndexerBolt`/`DeletionBolt` pairs, and the CloudSearch `CloudSearchIndexerBolt`. The Solr and
+`AbstractIndexerBolt#getDocumentID` under the hood: the OpenSearch
+`IndexerBolt`/`DeletionBolt` pair, and the CloudSearch `CloudSearchIndexerBolt`. The Solr and
SQL indexers build their document/row key directly from the URL and never call
`getDocumentID()`, so this setting is silently ignored for those two backends.
@@ -443,9 +443,8 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/opensearc
| key | default value | description
| opensearch.addresses | - | OpenSearch server address(es). An address without a scheme is contacted over plain http, on port 9200 if none is given; use `https://` when credentials are configured.
-| opensearch.user | - | Username for Basic authentication (optional). The credentials are only sent to the addresses listed, matched on scheme, host and port, not to nodes discovered by sniffing under another address; list those nodes in the addresses or disable `opensearch.<type>.sniff`, which is on by default. A warning is logged when sniffing is enabled together with credentials, and if an address other than a loopback one uses plain http.
+| opensearch.user | - | Username for Basic authentication (optional). The credentials are only sent to the addresses listed, matched on scheme, host and port. A warning is logged if an address other than a loopback one uses plain http.
| opensearch.password | - | Password for authentication (optional).
-| opensearch.<type>.sniff | true | Discover the other nodes of the cluster and send requests to them as well. The nodes found are contacted over https if one of the addresses uses https, over plain http otherwise.
| opensearch.disable.tls.validation | false | Do not check the certificates and host names of the OpenSearch nodes. A warning is logged when this is enabled.
| opensearch.concurrentRequests | 2 | Number of concurrent bulk requests.
| opensearch.indexer.index.name | content | Index name for crawled documents.
diff --git a/external/opensearch/archetype/src/main/resources/archetype-resources/opensearch-conf.yaml b/external/opensearch/archetype/src/main/resources/archetype-resources/opensearch-conf.yaml
index d1b8d450..6152b943 100644
--- a/external/opensearch/archetype/src/main/resources/archetype-resources/opensearch-conf.yaml
+++ b/external/opensearch/archetype/src/main/resources/archetype-resources/opensearch-conf.yaml
@@ -42,17 +42,14 @@ config:
opensearch.indexer.bulkActions: 100
opensearch.indexer.flushInterval: "2s"
opensearch.indexer.concurrentRequests: 1
- opensearch.indexer.sniff: true
# MetricsConsumer
# opensearch.metrics.addresses: "http://localhost:9200"
opensearch.metrics.index.name: "metrics"
- opensearch.metrics.sniff: true
# Queues index
opensearch.queues.addresses: "http://localhost:9200"
opensearch.queues.index.name: "queues"
- opensearch.queues.sniff: true
# cache of known queues, see https://github.com/ben-manes/caffeine/wiki/Specification
# opensearch.queues.cache.spec: "maximumSize=10000"
@@ -70,7 +67,6 @@ config:
opensearch.status.bulkActions: 500
opensearch.status.flushInterval: "5s"
opensearch.status.concurrentRequests: 1
- opensearch.status.sniff: true
# spout config #
diff --git a/external/opensearch/opensearch-conf.yaml b/external/opensearch/opensearch-conf.yaml
index 595e170c..4a31a14c 100644
--- a/external/opensearch/opensearch-conf.yaml
+++ b/external/opensearch/opensearch-conf.yaml
@@ -28,9 +28,6 @@ config:
#opensearch.password: "PASSWORD"
opensearch.concurrentRequests: 2
- # Sets the response buffer to the specified value in MB.
- # opensearch.responseBufferSize: 100
-
# Disable TLS validation for connection to OpenSearch, a warning is logged
# when this is enabled
# opensearch.disable.tls.validation: false
@@ -45,22 +42,15 @@ config:
opensearch.indexer.bulkActions: 100
opensearch.indexer.flushInterval: "2s"
opensearch.indexer.concurrentRequests: 1
- opensearch.indexer.sniff: true
- # Sets the response buffer to the specified value in MB.
- # opensearch.indexer.responseBufferSize: 100
# MetricsConsumer
# opensearch.metrics.addresses: "http://localhost:9200"
opensearch.metrics.index.name: "metrics"
- opensearch.metrics.sniff: true
- # Sets the response buffer to the specified value in MB.
- # opensearch.metrics.responseBufferSize: 100
# Queues index
opensearch.queues.addresses: "http://localhost:9200"
opensearch.queues.index.name: "queues"
- opensearch.queues.sniff: true
# cache of known queues, see https://github.com/ben-manes/caffeine/wiki/Specification
# opensearch.queues.cache.spec: "maximumSize=10000"
@@ -78,9 +68,6 @@ config:
opensearch.status.bulkActions: 500
opensearch.status.flushInterval: "5s"
opensearch.status.concurrentRequests: 1
- opensearch.status.sniff: true
- # Sets the response buffer to the specified value in MB.
- # opensearch.status.responseBufferSize: 100
# spout config #
diff --git a/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/DelegateRefresher.java b/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/DelegateRefresher.java
index fb03a9f0..7c49745b 100644
--- a/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/DelegateRefresher.java
+++ b/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/DelegateRefresher.java
@@ -38,10 +38,6 @@ import org.slf4j.LoggerFactory;
* org.apache.stormcrawler.opensearch.parse.filter.JSONResourceWrapper} to eliminate duplicated
* setup/refresh/cleanup logic.
*
- * <p>This is the opensearch-java (OpenSearch Java Client 3.x / HC5) counterpart of the class with
- * the same name in the {@code external/opensearch} module. It uses the typed {@link
- * OpenSearchClient} instead of the deprecated {@code RestHighLevelClient}.
- *
* @param <T> the base type that the delegate must extend (e.g. URLFilter or ParseFilter)
*/
public class DelegateRefresher<T> {
diff --git a/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/OpenSearchConnection.java b/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/OpenSearchConnection.java
index eeb47362..f822c6d7 100644
--- a/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/OpenSearchConnection.java
+++ b/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/OpenSearchConnection.java
@@ -276,29 +276,29 @@ public final class OpenSearchConnection {
private record ClientResources(OpenSearchClient client, OpenSearchTransport transport) {}
/**
- * Logs a WARN for legacy configuration keys that are no longer honoured by this module, so that
- * users migrating from {@code external/opensearch} notice silently-dropped tuning. See the
- * module README for the full list of differences.
+ * Logs a WARN for configuration keys that earlier releases of this module read and this one
+ * ignores, so that users upgrading notice silently-dropped tuning. See the module README for
+ * the full list of differences.
*/
private static void warnOnRemovedKeys(Map<String, Object> stormConf, String dottedType) {
final String responseBufferKey = Constants.PARAMPREFIX + dottedType + "responseBufferSize";
if (stormConf.containsKey(responseBufferKey)) {
LOG.warn(
- "Configuration key '{}' is set but no longer supported by the opensearch-java module. "
+ "Configuration key '{}' is set but no longer supported. "
+ "The HC5-based async transport does not expose an equivalent per-request "
+ "heap-buffer override. The setting is ignored — remove it from your "
- + "configuration. See external/opensearch-java/README.md for details.",
+ + "configuration. See external/opensearch/README.md for details.",
responseBufferKey);
}
final String sniffKey = Constants.PARAMPREFIX + dottedType + "sniff";
if (stormConf.containsKey(sniffKey)) {
LOG.warn(
- "Configuration key '{}' is set but no longer supported by the opensearch-java module. "
+ "Configuration key '{}' is set but no longer supported. "
+ "The OpenSearch Java Client 3.x does not ship a Sniffer equivalent, so "
+ "automatic node discovery is not available. Keep the 'addresses' list up to "
+ "date manually or put a load balancer in front of the cluster. "
- + "See external/opensearch-java/README.md for details.",
+ + "See external/opensearch/README.md for details.",
sniffKey);
}
}