You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by rz...@apache.org on 2026/09/23 08:09:10 UTC
(stormcrawler) branch fix-2098-urlfrontier-tls created (now 2133f70f)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a change to branch fix-2098-urlfrontier-tls
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
at 2133f70f Document the TLS proxy needed until the frontier supports TLS and the behaviour on a failed handshake
This branch includes the following new commits:
new 2133f70f Document the TLS proxy needed until the frontier supports TLS and the behaviour on a failed handshake
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(stormcrawler) 01/01: Document the TLS proxy needed until the frontier supports TLS and the behaviour on a failed handshake
Posted by rz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch fix-2098-urlfrontier-tls
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 2133f70fa0a21048223f8fdf42f7a19215019a69
Author: Richard Zowalla <rz...@apache.org>
AuthorDate: Wed Sep 23 10:09:03 2026 +0200
Document the TLS proxy needed until the frontier supports TLS and the behaviour on a failed handshake
---
external/urlfrontier/README.md | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/external/urlfrontier/README.md b/external/urlfrontier/README.md
index 7c159d1b..bcce4319 100644
--- a/external/urlfrontier/README.md
+++ b/external/urlfrontier/README.md
@@ -28,7 +28,13 @@ urlfrontier.max.urls.per.bucket:10
The gRPC channels to the frontier are plaintext unless TLS is enabled. Plaintext is kept as the
default so that existing deployments keep working. The channel carries the URLs and their metadata, so enable TLS whenever the
-frontier runs on another host:
+frontier runs on another host.
+
+URLFrontier 2.6 only listens in plaintext, so until
+[crawler-commons/url-frontier#219](https://github.com/crawler-commons/url-frontier/pull/219) is
+released, the frontier needs a TLS terminating proxy in front of it, and `urlfrontier.address` or
+`urlfrontier.host` and `urlfrontier.port` must point at that proxy. Once the frontier supports TLS
+itself, the same settings connect to it directly:
```yaml
urlfrontier.tls.enabled: true
@@ -45,8 +51,10 @@ urlfrontier.tls.client.private.key.password: changeit
The server certificate must be valid for the host name in `urlfrontier.address` or
`urlfrontier.host`. Setting only one of `urlfrontier.tls.client.cert.chain` and
`urlfrontier.tls.client.private.key`, or pointing a key at a file which cannot be read, fails
-the component at startup. The settings apply to `Spout`, `StatusUpdaterBolt` and
-`QueueRegulatorBolt`.
+the component at startup. A failed TLS handshake, on the other hand, does not: the channel keeps
+reconnecting, `Spout` and `StatusUpdaterBolt` wait for it without a deadline, and the topology
+runs without fetching or updating anything. The settings apply to `Spout`, `StatusUpdaterBolt`
+and `QueueRegulatorBolt`.
## Sending discovered URLs in batches