You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by Richard Zowalla <rz...@apache.org> on 2026/05/22 13:03:43 UTC

[VOTE] Apache StormCrawler 3.6.0 Release Candidate

Hi folks,

I have posted a first release candidate for the Apache StormCrawler 3.6.0 release and it is ready for testing.

This release introduces native support for Apache Storm v2 metrics with a backward-compatible bridge (#585), a new stormcrawler-opensearch-java module built on the OpenSearch Java client (#1868), and several Playwright enhancements: a new PageAction extension point (#1897), a JsRenderingDetector parse filter and bolt (#1898), and capture of the rendered DOM on non-2xx responses (#1891).

It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration (#1833), makes Metadata keys consistently case-insensitive (#1827), and resolves a number of resource leaks and thread-safety issues across the OkHttp and Apache HttpClient protocols and the OpenSearch modules. Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation of the java.net.URL constructors (#1834), and upgrades multiple dependencies.

Thank you to everyone who contributed to this release, including all of our users and the people who submitted bug reports, contributed code or documentation enhancements.

The release was made using the Apache StormCrawler release process, documented here:
https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Source:

https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/

Tag:

https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0

Commit Hash:

32cf6db6bc6da300da703eaf4d907a8f6091f0d5

Maven Repo:

https://repository.apache.org/content/repositories/orgapachestormcrawler-1037

<repositories>
<repository>
<id>stormcrawler-3.6.0-rc1</id>
<name>Testing StormCrawler 3.6.0 release candidate</name>
<url>
https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
</url>
</repository>
</repositories>

Release notes:

https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0

Reminder: The up-2-date KEYS file for signature verification can be
found here: https://downloads.apache.org/stormcrawler/KEYS

Please vote on releasing these packages as Apache StormCrawler 3.6.0
The vote is open for at least the next 72 hours.

Only votes from the StormCrawler PMC are binding, but everyone is welcome to check the release candidate and vote.
The vote passes if at least three binding +1 votes are cast.

Please VOTE

[+1] go ship it
[+0] meh, don't care
[-1] stop, there is a ${showstopper}

Thanks!

Richard

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Markos Volikas <mv...@apache.org>.
Hi everyone!

+1 from me (binding)

     - Checked hashes
     - Built from source
     - Ran a crawl with Solr 10.0.0 in cluster mode and Storm 2.8.8 
(everything on the host / no containers)

Thanks Richard for preparing the release,

Markos

On 5/22/26 16:03, Richard Zowalla wrote:
> Hi folks,
>
> I have posted a first release candidate for the Apache StormCrawler 3.6.0 release and it is ready for testing.
>
> This release introduces native support for Apache Storm v2 metrics with a backward-compatible bridge (#585), a new stormcrawler-opensearch-java module built on the OpenSearch Java client (#1868), and several Playwright enhancements: a new PageAction extension point (#1897), a JsRenderingDetector parse filter and bolt (#1898), and capture of the rendered DOM on non-2xx responses (#1891).
>
> It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration (#1833), makes Metadata keys consistently case-insensitive (#1827), and resolves a number of resource leaks and thread-safety issues across the OkHttp and Apache HttpClient protocols and the OpenSearch modules. Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation of the java.net.URL constructors (#1834), and upgrades multiple dependencies.
>
> Thank you to everyone who contributed to this release, including all of our users and the people who submitted bug reports, contributed code or documentation enhancements.
>
> The release was made using the Apache StormCrawler release process, documented here:
> https://github.com/apache/stormcrawler/blob/main/RELEASING.md
>
> Source:
>
> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
>
> Tag:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Commit Hash:
>
> 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
>
> Maven Repo:
>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
>
> <repositories>
> <repository>
> <id>stormcrawler-3.6.0-rc1</id>
> <name>Testing StormCrawler 3.6.0 release candidate</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> </url>
> </repository>
> </repositories>
>
> Release notes:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Reminder: The up-2-date KEYS file for signature verification can be
> found here: https://downloads.apache.org/stormcrawler/KEYS
>
> Please vote on releasing these packages as Apache StormCrawler 3.6.0
> The vote is open for at least the next 72 hours.
>
> Only votes from the StormCrawler PMC are binding, but everyone is welcome to check the release candidate and vote.
> The vote passes if at least three binding +1 votes are cast.
>
> Please VOTE
>
> [+1] go ship it
> [+0] meh, don't care
> [-1] stop, there is a ${showstopper}
>
> Thanks!
>
> Richard

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Dávid Szigecsán <si...@gmail.com>.
+1 (binding)

- LICENSE, NOTICE and README.md present
- pom.xml files are good
- `mvn clean install` works fine on my side
Environment:
Apache Maven 3.9.11 (3e54c93a704957b63ee3494413a2b544fd3d825b)
Maven home: C:\Program Files\JetBrains\IntelliJ IDEA
2025.2.1\plugins\maven\lib\maven3
Java version: 17.0.16, vendor: Eclipse Adoptium, runtime:
C:\Users\sigee\.jdks\temurin-17.0.16
Default locale: hu_HU, platform encoding: IBM852
OS name: "windows 11", version: "10.0", arch: "amd64", family: "windows"

I also have a tiny note:
In the README.md:L19, opensearch/README.md:L19, solr/README.md:L9,
warc/README.md:L33,L83 and RELEASING.md:L112-L14 there are version `3.4.0`
mentioned.

Regards,
Sigee

Davide Polato <dp...@apache.org> ezt írta (időpont: 2026. máj. 22., P,
16:00):

> +1 (non-binding)
>
> What I checked on Fedora with JDK 17:
> - PGP signature against KEYS: good
> - SHA-512 matches the published value
> - LICENSE, NOTICE and README.md present
> - No -SNAPSHOT in any pom.xml, no pre-built jars or classes in the source
> - `mvn clean install -DskipTests` builds cleanly
>
> One nit worth flagging for the future, not blocker:
> - NOTICE says "Copyright 2025" but we're in 2026.
>
> Thanks for cutting the RC, Richard.
>
> Davide
>
> Il giorno ven 22 mag 2026 alle ore 15:04 Richard Zowalla <rz...@apache.org>
> ha scritto:
>
> > Hi folks,
> >
> > I have posted a first release candidate for the Apache StormCrawler 3.6.0
> > release and it is ready for testing.
> >
> > This release introduces native support for Apache Storm v2 metrics with a
> > backward-compatible bridge (#585), a new stormcrawler-opensearch-java
> > module built on the OpenSearch Java client (#1868), and several
> Playwright
> > enhancements: a new PageAction extension point (#1897), a
> > JsRenderingDetector parse filter and bolt (#1898), and capture of the
> > rendered DOM on non-2xx responses (#1891).
> >
> > It also replaces XSoup with JSoup's built-in XPath support (#1856), adds
> a
> > bolt-level timeout for fetcher threads (#1861), fixes the Solr 10
> migration
> > (#1833), makes Metadata keys consistently case-insensitive (#1827), and
> > resolves a number of resource leaks and thread-safety issues across the
> > OkHttp and Apache HttpClient protocols and the OpenSearch modules.
> > Additionally, it upgrades to Apache Storm 2.8.8, migrates from
> Storm-shaded
> > Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+
> deprecation
> > of the java.net.URL constructors (#1834), and upgrades multiple
> > dependencies.
> >
> > Thank you to everyone who contributed to this release, including all of
> > our users and the people who submitted bug reports, contributed code or
> > documentation enhancements.
> >
> > The release was made using the Apache StormCrawler release process,
> > documented here:
> > https://github.com/apache/stormcrawler/blob/main/RELEASING.md
> >
> > Source:
> >
> >
> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
> >
> > Tag:
> >
> > https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
> >
> > Commit Hash:
> >
> > 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
> >
> > Maven Repo:
> >
> >
> >
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> >
> > <repositories>
> > <repository>
> > <id>stormcrawler-3.6.0-rc1</id>
> > <name>Testing StormCrawler 3.6.0 release candidate</name>
> > <url>
> >
> >
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> > </url>
> > </repository>
> > </repositories>
> >
> > Release notes:
> >
> > https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
> >
> > Reminder: The up-2-date KEYS file for signature verification can be
> > found here: https://downloads.apache.org/stormcrawler/KEYS
> >
> > Please vote on releasing these packages as Apache StormCrawler 3.6.0
> > The vote is open for at least the next 72 hours.
> >
> > Only votes from the StormCrawler PMC are binding, but everyone is welcome
> > to check the release candidate and vote.
> > The vote passes if at least three binding +1 votes are cast.
> >
> > Please VOTE
> >
> > [+1] go ship it
> > [+0] meh, don't care
> > [-1] stop, there is a ${showstopper}
> >
> > Thanks!
> >
> > Richard
> >
>

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Davide Polato <dp...@apache.org>.
+1 (non-binding)

What I checked on Fedora with JDK 17:
- PGP signature against KEYS: good
- SHA-512 matches the published value
- LICENSE, NOTICE and README.md present
- No -SNAPSHOT in any pom.xml, no pre-built jars or classes in the source
- `mvn clean install -DskipTests` builds cleanly

One nit worth flagging for the future, not blocker:
- NOTICE says "Copyright 2025" but we're in 2026.

Thanks for cutting the RC, Richard.

Davide

Il giorno ven 22 mag 2026 alle ore 15:04 Richard Zowalla <rz...@apache.org>
ha scritto:

> Hi folks,
>
> I have posted a first release candidate for the Apache StormCrawler 3.6.0
> release and it is ready for testing.
>
> This release introduces native support for Apache Storm v2 metrics with a
> backward-compatible bridge (#585), a new stormcrawler-opensearch-java
> module built on the OpenSearch Java client (#1868), and several Playwright
> enhancements: a new PageAction extension point (#1897), a
> JsRenderingDetector parse filter and bolt (#1898), and capture of the
> rendered DOM on non-2xx responses (#1891).
>
> It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a
> bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration
> (#1833), makes Metadata keys consistently case-insensitive (#1827), and
> resolves a number of resource leaks and thread-safety issues across the
> OkHttp and Apache HttpClient protocols and the OpenSearch modules.
> Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded
> Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation
> of the java.net.URL constructors (#1834), and upgrades multiple
> dependencies.
>
> Thank you to everyone who contributed to this release, including all of
> our users and the people who submitted bug reports, contributed code or
> documentation enhancements.
>
> The release was made using the Apache StormCrawler release process,
> documented here:
> https://github.com/apache/stormcrawler/blob/main/RELEASING.md
>
> Source:
>
> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
>
> Tag:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Commit Hash:
>
> 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
>
> Maven Repo:
>
>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
>
> <repositories>
> <repository>
> <id>stormcrawler-3.6.0-rc1</id>
> <name>Testing StormCrawler 3.6.0 release candidate</name>
> <url>
>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> </url>
> </repository>
> </repositories>
>
> Release notes:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Reminder: The up-2-date KEYS file for signature verification can be
> found here: https://downloads.apache.org/stormcrawler/KEYS
>
> Please vote on releasing these packages as Apache StormCrawler 3.6.0
> The vote is open for at least the next 72 hours.
>
> Only votes from the StormCrawler PMC are binding, but everyone is welcome
> to check the release candidate and vote.
> The vote passes if at least three binding +1 votes are cast.
>
> Please VOTE
>
> [+1] go ship it
> [+0] meh, don't care
> [-1] stop, there is a ${showstopper}
>
> Thanks!
>
> Richard
>

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Richard Zowalla <rz...@apache.org>.
Here is my own +1 (binding)

Did some tests with my usual topologies - all fine.

> Am 25.05.2026 um 23:18 schrieb Sebastian Nagel <wa...@googlemail.com>:
> 
> Hi everybody,
> 
> 
> +1 to release (binding)
> 
> 
> Tested running WARC-reading and WARC-writing topologies,
> in the following setups:
> - Storm 2.8.8 (local mode)
> - Storm 2.8.7 Docker (local mode) and Docker-compose (distributed mode)
> 
> I'm happy to repeat the Docker tests when the Storm 2.8.8 images are available
> on Docker Hub.
> 
> 
> Thanks, Richard, for spinning the release candidate!
> 
> 
> ~Sebastian
> 
> 
> On 5/22/26 15:03, Richard Zowalla wrote:
>> Hi folks,
>> I have posted a first release candidate for the Apache StormCrawler 3.6.0 release and it is ready for testing.
>> This release introduces native support for Apache Storm v2 metrics with a backward-compatible bridge (#585), a new stormcrawler-opensearch-java module built on the OpenSearch Java client (#1868), and several Playwright enhancements: a new PageAction extension point (#1897), a JsRenderingDetector parse filter and bolt (#1898), and capture of the rendered DOM on non-2xx responses (#1891).
>> It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration (#1833), makes Metadata keys consistently case-insensitive (#1827), and resolves a number of resource leaks and thread-safety issues across the OkHttp and Apache HttpClient protocols and the OpenSearch modules. Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation of the java.net.URL constructors (#1834), and upgrades multiple dependencies.
>> Thank you to everyone who contributed to this release, including all of our users and the people who submitted bug reports, contributed code or documentation enhancements.
>> The release was made using the Apache StormCrawler release process, documented here:
>> https://github.com/apache/stormcrawler/blob/main/RELEASING.md
>> Source:
>> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
>> Tag:
>> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>> Commit Hash:
>> 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
>> Maven Repo:
>> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
>> <repositories>
>> <repository>
>> <id>stormcrawler-3.6.0-rc1</id>
>> <name>Testing StormCrawler 3.6.0 release candidate</name>
>> <url>
>> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
>> </url>
>> </repository>
>> </repositories>
>> Release notes:
>> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>> Reminder: The up-2-date KEYS file for signature verification can be
>> found here: https://downloads.apache.org/stormcrawler/KEYS
>> Please vote on releasing these packages as Apache StormCrawler 3.6.0
>> The vote is open for at least the next 72 hours.
>> Only votes from the StormCrawler PMC are binding, but everyone is welcome to check the release candidate and vote.
>> The vote passes if at least three binding +1 votes are cast.
>> Please VOTE
>> [+1] go ship it
>> [+0] meh, don't care
>> [-1] stop, there is a ${showstopper}
>> Thanks!
>> Richard
>

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi everybody,


+1 to release (binding)


Tested running WARC-reading and WARC-writing topologies,
in the following setups:
- Storm 2.8.8 (local mode)
- Storm 2.8.7 Docker (local mode) and Docker-compose (distributed mode)

I'm happy to repeat the Docker tests when the Storm 2.8.8 images are available
on Docker Hub.


Thanks, Richard, for spinning the release candidate!


~Sebastian


On 5/22/26 15:03, Richard Zowalla wrote:
> Hi folks,
> 
> I have posted a first release candidate for the Apache StormCrawler 3.6.0 release and it is ready for testing.
> 
> This release introduces native support for Apache Storm v2 metrics with a backward-compatible bridge (#585), a new stormcrawler-opensearch-java module built on the OpenSearch Java client (#1868), and several Playwright enhancements: a new PageAction extension point (#1897), a JsRenderingDetector parse filter and bolt (#1898), and capture of the rendered DOM on non-2xx responses (#1891).
> 
> It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration (#1833), makes Metadata keys consistently case-insensitive (#1827), and resolves a number of resource leaks and thread-safety issues across the OkHttp and Apache HttpClient protocols and the OpenSearch modules. Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation of the java.net.URL constructors (#1834), and upgrades multiple dependencies.
> 
> Thank you to everyone who contributed to this release, including all of our users and the people who submitted bug reports, contributed code or documentation enhancements.
> 
> The release was made using the Apache StormCrawler release process, documented here:
> https://github.com/apache/stormcrawler/blob/main/RELEASING.md
> 
> Source:
> 
> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
> 
> Tag:
> 
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
> 
> Commit Hash:
> 
> 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
> 
> Maven Repo:
> 
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> 
> <repositories>
> <repository>
> <id>stormcrawler-3.6.0-rc1</id>
> <name>Testing StormCrawler 3.6.0 release candidate</name>
> <url>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> </url>
> </repository>
> </repositories>
> 
> Release notes:
> 
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
> 
> Reminder: The up-2-date KEYS file for signature verification can be
> found here: https://downloads.apache.org/stormcrawler/KEYS
> 
> Please vote on releasing these packages as Apache StormCrawler 3.6.0
> The vote is open for at least the next 72 hours.
> 
> Only votes from the StormCrawler PMC are binding, but everyone is welcome to check the release candidate and vote.
> The vote passes if at least three binding +1 votes are cast.
> 
> Please VOTE
> 
> [+1] go ship it
> [+0] meh, don't care
> [-1] stop, there is a ${showstopper}
> 
> Thanks!
> 
> Richard

Re: [VOTE] Apache StormCrawler 3.6.0 Release Candidate

Posted by Julien Nioche <ju...@digitalpebble.com>.
Hi,

Compiled from source
Tested successfully with 2 crawls: local and distributed mode

+1 to release (binding)

Thanks

Julien


On Fri, 22 May 2026 at 14:03, Richard Zowalla <rz...@apache.org> wrote:

> Hi folks,
>
> I have posted a first release candidate for the Apache StormCrawler 3.6.0
> release and it is ready for testing.
>
> This release introduces native support for Apache Storm v2 metrics with a
> backward-compatible bridge (#585), a new stormcrawler-opensearch-java
> module built on the OpenSearch Java client (#1868), and several Playwright
> enhancements: a new PageAction extension point (#1897), a
> JsRenderingDetector parse filter and bolt (#1898), and capture of the
> rendered DOM on non-2xx responses (#1891).
>
> It also replaces XSoup with JSoup's built-in XPath support (#1856), adds a
> bolt-level timeout for fetcher threads (#1861), fixes the Solr 10 migration
> (#1833), makes Metadata keys consistently case-insensitive (#1827), and
> resolves a number of resource leaks and thread-safety issues across the
> OkHttp and Apache HttpClient protocols and the OpenSearch modules.
> Additionally, it upgrades to Apache Storm 2.8.8, migrates from Storm-shaded
> Commons Lang to Commons Lang 3 (#1835), addresses the Java 20+ deprecation
> of the java.net.URL constructors (#1834), and upgrades multiple
> dependencies.
>
> Thank you to everyone who contributed to this release, including all of
> our users and the people who submitted bug reports, contributed code or
> documentation enhancements.
>
> The release was made using the Apache StormCrawler release process,
> documented here:
> https://github.com/apache/stormcrawler/blob/main/RELEASING.md
>
> Source:
>
> https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.6.0-rc1/
>
> Tag:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Commit Hash:
>
> 32cf6db6bc6da300da703eaf4d907a8f6091f0d5
>
> Maven Repo:
>
>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
>
> <repositories>
> <repository>
> <id>stormcrawler-3.6.0-rc1</id>
> <name>Testing StormCrawler 3.6.0 release candidate</name>
> <url>
>
> https://repository.apache.org/content/repositories/orgapachestormcrawler-1037
> </url>
> </repository>
> </repositories>
>
> Release notes:
>
> https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.6.0
>
> Reminder: The up-2-date KEYS file for signature verification can be
> found here: https://downloads.apache.org/stormcrawler/KEYS
>
> Please vote on releasing these packages as Apache StormCrawler 3.6.0
> The vote is open for at least the next 72 hours.
>
> Only votes from the StormCrawler PMC are binding, but everyone is welcome
> to check the release candidate and vote.
> The vote passes if at least three binding +1 votes are cast.
>
> Please VOTE
>
> [+1] go ship it
> [+0] meh, don't care
> [-1] stop, there is a ${showstopper}
>
> Thanks!
>
> Richard
>