You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by jn...@apache.org on 2025/09/02 13:07:22 UTC
(stormcrawler) branch main updated: #1604 - Externalise Selenium (#1646)
This is an automated email from the ASF dual-hosted git repository.
jnioche pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
The following commit(s) were added to refs/heads/main by this push:
new fee1d0d3 #1604 - Externalise Selenium (#1646)
fee1d0d3 is described below
commit fee1d0d3541083ea34112d0b76537eb0f54c4a19
Author: Julien Nioche <ju...@digitalpebble.com>
AuthorDate: Tue Sep 2 14:07:18 2025 +0100
#1604 - Externalise Selenium (#1646)
* Externalise Selenium, fixes #1604
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
* drop jacoco class ratio - removing classes from core impacts it
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
* fix JaCoCo thresholds + remove iterator in loop
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
* Update external/selenium/README.md
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
* Update external/selenium/pom.xml
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
* Update external/selenium/README.md
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
* Update external/selenium/README.md
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
---------
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
---
THIRD-PARTY.txt | 5 +-
core/pom.xml | 31 ++----
core/src/main/resources/crawler-default.yaml | 24 -----
external/selenium/README.md | 43 ++++++++
external/selenium/pom.xml | 108 +++++++++++++++++++++
external/selenium/selenium-conf.yaml | 41 ++++++++
.../protocol/selenium/NavigationFilter.java | 0
.../protocol/selenium/NavigationFilters.java | 0
.../protocol/selenium/RemoteDriverProtocol.java | 0
.../protocol/selenium/SeleniumProtocol.java | 0
.../protocol/selenium/ProtocolTest.java | 0
pom.xml | 1 +
12 files changed, 201 insertions(+), 52 deletions(-)
diff --git a/THIRD-PARTY.txt b/THIRD-PARTY.txt
index 52727895..77132a62 100644
--- a/THIRD-PARTY.txt
+++ b/THIRD-PARTY.txt
@@ -140,9 +140,7 @@ List of third-party dependencies grouped by their license type.
* Gson (com.google.code.gson:gson:2.11.0 - https://github.com/google/gson)
* Gson (com.google.code.gson:gson:2.12.1 - https://github.com/google/gson)
* Guava: Google Core Libraries for Java (com.google.guava:guava:18.0 - http://code.google.com/p/guava-libraries/guava)
- * Guava: Google Core Libraries for Java (com.google.guava:guava:33.2.1-android - https://github.com/google/guava)
* Guava: Google Core Libraries for Java (com.google.guava:guava:33.4.8-jre - https://github.com/google/guava)
- * Guava InternalFutureFailureAccess and InternalFutures (com.google.guava:failureaccess:1.0.2 - https://github.com/google/guava/failureaccess)
* Guava InternalFutureFailureAccess and InternalFutures (com.google.guava:failureaccess:1.0.3 - https://github.com/google/guava/failureaccess)
* Guava ListenableFuture only (com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava - https://github.com/google/guava/listenablefuture)
* IntelliJ IDEA Annotations (com.intellij:annotations:12.0 - http://www.jetbrains.org)
@@ -213,8 +211,10 @@ List of third-party dependencies grouped by their license type.
* opensearch-task-commons (org.opensearch:opensearch-task-commons:2.19.3 - https://github.com/opensearch-project/OpenSearch.git)
* opensearch-telemetry (org.opensearch:opensearch-telemetry:2.19.3 - https://github.com/opensearch-project/OpenSearch.git)
* opensearch-x-content (org.opensearch:opensearch-x-content:2.19.3 - https://github.com/opensearch-project/OpenSearch.git)
+ * OpenTelemetry Java (io.opentelemetry:opentelemetry-api:1.49.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-api:1.53.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-common:1.53.0 - https://github.com/open-telemetry/opentelemetry-java)
+ * OpenTelemetry Java (io.opentelemetry:opentelemetry-context:1.49.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-context:1.53.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-exporter-logging:1.53.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-sdk:1.53.0 - https://github.com/open-telemetry/opentelemetry-java)
@@ -353,7 +353,6 @@ List of third-party dependencies grouped by their license type.
MIT License
* Animal Sniffer Annotations (org.codehaus.mojo:animal-sniffer-annotations:1.24 - https://www.mojohaus.org/animal-sniffer/animal-sniffer-annotations)
- * Checker Qual (org.checkerframework:checker-qual:3.42.0 - https://checkerframework.org/)
* dd-plist (com.googlecode.plist:dd-plist:1.28 - http://www.github.com/3breadt/dd-plist)
* JCodings (org.jruby.jcodings:jcodings:1.0.58 - http://nexus.sonatype.org/oss-repository-hosting.html/jcodings)
* Joni (org.jruby.joni:joni:2.2.1 - http://nexus.sonatype.org/oss-repository-hosting.html/joni)
diff --git a/core/pom.xml b/core/pom.xml
index 86d54c35..2fd6d374 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -46,14 +46,14 @@ under the License.
<commons.lang.version>2.6</commons.lang.version>
<wiremock.version>3.13.1</wiremock.version>
<rometools.version>2.1.0</rometools.version>
- <selenium.version>4.35.0</selenium.version>
<cli.version>1.10.0</cli.version>
<okhttp.version>5.1.0</okhttp.version>
<caffeine.version>3.2.2</caffeine.version>
<xsoup.version>0.3.7</xsoup.version>
<awaitility.version>4.3.0</awaitility.version>
+ <guava.version>33.4.8-jre</guava.version>
<jacoco.haltOnFailure>true</jacoco.haltOnFailure>
- <jacoco.classRatio>0.73</jacoco.classRatio>
+ <jacoco.classRatio>0.72</jacoco.classRatio>
<jacoco.instructionRatio>0.54</jacoco.instructionRatio>
<jacoco.methodRatio>0.62</jacoco.methodRatio>
<jacoco.branchRatio>0.49</jacoco.branchRatio>
@@ -220,18 +220,6 @@ under the License.
<version>${commons.lang.version}</version>
</dependency>
- <dependency>
- <groupId>org.seleniumhq.selenium</groupId>
- <artifactId>selenium-remote-driver</artifactId>
- <version>${selenium.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.seleniumhq.selenium</groupId>
- <artifactId>selenium-support</artifactId>
- <version>${selenium.version}</version>
- </dependency>
-
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>xsoup</artifactId>
@@ -258,19 +246,12 @@ under the License.
</dependency>
<dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>selenium</artifactId>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.seleniumhq.selenium</groupId>
- <artifactId>selenium-chrome-driver</artifactId>
- <version>${selenium.version}</version>
- <scope>test</scope>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
</dependency>
-
</dependencies>
+
<dependencyManagement>
<dependencies>
<dependency>
diff --git a/core/src/main/resources/crawler-default.yaml b/core/src/main/resources/crawler-default.yaml
index 27021730..56350b67 100644
--- a/core/src/main/resources/crawler-default.yaml
+++ b/core/src/main/resources/crawler-default.yaml
@@ -208,30 +208,6 @@ config:
# or transferred protocol metadata must also be prefixed.
protocol.md.prefix: "protocol."
- # navigationfilters.config.file: "navigationfilters.json"
- # selenium.addresses: "http://localhost:9515"
-
- selenium.tracing: false
-
- # rely on selenium's default values
- # set to a value >= 0 to override
- selenium.timeouts:
- script: -1
- pageLoad: -1
- implicit: -1
-
- # selenium.capabilities:
- # a browser name is required
- # browserName:"chrome"
- # illustrates the use of the variable for user agent
- # phantomjs.page.settings.userAgent: "$userAgent"
- # ChromeDriver config
- # goog:chromeOptions:
- # args:
- # - "--headless"
- # - "--disable-gpu"
- # - "--mute-audio"
-
# no url or parsefilters by default
# parsefilters.config.file: "parsefilters.json"
# urlfilters.config.file: "urlfilters.json"
diff --git a/external/selenium/README.md b/external/selenium/README.md
new file mode 100644
index 00000000..f71bba0c
--- /dev/null
+++ b/external/selenium/README.md
@@ -0,0 +1,43 @@
+# Selenium Protocol for Apache StormCrawler
+
+This project provides a **[Selenium](https://www.selenium.dev/)-based protocol implementation** for Apache StormCrawler.
+
+---
+
+## Overview
+
+The [Selenium](https://www.selenium.dev) protocol allows StormCrawler to interact with dynamic web pages using Selenium WebDriver. It is particularly useful for crawling JavaScript-heavy sites that require a real browser environment.
+
+---
+
+## Configuration
+
+Add `selenium-conf.yaml` to your topology configuration. Below is a sample configuration:
+
+```yaml
+# navigationfilters.config.file: "navigationfilters.json"
+# selenium.addresses: "http://localhost:9515"
+
+# Enable or disable Selenium tracing (default: false)
+selenium.tracing: false
+
+# Selenium timeouts (rely on Selenium defaults if set to -1)
+selenium.timeouts:
+ script: -1 # Maximum time for scripts to run
+ pageLoad: -1 # Maximum time to wait for page load
+ implicit: -1 # Implicit wait time for finding elements
+
+# Selenium capabilities
+# selenium.capabilities:
+# browserName: "chrome" # Required: choose your browser
+# phantomjs.page.settings.userAgent: "$userAgent" # Example: set custom user agent
+#
+# # ChromeDriver specific options
+# goog:chromeOptions:
+# args:
+# - "--headless" # Run Chrome in headless mode
+# - "--disable-gpu" # Disable GPU acceleration
+# - "--mute-audio" # Mute audio output
+
+
+
diff --git a/external/selenium/pom.xml b/external/selenium/pom.xml
new file mode 100644
index 00000000..36c815b8
--- /dev/null
+++ b/external/selenium/pom.xml
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.stormcrawler</groupId>
+ <artifactId>stormcrawler-external</artifactId>
+ <version>3.4.1-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>stormcrawler-selenium</artifactId>
+ <packaging>jar</packaging>
+
+ <name>stormcrawler-selenium</name>
+ <url>https://github.com/apache/stormcrawler/tree/master/external/selenium</url>
+ <description>Selenium-based Protocol for StormCrawler</description>
+
+ <properties>
+ <selenium.version>4.35.0</selenium.version>
+ <awaitility.version>4.3.0</awaitility.version>
+ <wiremock.version>3.13.1</wiremock.version>
+ <jacoco.haltOnFailure>true</jacoco.haltOnFailure>
+ <jacoco.classRatio>1.00</jacoco.classRatio>
+ <jacoco.instructionRatio>0.69</jacoco.instructionRatio>
+ <jacoco.methodRatio>0.80</jacoco.methodRatio>
+ <jacoco.branchRatio>0.62</jacoco.branchRatio>
+ <jacoco.lineRatio>0.66</jacoco.lineRatio>
+ <jacoco.complexityRatio>0.55</jacoco.complexityRatio>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.seleniumhq.selenium</groupId>
+ <artifactId>selenium-remote-driver</artifactId>
+ <version>${selenium.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.seleniumhq.selenium</groupId>
+ <artifactId>selenium-support</artifactId>
+ <version>${selenium.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.stormcrawler</groupId>
+ <artifactId>stormcrawler-core</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.wiremock</groupId>
+ <artifactId>wiremock</artifactId>
+ <version>${wiremock.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.awaitility</groupId>
+ <artifactId>awaitility</artifactId>
+ <version>${awaitility.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>selenium</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.seleniumhq.selenium</groupId>
+ <artifactId>selenium-chrome-driver</artifactId>
+ <version>${selenium.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ </dependencies>
+
+</project>
diff --git a/external/selenium/selenium-conf.yaml b/external/selenium/selenium-conf.yaml
new file mode 100644
index 00000000..83250ed7
--- /dev/null
+++ b/external/selenium/selenium-conf.yaml
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+config:
+ # navigationfilters.config.file: "navigationfilters.json"
+ # selenium.addresses: "http://localhost:9515"
+
+ selenium.tracing: false
+
+ # rely on selenium's default values
+ # set to a value >= 0 to override
+ selenium.timeouts:
+ script: -1
+ pageLoad: -1
+ implicit: -1
+
+ # selenium.capabilities:
+ # a browser name is required
+ # browserName:"chrome"
+ # illustrates the use of the variable for user agent
+ # phantomjs.page.settings.userAgent: "$userAgent"
+ # ChromeDriver config
+ # goog:chromeOptions:
+ # args:
+ # - "--headless"
+ # - "--disable-gpu"
+ # - "--mute-audio"
+
+
diff --git a/core/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilter.java b/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilter.java
similarity index 100%
rename from core/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilter.java
rename to external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilter.java
diff --git a/core/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilters.java b/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilters.java
similarity index 100%
rename from core/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilters.java
rename to external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/NavigationFilters.java
diff --git a/core/src/main/java/org/apache/stormcrawler/protocol/selenium/RemoteDriverProtocol.java b/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/RemoteDriverProtocol.java
similarity index 100%
rename from core/src/main/java/org/apache/stormcrawler/protocol/selenium/RemoteDriverProtocol.java
rename to external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/RemoteDriverProtocol.java
diff --git a/core/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java b/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java
similarity index 100%
rename from core/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java
rename to external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java
diff --git a/core/src/test/java/org/apache/stormcrawler/protocol/selenium/ProtocolTest.java b/external/selenium/src/test/java/org/apache/stormcrawler/protocol/selenium/ProtocolTest.java
similarity index 100%
rename from core/src/test/java/org/apache/stormcrawler/protocol/selenium/ProtocolTest.java
rename to external/selenium/src/test/java/org/apache/stormcrawler/protocol/selenium/ProtocolTest.java
diff --git a/pom.xml b/pom.xml
index 484f6c6c..2bde4648 100644
--- a/pom.xml
+++ b/pom.xml
@@ -645,6 +645,7 @@ under the License.
<module>external/langid</module>
<module>external/opensearch</module>
<module>external/playwright</module>
+ <module>external/selenium</module>
<module>external/solr</module>
<module>external/sql</module>
<module>external/tika</module>