You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by rz...@apache.org on 2025/07/16 09:10:16 UTC
(stormcrawler) branch main updated: Bump com.github.crawler-commons:crawler-commons from 1.4 to 1.5 (#1590)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
The following commit(s) were added to refs/heads/main by this push:
new d66c4a34 Bump com.github.crawler-commons:crawler-commons from 1.4 to 1.5 (#1590)
d66c4a34 is described below
commit d66c4a34a569ac6347aed72349826c52e5228723
Author: dependabot[bot] <49...@users.noreply.github.com>
AuthorDate: Wed Jul 16 11:10:13 2025 +0200
Bump com.github.crawler-commons:crawler-commons from 1.4 to 1.5 (#1590)
* Bump com.github.crawler-commons:crawler-commons from 1.4 to 1.5
Bumps [com.github.crawler-commons:crawler-commons](https://github.com/crawler-commons/crawler-commons) from 1.4 to 1.5.
- [Release notes](https://github.com/crawler-commons/crawler-commons/releases)
- [Changelog](https://github.com/crawler-commons/crawler-commons/blob/master/CHANGES.txt)
- [Commits](https://github.com/crawler-commons/crawler-commons/compare/crawler-commons-1.4...crawler-commons-1.5)
---
updated-dependencies:
- dependency-name: com.github.crawler-commons:crawler-commons
dependency-version: '1.5'
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <su...@github.com>
* Fix compile error
* format
---------
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
Co-authored-by: Richard Zowalla <rz...@apache.org>
---
THIRD-PARTY.txt | 2 +-
core/pom.xml | 2 +-
core/src/main/java/org/apache/stormcrawler/protocol/RobotRules.java | 6 ++++++
3 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/THIRD-PARTY.txt b/THIRD-PARTY.txt
index 43d35b17..0f92bddc 100644
--- a/THIRD-PARTY.txt
+++ b/THIRD-PARTY.txt
@@ -130,7 +130,7 @@ List of third-party dependencies grouped by their license type.
* Commons Lang (commons-lang:commons-lang:2.6 - http://commons.apache.org/lang/)
* Commons Logging (commons-logging:commons-logging:1.1.3 - http://commons.apache.org/proper/commons-logging/)
* compiler (com.github.spullara.mustache.java:compiler:0.9.14 - http://github.com/spullara/mustache.java)
- * Crawler-commons (com.github.crawler-commons:crawler-commons:1.4 - https://github.com/crawler-commons/crawler-commons)
+ * Crawler-commons (com.github.crawler-commons:crawler-commons:1.5 - https://github.com/crawler-commons/crawler-commons)
* error-prone annotations (com.google.errorprone:error_prone_annotations:2.28.0 - https://errorprone.info/error_prone_annotations)
* error-prone annotations (com.google.errorprone:error_prone_annotations:2.36.0 - https://errorprone.info/error_prone_annotations)
* error-prone annotations (com.google.errorprone:error_prone_annotations:2.40.0 - https://errorprone.info/error_prone_annotations)
diff --git a/core/pom.xml b/core/pom.xml
index 0e3e2414..a3c8b148 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -37,7 +37,7 @@ under the License.
<properties>
<!-- dependency versions -->
- <crawler-commons.version>1.4</crawler-commons.version>
+ <crawler-commons.version>1.5</crawler-commons.version>
<jsoup.version>1.21.1</jsoup.version>
<icu4j.version>77.1</icu4j.version>
<xerces.version>2.12.2</xerces.version>
diff --git a/core/src/main/java/org/apache/stormcrawler/protocol/RobotRules.java b/core/src/main/java/org/apache/stormcrawler/protocol/RobotRules.java
index 712538a6..71367646 100644
--- a/core/src/main/java/org/apache/stormcrawler/protocol/RobotRules.java
+++ b/core/src/main/java/org/apache/stormcrawler/protocol/RobotRules.java
@@ -17,6 +17,7 @@
package org.apache.stormcrawler.protocol;
import crawlercommons.robots.BaseRobotRules;
+import java.net.URL;
import java.util.List;
/**
@@ -38,6 +39,11 @@ public class RobotRules extends crawlercommons.robots.BaseRobotRules {
return base.isAllowed(url);
}
+ @Override
+ public boolean isAllowed(URL url) {
+ return base.isAllowed(url);
+ }
+
@Override
public boolean isAllowAll() {
return base.isAllowAll();