You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "jnioche (via GitHub)" <gi...@apache.org> on 2023/10/30 18:47:42 UTC
[PR] [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3 [nutch]
jnioche opened a new pull request, #793:
URL: https://github.com/apache/nutch/pull/793
and support gzipped input
See https://issues.apache.org/jira/browse/NUTCH-3017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel closed pull request #793: [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3
URL: https://github.com/apache/nutch/pull/793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on PR #793:
URL: https://github.com/apache/nutch/pull/793#issuecomment-1801814549
Thanks, @jnioche!
Merged into master, adding the lines to make use of Hadoop-provided compression codecs.
Successfully tested in local and pseudo-distributed mode with various codecs (gzip / .gz, bzip2, ZStandard / .zst).
One final note: if the fast-urlfilter is not found, the Nutch job (local mode) or the tasks (distributed mode) fail with an exception. I didn't change this behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on code in PR #793:
URL: https://github.com/apache/nutch/pull/793#discussion_r1377375552
##########
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java:
##########
@@ -181,9 +186,23 @@ public String filter(String url) {
public void reloadRules() throws IOException {
String fileRules = conf.get(URLFILTER_FAST_FILE);
- try (Reader reader = conf.getConfResourceAsReader(fileRules)) {
- reloadRules(reader);
+
+ InputStream is;
+
+ Path fileRulesPath = new Path(fileRules);
+ if (fileRulesPath.toUri().getScheme() != null) {
+ FileSystem fs = fileRulesPath.getFileSystem(conf);
+ is = fs.open(fileRulesPath);
+ }
Review Comment:
Since we have Hadoop, could try all supported compression codecs (gzip, bzip2, zstd, etc.). Something such as (not tested):
```java
CompressionCodecFactory cf = new CompressionCodecFactory(conf);
CompressionCodec codec = cf.getCodec(fileRulesPath);
if (codec != null) {
is = codec.createInputStream(is);
}
```
See [cf.getCodec(...)](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodecFactory.html#getCodec-org.apache.hadoop.fs.Path-) and [codec.createInputStream(...)](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createInputStream-java.io.InputStream-).
If the rules file is contained in the job jar, it shouldn't be compressed anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [NUTCH-3017] Allow fast-urlfilter to load from HDFS/S3 [nutch]
Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.
sebastian-nagel commented on code in PR #793:
URL: https://github.com/apache/nutch/pull/793#discussion_r1377375552
##########
src/plugin/urlfilter-fast/src/java/org/apache/nutch/urlfilter/fast/FastURLFilter.java:
##########
@@ -181,9 +186,23 @@ public String filter(String url) {
public void reloadRules() throws IOException {
String fileRules = conf.get(URLFILTER_FAST_FILE);
- try (Reader reader = conf.getConfResourceAsReader(fileRules)) {
- reloadRules(reader);
+
+ InputStream is;
+
+ Path fileRulesPath = new Path(fileRules);
+ if (fileRulesPath.toUri().getScheme() != null) {
+ FileSystem fs = fileRulesPath.getFileSystem(conf);
+ is = fs.open(fileRulesPath);
+ }
Review Comment:
Since we have Hadoop, could try all supported compression codecs (gzip, bzip2, zstd, etc.). Something such as (not tested):
```java
CompressionCodec codec = new CompressionCodecFactory(conf).getCodec(fileRulesPath);
if (codec != null) {
is = codec.createInputStream(is);
}
```
See [cf.getCodec(...)](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodecFactory.html#getCodec-org.apache.hadoop.fs.Path-) and [codec.createInputStream(...)](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createInputStream-java.io.InputStream-).
If the rules file is contained in the job jar, it shouldn't be compressed anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org