[jira] [Commented] (NUTCH-2935) DeduplicationJob: failure on URLs with invalid percent encoding

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476045#comment-17476045 ] 

ASF GitHub Bot commented on NUTCH-2935:
---------------------------------------

sebastian-nagel opened a new pull request #723:
URL: https://github.com/apache/nutch/pull/723


   - catch IllegalArgumentException when unescaping percent-encoding in URLs
   - if one URL of two compared URLs is valid, keep it as non-duplicate
   - add unit tests for DeduplicationJob


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> DeduplicationJob: failure on URLs with invalid percent encoding
> ---------------------------------------------------------------
>
>                 Key: NUTCH-2935
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2935
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb
>    Affects Versions: 1.18
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.19
>
>
> The DeduplicationJob may fail with an IllegalArgumentException on invalid percent encodings in URLs:
> {noformat}
> 2021-11-25 04:36:41,747 INFO mapreduce.Job: Task Id : attempt_1637669672674_0018_r_000193_0, Status : FAILED
> Error: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 0 in: "YR"
>         at java.base/java.net.URLDecoder.decode(URLDecoder.java:232)
>         at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
>         at org.apache.nutch.crawl.DeduplicationJob$DedupReducer.getDuplicate(DeduplicationJob.java:211)
> ...
> Exception in thread "main" java.lang.RuntimeException: Crawl job did not succeed, job status:FAILED, reason: Task failed task_1637669672674_0018_r_000193
> Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0
> {noformat}
> The IllegalArgumentException should be caught, logged and if only one of the two URLs with duplicated content is invalid, it should be flagged as duplicate while the valid URL "survives".



--
This message was sent by Atlassian Jira
(v8.20.1#820001)