[jira] [Created] (NUTCH-2935) DeduplicationJob: failure on URLs with invalid percent encoding

["Sebastian Nagel (Jira)" <ji...@apache.org>]

Sebastian Nagel created NUTCH-2935:
--------------------------------------

             Summary: DeduplicationJob: failure on URLs with invalid percent encoding
                 Key: NUTCH-2935
                 URL: https://issues.apache.org/jira/browse/NUTCH-2935
             Project: Nutch
          Issue Type: Bug
          Components: crawldb
    Affects Versions: 1.18
            Reporter: Sebastian Nagel
            Assignee: Sebastian Nagel
             Fix For: 1.19


The DeduplicationJob may fail with an IllegalArgumentException on invalid percent encodings in URLs:
{noformat}
2021-11-25 04:36:41,747 INFO mapreduce.Job: Task Id : attempt_1637669672674_0018_r_000193_0, Status : FAILED
Error: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 0 in: "YR"
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:232)
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
        at org.apache.nutch.crawl.DeduplicationJob$DedupReducer.getDuplicate(DeduplicationJob.java:211)
...
Exception in thread "main" java.lang.RuntimeException: Crawl job did not succeed, job status:FAILED, reason: Task failed task_1637669672674_0018_r_000193
Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0
{noformat}

The IllegalArgumentException should be caught, logged and if only one of the two URLs with duplicated content is invalid, it should be flagged as duplicate while the valid URL "survives".



--
This message was sent by Atlassian Jira
(v8.20.1#820001)