[jira] [Created] (NUTCH-2547) urlnormalizer-basic fails on special characters in path/query

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

Sebastian Nagel created NUTCH-2547:
--------------------------------------

             Summary: urlnormalizer-basic fails on special characters in path/query
                 Key: NUTCH-2547
                 URL: https://issues.apache.org/jira/browse/NUTCH-2547
             Project: Nutch
          Issue Type: Bug
          Components: plugin
    Affects Versions: 1.14
            Reporter: Sebastian Nagel
             Fix For: 1.15


If a URL contains one of the characters {{|"<>^`}} or a single {{%}} (not followed by a 2-characther hex-value), BasicURLNormalizer fails to normalize the URL path (here: remove {{/c/..}}):
{noformat}
% for c in "" $(echo '|%"^<>`' | grep -o .); do
    echo "http://www.example.com/a/c/../b/search?q=foobar$c"
  done \
  | nutch normalizerchecker -normalizer urlnormalizer-basic -stdin
Checking combination of these URLNormalizers: BasicURLNormalizer 
http://www.example.com/a/b/search?q=foobar
http://www.example.com/a/c/../b/search?q=foobar|
http://www.example.com/a/c/../b/search?q=foobar%
http://www.example.com/a/c/../b/search?q=foobar"
http://www.example.com/a/c/../b/search?q=foobar^
http://www.example.com/a/c/../b/search?q=foobar<
http://www.example.com/a/c/../b/search?q=foobar>
http://www.example.com/a/c/../b/search?q=foobar`
{noformat}

The reason is that these characters (should check for more, including control characters) are not valid as part of a [URI|https://docs.oracle.com/javase/9/docs/api/java/net/URI.html] (cf. [RFC3986|https://tools.ietf.org/html/rfc3986]). BasicURLNormalizer normalizes the path by converting the URL to a URI and calling [normalize()|https://docs.oracle.com/javase/9/docs/api/java/net/URI.html#normalize--].

There are two possible solutions:
# do not use java.net.URI
# ensure that every URL returned (or used internally) by urlnormalizer-basic is a valid URI (resp. its String representation).

I would opt for #2 because the class URI is used practically everywhere in Nutch and libraries (e.g. HttpClient). Any thoughts?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)