You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Doug Cook (JIRA)" <ji...@apache.org> on 2007/10/10 17:56:50 UTC
[jira] Created: (NUTCH-566) Sun's URL class has bug in creation of
relative query URLs
Sun's URL class has bug in creation of relative query URLs
----------------------------------------------------------
Key: NUTCH-566
URL: https://issues.apache.org/jira/browse/NUTCH-566
Project: Nutch
Issue Type: Bug
Components: fetcher
Affects Versions: 0.9.0, 0.8.1, 0.8
Environment: MacOS X and Linux (CentOS 4.5) both
Reporter: Doug Cook
Priority: Minor
I'm using 0.81, but this will affect all other versions as well.
Relative links of the form "?blah" are resolved incorrectly. For example, with a base URL of http://www.fleurie.org/entreprise.asp, and a relative link of "?id_entrep=111", Nutch will resolve this pair to the link
"http://www.fleurie.org/?id_entrep=111". No such URL exists, and all browsers I tried will resolve the pair to "http://www.fleurie.org/entreprise.asp?id_entrep=111".
I tracked this down to what could be called a bug in Sun's URL class. According to Sun's spec, they parse the relative URL according to RFC 2396. But the original RFC for relative links was RFC 1808, and the two RFCs differ in how they handle relative links beginning with "?". Most browsers (Netscape/Mozilla, IE, Safari) implemented RFC 1808, and stuck with it (for compatibility and also because the behavior makes more sense). Apparently even the people that wrote RFC 2396 recognized that this was a mistake, and the specified behavior was changed in RFC 3986 to match what browsers do.
For a discussion of this, see http://gbiv.com/protocols/uri/rev-2002/issues.html#003-relative-query
Sun's URL implementation, however, still implements RFC2396, as far as I can tell, and is out of step with the rest of the world.
This breaks link extraction on a number of sites.
I implemented a simple workaround, which I'm attaching. It is a static method to create URLs which behaves exactly as new URL(URL base, String relativePath), and I use it as a drop-in replacement for that in DOMContentUtils, Javascript link extraction, etc. Obviously, it really only matters wherever links are extracted. I haven't included the calling code from DOMContentUtils, etc. because my local versions are largely rewritten, but it should be pretty obvious.
I put it in the org.apache.nutch.net directory, but obviously feel free to move it to another place if you feel it belongs there!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-566) Sun's URL class has bug in creation
of relative query URLs
Posted by "Doug Cook (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539146 ]
Doug Cook commented on NUTCH-566:
---------------------------------
Hi Doğacan.
Thanks for following up. The issue has gotten a little more complicated since I first posted it.
I recently found, while looking for something else, that this is a dup of NUTCH-436. A solution for that has been committed.
Both solutions are incomplete, however. The problem with the fix for NUTCH-436 is that it is applied locally to DOMContentUtils, but the bug affects Nutch globally anywhere links are extracted. In this sense my solution is superior because it creates a central utility function for the workaround, and calls that from all the places links are extracted (DOMContentUtils is the most common, but there are also SWFParser, JSParseFilter, and TextParser).
The patch for Nutch 436 may be more complete in that it handles more cases than my simple fix. My fix handles the most common (only) case I've seen in practice (relative link beginning with '?'), and has the advantage of being simple, but the patch for Nutch-436 handles things like relative links beginning with ';'. I haven't had a chance to analyze them both to see which is more 'correct' (probably Nutch-436) and merge the two solutions, if necessary.
I've been so swamped trying to get my product launched (it's a 1-person company!) that I haven't had time to follow up on all the contributions I'd like to make. Usually I get a fix working just well enough to fit my local needs and then I have to move on to tackling the next bug. At this point I've got hundreds of local fixes and improvements, some of which are probably useful for the Nutch community... hopefully after I launch I'll have time both to contribute some goodies back and also to pick up all the improvements that have been made to the trunk since I branched my code...
-Doug
> Sun's URL class has bug in creation of relative query URLs
> ----------------------------------------------------------
>
> Key: NUTCH-566
> URL: https://issues.apache.org/jira/browse/NUTCH-566
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8, 0.8.1, 0.9.0
> Environment: MacOS X and Linux (CentOS 4.5) both
> Reporter: Doug Cook
> Priority: Minor
> Attachments: RelativeURL.java
>
>
> I'm using 0.81, but this will affect all other versions as well.
> Relative links of the form "?blah" are resolved incorrectly. For example, with a base URL of http://www.fleurie.org/entreprise.asp, and a relative link of "?id_entrep=111", Nutch will resolve this pair to the link
> "http://www.fleurie.org/?id_entrep=111". No such URL exists, and all browsers I tried will resolve the pair to "http://www.fleurie.org/entreprise.asp?id_entrep=111".
> I tracked this down to what could be called a bug in Sun's URL class. According to Sun's spec, they parse the relative URL according to RFC 2396. But the original RFC for relative links was RFC 1808, and the two RFCs differ in how they handle relative links beginning with "?". Most browsers (Netscape/Mozilla, IE, Safari) implemented RFC 1808, and stuck with it (for compatibility and also because the behavior makes more sense). Apparently even the people that wrote RFC 2396 recognized that this was a mistake, and the specified behavior was changed in RFC 3986 to match what browsers do.
> For a discussion of this, see http://gbiv.com/protocols/uri/rev-2002/issues.html#003-relative-query
> Sun's URL implementation, however, still implements RFC2396, as far as I can tell, and is out of step with the rest of the world.
> This breaks link extraction on a number of sites.
> I implemented a simple workaround, which I'm attaching. It is a static method to create URLs which behaves exactly as new URL(URL base, String relativePath), and I use it as a drop-in replacement for that in DOMContentUtils, Javascript link extraction, etc. Obviously, it really only matters wherever links are extracted. I haven't included the calling code from DOMContentUtils, etc. because my local versions are largely rewritten, but it should be pretty obvious.
> I put it in the org.apache.nutch.net directory, but obviously feel free to move it to another place if you feel it belongs there!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-566) Sun's URL class has bug in creation of
relative query URLs
Posted by "Doug Cook (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cook updated NUTCH-566:
----------------------------
Attachment: RelativeURL.java
Here's a static method to work around the problem.
> Sun's URL class has bug in creation of relative query URLs
> ----------------------------------------------------------
>
> Key: NUTCH-566
> URL: https://issues.apache.org/jira/browse/NUTCH-566
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8, 0.8.1, 0.9.0
> Environment: MacOS X and Linux (CentOS 4.5) both
> Reporter: Doug Cook
> Priority: Minor
> Attachments: RelativeURL.java
>
>
> I'm using 0.81, but this will affect all other versions as well.
> Relative links of the form "?blah" are resolved incorrectly. For example, with a base URL of http://www.fleurie.org/entreprise.asp, and a relative link of "?id_entrep=111", Nutch will resolve this pair to the link
> "http://www.fleurie.org/?id_entrep=111". No such URL exists, and all browsers I tried will resolve the pair to "http://www.fleurie.org/entreprise.asp?id_entrep=111".
> I tracked this down to what could be called a bug in Sun's URL class. According to Sun's spec, they parse the relative URL according to RFC 2396. But the original RFC for relative links was RFC 1808, and the two RFCs differ in how they handle relative links beginning with "?". Most browsers (Netscape/Mozilla, IE, Safari) implemented RFC 1808, and stuck with it (for compatibility and also because the behavior makes more sense). Apparently even the people that wrote RFC 2396 recognized that this was a mistake, and the specified behavior was changed in RFC 3986 to match what browsers do.
> For a discussion of this, see http://gbiv.com/protocols/uri/rev-2002/issues.html#003-relative-query
> Sun's URL implementation, however, still implements RFC2396, as far as I can tell, and is out of step with the rest of the world.
> This breaks link extraction on a number of sites.
> I implemented a simple workaround, which I'm attaching. It is a static method to create URLs which behaves exactly as new URL(URL base, String relativePath), and I use it as a drop-in replacement for that in DOMContentUtils, Javascript link extraction, etc. Obviously, it really only matters wherever links are extracted. I haven't included the calling code from DOMContentUtils, etc. because my local versions are largely rewritten, but it should be pretty obvious.
> I put it in the org.apache.nutch.net directory, but obviously feel free to move it to another place if you feel it belongs there!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-566) Sun's URL class has bug in creation
of relative query URLs
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578957#action_12578957 ]
Andrzej Bialecki commented on NUTCH-566:
-----------------------------------------
I agree that this should be put into a utility class. We already have one in trunk, org.apache.nutch.util.URLUtil. Could any of you provide an updated patch, relative to the current trunk?
> Sun's URL class has bug in creation of relative query URLs
> ----------------------------------------------------------
>
> Key: NUTCH-566
> URL: https://issues.apache.org/jira/browse/NUTCH-566
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8, 0.8.1, 0.9.0
> Environment: MacOS X and Linux (CentOS 4.5) both
> Reporter: Doug Cook
> Priority: Minor
> Attachments: RelativeURL.java
>
>
> I'm using 0.81, but this will affect all other versions as well.
> Relative links of the form "?blah" are resolved incorrectly. For example, with a base URL of http://www.fleurie.org/entreprise.asp, and a relative link of "?id_entrep=111", Nutch will resolve this pair to the link
> "http://www.fleurie.org/?id_entrep=111". No such URL exists, and all browsers I tried will resolve the pair to "http://www.fleurie.org/entreprise.asp?id_entrep=111".
> I tracked this down to what could be called a bug in Sun's URL class. According to Sun's spec, they parse the relative URL according to RFC 2396. But the original RFC for relative links was RFC 1808, and the two RFCs differ in how they handle relative links beginning with "?". Most browsers (Netscape/Mozilla, IE, Safari) implemented RFC 1808, and stuck with it (for compatibility and also because the behavior makes more sense). Apparently even the people that wrote RFC 2396 recognized that this was a mistake, and the specified behavior was changed in RFC 3986 to match what browsers do.
> For a discussion of this, see http://gbiv.com/protocols/uri/rev-2002/issues.html#003-relative-query
> Sun's URL implementation, however, still implements RFC2396, as far as I can tell, and is out of step with the rest of the world.
> This breaks link extraction on a number of sites.
> I implemented a simple workaround, which I'm attaching. It is a static method to create URLs which behaves exactly as new URL(URL base, String relativePath), and I use it as a drop-in replacement for that in DOMContentUtils, Javascript link extraction, etc. Obviously, it really only matters wherever links are extracted. I haven't included the calling code from DOMContentUtils, etc. because my local versions are largely rewritten, but it should be pretty obvious.
> I put it in the org.apache.nutch.net directory, but obviously feel free to move it to another place if you feel it belongs there!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-566) Sun's URL class has bug in creation
of relative query URLs
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539127 ]
Doğacan Güney commented on NUTCH-566:
-------------------------------------
I am going to commit this one, but I am not sure what needs to be updated besides parse-html and parse-js. Any suggestions?
> Sun's URL class has bug in creation of relative query URLs
> ----------------------------------------------------------
>
> Key: NUTCH-566
> URL: https://issues.apache.org/jira/browse/NUTCH-566
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8, 0.8.1, 0.9.0
> Environment: MacOS X and Linux (CentOS 4.5) both
> Reporter: Doug Cook
> Priority: Minor
> Attachments: RelativeURL.java
>
>
> I'm using 0.81, but this will affect all other versions as well.
> Relative links of the form "?blah" are resolved incorrectly. For example, with a base URL of http://www.fleurie.org/entreprise.asp, and a relative link of "?id_entrep=111", Nutch will resolve this pair to the link
> "http://www.fleurie.org/?id_entrep=111". No such URL exists, and all browsers I tried will resolve the pair to "http://www.fleurie.org/entreprise.asp?id_entrep=111".
> I tracked this down to what could be called a bug in Sun's URL class. According to Sun's spec, they parse the relative URL according to RFC 2396. But the original RFC for relative links was RFC 1808, and the two RFCs differ in how they handle relative links beginning with "?". Most browsers (Netscape/Mozilla, IE, Safari) implemented RFC 1808, and stuck with it (for compatibility and also because the behavior makes more sense). Apparently even the people that wrote RFC 2396 recognized that this was a mistake, and the specified behavior was changed in RFC 3986 to match what browsers do.
> For a discussion of this, see http://gbiv.com/protocols/uri/rev-2002/issues.html#003-relative-query
> Sun's URL implementation, however, still implements RFC2396, as far as I can tell, and is out of step with the rest of the world.
> This breaks link extraction on a number of sites.
> I implemented a simple workaround, which I'm attaching. It is a static method to create URLs which behaves exactly as new URL(URL base, String relativePath), and I use it as a drop-in replacement for that in DOMContentUtils, Javascript link extraction, etc. Obviously, it really only matters wherever links are extracted. I haven't included the calling code from DOMContentUtils, etc. because my local versions are largely rewritten, but it should be pretty obvious.
> I put it in the org.apache.nutch.net directory, but obviously feel free to move it to another place if you feel it belongs there!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.