You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/09/19 18:55:08 UTC
[jira] [Created] (NUTCH-1115) Option to disable fixing of embedded
params in DomContentUtils
Option to disable fixing of embedded params in DomContentUtils
--------------------------------------------------------------
Key: NUTCH-1115
URL: https://issues.apache.org/jira/browse/NUTCH-1115
Project: Nutch
Issue Type: Improvement
Components: parser
Affects Versions: 1.3
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.4
Add option to disable fixing of embedded params:
http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1115) Option to disable fixing of embedded
params in DomContentUtils
Posted by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1115.
--------------------------------
Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch, NUTCH-1115-1.4-2.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1115) Option to disable fixing of
embedded params in DomContentUtils
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112512#comment-13112512 ]
Markus Jelsma commented on NUTCH-1115:
--------------------------------------
Here's finally an example of a correct and HTTP 200 URL:http://www.funkybabes.nl/;ROOOWAN/fotoboek . By default all outlinks fix embedded params so you'll get stuff like:
http://www.funkybabes.nl/M*a*n*d*y_87;ROOOWAN/fotoboek
http://www.funkybabes.nl/forumregels;ROOOWAN/fotoboek
which are actually 404's. With this fix enabled (or to say embedded params disabled) we get the correct outlinks without the ;ROOOWAN/fotoboek suffix.
The current behaviour is really nasty and can easily get out of hand.
btw: the patch is incorrect as it defaults the directive to FALSE.
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1115) Option to disable fixing of embedded
params in DomContentUtils
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1115:
---------------------------------
Attachment: NUTCH-1115-1.4-2.patch
Yes. Here's the complete patch for both parser implementation and nutch-default section.
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch, NUTCH-1115-1.4-2.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1115) Option to disable fixing of embedded
params in DomContentUtils
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1115.
----------------------------------
Resolution: Fixed
Committed for 1.4 in rev. 1174147. Fixes a significant pollution of the crawldb.
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch, NUTCH-1115-1.4-2.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1115) Option to disable fixing of
embedded params in DomContentUtils
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113148#comment-13113148 ]
Hudson commented on NUTCH-1115:
-------------------------------
Integrated in Nutch-branch-1.4 #14 (See [https://builds.apache.org/job/Nutch-branch-1.4/14/])
Recommitted CHANGELOG entry for NUTCH-1115. Was overwritten by NUTCH-1078 commit
NUTCH-1115 Option to disable fixing of URL embedded parameters in DomContentUtils
markus : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174222
Files :
* /nutch/branches/branch-1.4/CHANGES.txt
markus : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174147
Files :
* /nutch/branches/branch-1.4/conf/nutch-default.xml
* /nutch/branches/branch-1.4/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/DOMContentUtils.java
* /nutch/branches/branch-1.4/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/DOMContentUtils.java
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch, NUTCH-1115-1.4-2.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1115) Option to disable fixing of embedded
params in DomContentUtils
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1115:
---------------------------------
Attachment: NUTCH-1115-1.4-1.patch
Here's a patch for parse-tika's DomContentUtils. I added a param parser.fix.embeddedparams that defaults to true.
It fixes the issue!
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1115) Option to disable fixing of
embedded params in DomContentUtils
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112576#comment-13112576 ]
Julien Nioche commented on NUTCH-1115:
--------------------------------------
+1 Don't forget to add the same logic to DomContentUtils in Parse-Html
> Option to disable fixing of embedded params in DomContentUtils
> --------------------------------------------------------------
>
> Key: NUTCH-1115
> URL: https://issues.apache.org/jira/browse/NUTCH-1115
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4
>
> Attachments: NUTCH-1115-1.4-1.patch
>
>
> Add option to disable fixing of embedded params:
> http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html
> When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira