You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2014/07/15 13:53:04 UTC
[jira] [Commented] (NUTCH-926) Redirections from META tag don't get
filtered
[ https://issues.apache.org/jira/browse/NUTCH-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061977#comment-14061977 ]
Hudson commented on NUTCH-926:
------------------------------
SUCCESS: Integrated in Nutch-trunk #2706 (See [https://builds.apache.org/job/Nutch-trunk/2706/])
NUTCH-926 Redirections from META tag don't get filtered (jnioche: http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1610659)
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/java/org/apache/nutch/parse/ParseOutputFormat.java
> Redirections from META tag don't get filtered
> ---------------------------------------------
>
> Key: NUTCH-926
> URL: https://issues.apache.org/jira/browse/NUTCH-926
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2
> Environment: gnu/linux centOs
> Reporter: Marco Novo
> Fix For: 1.9
>
> Attachments: NUTCH-926-trunk.patch, ParseOutputFormat.java.patch
>
>
> We have nutch set to crawl a domain urllist and we want to fetch only passed domains (hosts) not subdomains.
> So
> WWW.DOMAIN1.COM
> ..
> ..
> ..
> WWW.RIGHTDOMAIN.COM
> ..
> ..
> ..
> ..
> WWW.DOMAIN.COM
> We sets nutch to:
> NOT FOLLOW EXERNAL LINKS
> During crawling of WWW.RIGHTDOMAIN.COM
> if a page contains
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
> <html>
> <head>
> <title></title>
> <META http-equiv="refresh" content="0;
> url=http://WRONG.RIGHTDOMAIN.COM">
> </head>
> <body>
> </body>
> </html>
> Nutch continues to crawl the WRONG subdomains! But it should not do this!!
> During crawling of WWW.RIGHTDOMAIN.COM
> if a page contains
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
> <html>
> <head>
> <title></title>
> <META http-equiv="refresh" content="0;
> url=http://WWW.WRONGDOMAIN.COM">
> </head>
> <body>
> </body>
> </html>
> Nutch continues to crawl the WRONG domain! But it should not do this! If that we will spider all the web....
> We think the problem is in org.apache.nutch.parse ParseOutputFormat. We have done a patch so we will attach it
--
This message was sent by Atlassian JIRA
(v6.2#6252)