You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Roberto Gardenier (JIRA)" <ji...@apache.org> on 2012/05/01 13:57:52 UTC
[jira] [Commented] (NUTCH-1343) Crawl sites with hashtags in url
[ https://issues.apache.org/jira/browse/NUTCH-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265751#comment-13265751 ]
Roberto Gardenier commented on NUTCH-1343:
------------------------------------------
Markus Jelsma,
I got notified that you have closed my jira ticket, chaning its resolution status to Invalid.
I wonder why you have closed my ticket and marked it invalid as i did not commit any changes or solutions?
With kind regards,
Roberto Gardenier
-----Oorspronkelijk bericht-----
Van: Markus Jelsma (JIRA) [mailto:jira@apache.org]
Verzonden: dinsdag 1 mei 2012 13:40
Aan: r.gardenier@simgroep.nl
Onderwerp: [jira] [Closed] (NUTCH-1343) Crawl sites with hashtags in url
[ https://issues.apache.org/jira/browse/NUTCH-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1343.
--------------------------------
Resolution: Invalid
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
> Crawl sites with hashtags in url
> --------------------------------
>
> Key: NUTCH-1343
> URL: https://issues.apache.org/jira/browse/NUTCH-1343
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Roberto Gardenier
> Priority: Blocker
>
> Hello,
> Im currently trying to crawl a site which uses hashtags in the urls. I dont seem to get any results and Im hoping im just overlooking something.
> Site structure is as follows:
> http://domain.com (landingpage)
> http://domain.com/#/page1
> http://domain.com/#/page1/subpage1
> http://domain.com/#/page2
> http://domain.com/#/page2/subpage1
> and so on.
> I've pointed nutch to http://domain.com as start url and in my filter i've placed all kind of rules.
> First i thought this would be sufficient:
> +http\://domain\.com\/#
> But then i realised that # is used for comments so i escaped it:
> +http\://domain\.com\/\#
> Still no results. So i thought i could use the asterix for it:
> +http\://domain\.com\/*
> Still no luck.. So i started using various regex stuff but without success.
> I noticed the following messages in hadoop.log:
> INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
> Ive researched on this setting but i dont know for sure if this affects my problem in a way. This property is set to false in my configs.
> I dont know if this is even related to the situation above but maybe it helps.
> Any help is very much appreciated! I've tried googling the problem but i couldnt find documentation or anyone else with this problem.
> Many thanks in advance.
> With kind regard,
> Roberto Gardenier
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira