Newbie questions about followed links

[Jeroen Verhagen <je...@gmail.com>]

Hi all,

I started experimenting with Nutch using the NutchTutorial. I got a
succesful crawl to work using the command 'bin/nutch crawl urls -dir
crawl' (no limitations on depth or number of documents). I noticed
that Nutch finishes quite fast. When I looked in the source-html of
the main page being crawled I noticed that Nutch never followed links
that look like these:

<a href="content.jsp?objectid=22619">Route</a>
<br/>
<a href="content.jsp?objectid=5931">Openingstijden</a>
<br/>

Surely these links look ordinary enough to be seen and followed by
nutch? Could someone please tell me what could be causing these links
not be followed?

Thanks for any help,

Jeroen

Re: [SOLVED] Newbie questions about followed links

[djames <dj...@supinfo.com>]

Hi,

With your configuration of nutch, the crawl dont take the link with dynamic
parameter.

you must edit your regex filter at this line:
# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
-- 
View this message in context: http://www.nabble.com/Newbie-questions-about-followed-links-tf3368446.html#a9372291
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Newbie questions about followed links

[Jeroen Verhagen <je...@gmail.com>]

Hi Hasan,

On 3/8/07, Hasan Diwan <ha...@gmail.com> wrote:
>
> conf/urlfilter.txt.template contains the line:
> -[?*!@=]
>
> Remove the '?' and the links will be followed.

Thanks, that made it work.

I had to comment out the whole line '-[?*!@=]' to make it work though
? Even though there do not seem to be @ charachter in the links for
example?

--

regards,

Jeroen

Re: Newbie questions about followed links

[Paul Liddelow <pa...@gmail.com>]

exactly what I was going to say!

Cheers
Paul

On 3/8/07, Hasan Diwan <ha...@gmail.com> wrote:
>
> Sir:
> On 08/03/07, Jeroen Verhagen <je...@gmail.com> wrote:
> > Surely these links look ordinary enough to be seen and followed by
> > nutch? Could someone please tell me what could be causing these links
> > not be followed?
>
> conf/urlfilter.txt.template contains the line:
> -[?*!@=]
>
> Remove the '?' and the links will be followed.
>
> --
> Cheers,
> Hasan Diwan <ha...@gmail.com>
>

Re: Newbie questions about followed links

[Hasan Diwan <ha...@gmail.com>]

Sir:
On 08/03/07, Jeroen Verhagen <je...@gmail.com> wrote:
> Surely these links look ordinary enough to be seen and followed by
> nutch? Could someone please tell me what could be causing these links
> not be followed?

conf/urlfilter.txt.template contains the line:
-[?*!@=]

Remove the '?' and the links will be followed.

-- 
Cheers,
Hasan Diwan <ha...@gmail.com>