You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Stefan Scheffler <ss...@avantgarde-labs.de> on 2012/10/02 09:19:56 UTC

priorised/scored fetching

Hi.
I crawl a webdatabase for *.html, *.pdf and *.doc documents, with a 
given topN. I want nutch to fetch first all of the html documents, then 
pdf and at last doc, because html is more important than pdf and so on.
Is there a way to make nutch follow such rules (maybe with a scoring 
algorithm)?

Regards
Stefan

-- 
Stefan Scheffler
Avantgarde Labs GbR
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: sscheffler@avantgarde-labs.de


Re: priorised/scored fetching

Posted by Stefan Scheffler <ss...@avantgarde-labs.de>.
Ah Ok. Thank you.
This sounds like my intention :)

Regards
Stefan

Am 02.10.2012 10:34, schrieb Julien Nioche:
> you should be able to do that with a custom scoring filter and give a score
> based on the mime type
>
> On 2 October 2012 08:28, Markus Jelsma <ma...@openindex.io> wrote:
>
>> Hi - There's nothing like that yet. What you can do is run a custom URL
>> filter for the generate step, allowing only HTML files and use your
>> standard URL filter for the other steps.
>>
>>
>>
>> -----Original message-----
>>> From:Stefan Scheffler <ss...@avantgarde-labs.de>
>>> Sent: Tue 02-Oct-2012 09:24
>>> To: user@nutch.apache.org
>>> Subject: priorised/scored fetching
>>>
>>> Hi.
>>> I crawl a webdatabase for *.html, *.pdf and *.doc documents, with a
>>> given topN. I want nutch to fetch first all of the html documents, then
>>> pdf and at last doc, because html is more important than pdf and so on.
>>> Is there a way to make nutch follow such rules (maybe with a scoring
>>> algorithm)?
>>>
>>> Regards
>>> Stefan
>>>
>>> --
>>> Stefan Scheffler
>>> Avantgarde Labs GbR
>>> Löbauer Straße 19, 01099 Dresden
>>> Telefon: + 49 (0) 351 21590834
>>> Email: sscheffler@avantgarde-labs.de
>>>
>>>
>
>


-- 
Stefan Scheffler
Avantgarde Labs GbR
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: sscheffler@avantgarde-labs.de


Re: priorised/scored fetching

Posted by Julien Nioche <li...@gmail.com>.
you should be able to do that with a custom scoring filter and give a score
based on the mime type

On 2 October 2012 08:28, Markus Jelsma <ma...@openindex.io> wrote:

> Hi - There's nothing like that yet. What you can do is run a custom URL
> filter for the generate step, allowing only HTML files and use your
> standard URL filter for the other steps.
>
>
>
> -----Original message-----
> > From:Stefan Scheffler <ss...@avantgarde-labs.de>
> > Sent: Tue 02-Oct-2012 09:24
> > To: user@nutch.apache.org
> > Subject: priorised/scored fetching
> >
> > Hi.
> > I crawl a webdatabase for *.html, *.pdf and *.doc documents, with a
> > given topN. I want nutch to fetch first all of the html documents, then
> > pdf and at last doc, because html is more important than pdf and so on.
> > Is there a way to make nutch follow such rules (maybe with a scoring
> > algorithm)?
> >
> > Regards
> > Stefan
> >
> > --
> > Stefan Scheffler
> > Avantgarde Labs GbR
> > Löbauer Straße 19, 01099 Dresden
> > Telefon: + 49 (0) 351 21590834
> > Email: sscheffler@avantgarde-labs.de
> >
> >
>



-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

RE: priorised/scored fetching

Posted by Markus Jelsma <ma...@openindex.io>.
Hi - There's nothing like that yet. What you can do is run a custom URL filter for the generate step, allowing only HTML files and use your standard URL filter for the other steps.

 
 
-----Original message-----
> From:Stefan Scheffler <ss...@avantgarde-labs.de>
> Sent: Tue 02-Oct-2012 09:24
> To: user@nutch.apache.org
> Subject: priorised/scored fetching
> 
> Hi.
> I crawl a webdatabase for *.html, *.pdf and *.doc documents, with a 
> given topN. I want nutch to fetch first all of the html documents, then 
> pdf and at last doc, because html is more important than pdf and so on.
> Is there a way to make nutch follow such rules (maybe with a scoring 
> algorithm)?
> 
> Regards
> Stefan
> 
> -- 
> Stefan Scheffler
> Avantgarde Labs GbR
> Löbauer Straße 19, 01099 Dresden
> Telefon: + 49 (0) 351 21590834
> Email: sscheffler@avantgarde-labs.de
> 
>