You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Fredrik Andersson <fi...@gmail.com> on 2005/07/28 14:28:59 UTC

0.7-dev, the search scoring

Hey guys!

I just ported a lot of old 0.6 code to 0.7-dev/mapred. Lots of stuff
has changed I see! One thing I can't quite grasp though, is why the
Hit.getScore() has been removed in favour for the TopDocs-thingie
instead? I wrote a quick add-on to support getting the score straight
from the hit, which worked fine, but it would be nice to hear a reason
as to why the method was removed in the first place!

Also, are there any "secret" WIKIs, mailinglists, forums or similar
for the 0.7 development? Would be very interesting to see what's
cooking!

Greetings,
Fredrik

Re: recursion: see recursion

Posted by yoursoft <yo...@freemail.hu>.
Insert in top of the regex-urlfilter.txt:
-http://.*(/.+?)/.*?\1/.*?\1.*?/

EM wrotte:

>What to do when encountering sites where nutch falls into recursion mode?
>
>Currently I'm solving this by removing these sites with the regex filter,
>but, is anything under development currently?
>
>By recursion I mean nutch fetching 
><sfdsdf>.com/<sth>/<sth>/<sth>/<sth>/<sth>/<sth> and on and on....
>
>Any tricks to limis the folder depth in the fetch mode?
>
>E.
>
>
>
>  
>


recursion: see recursion

Posted by EM <em...@cpuedge.com>.
What to do when encountering sites where nutch falls into recursion mode?

Currently I'm solving this by removing these sites with the regex filter,
but, is anything under development currently?

By recursion I mean nutch fetching 
<sfdsdf>.com/<sth>/<sth>/<sth>/<sth>/<sth>/<sth> and on and on....

Any tricks to limis the folder depth in the fetch mode?

E.


Re: 0.7-dev, the search scoring

Posted by Fredrik Andersson <fi...@gmail.com>.
Ah God, I am stupid ... thanks for that, Doug! I must have a bad
coding day today : )

Fredrik

On 7/28/05, Doug Cutting <cu...@nutch.org> wrote:
> Fredrik Andersson wrote:
> > I just ported a lot of old 0.6 code to 0.7-dev/mapred. Lots of stuff
> > has changed I see! One thing I can't quite grasp though, is why the
> > Hit.getScore() has been removed in favour for the TopDocs-thingie
> > instead?
> 
> Hit.getScore() was generalized to Hit.getSortValue() in order to support
> sorting results by things other than score.  If you sort by score, as is
> the default, then ((FloatWritable)Hit.getSortValue()).get() is the
> score.  But if you sort by, e.g., a date string, then
> ((UTF8)Hit.getSortValue()).toString() is the date string sorted on, and
> the score is unavailable.  Perhaps the score should be made available
> regardless?
> 
> Doug
>

Re: 0.7-dev, the search scoring

Posted by Doug Cutting <cu...@nutch.org>.
Fredrik Andersson wrote:
> I just ported a lot of old 0.6 code to 0.7-dev/mapred. Lots of stuff
> has changed I see! One thing I can't quite grasp though, is why the
> Hit.getScore() has been removed in favour for the TopDocs-thingie
> instead?

Hit.getScore() was generalized to Hit.getSortValue() in order to support 
sorting results by things other than score.  If you sort by score, as is 
the default, then ((FloatWritable)Hit.getSortValue()).get() is the 
score.  But if you sort by, e.g., a date string, then 
((UTF8)Hit.getSortValue()).toString() is the date string sorted on, and 
the score is unavailable.  Perhaps the score should be made available 
regardless?

Doug

Re: 0.7-dev, the search scoring

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jul 28, 2005, at 8:28 AM, Fredrik Andersson wrote:
> Also, are there any "secret" WIKIs, mailinglists, forums or similar
> for the 0.7 development?

No.  All discussions are in the open right here on nutch-dev and  
nutch-user.

     Erik