Ranking of injected urls vs crawled urls

[Harris Rappaport <hp...@gmail.com>]

Hi,
I am using nutch/solr to crawl and search some websites. A large number of
urls (all from the same domain) were injected directly, and the rest (from
different domains) were obtained by crawling. The problem is that the
injected urls seem to be ranking higher than the others, even when they
seem to match the search worse.

In nutch-default.xml it looks like the default scores for injected urls are
the same as everything else. Is something else driving the scores down to
less than the default for the others? How can I correct this?
Thanks,
Harris Rappaport

Re: Ranking of injected urls vs crawled urls

[Markus Jelsma <ma...@openindex.io>]

> I am using the default text search of solr, searching the text contents of
> web pages.
> 
> As I try to find a good example to show this problem, though (where an
> injected page is ranked above an uninjected one even though the uninjected
> one fits the search better), I can't seem to find one. I thought I had
> before, but it seems like I was mistaken. I think the apparent overscoring
> of the pages from injected urls were just because there were so much more
> of them than the crawled ones.
> 
> So unless you can think of a reason why injected urls would be scored
> higher than crawled ones, I think we can assume this wasn't actually a
> problem and was just my imagination.

So actually the returned results by Solr got you confused? Keep in mind that 
debugging scores in Solr alone can be difficult enough. Try to separate both 
systems in terms of debugging scores.
Nutch provides a boost field to Solr with the URL score as it comes from you 
scoring plugin, but Solr might not use it if you haven't told it to use it.


> 
> Thanks,
> Harris Rappaport
> 
> On Tue, Dec 6, 2011 at 4:18 AM, Markus Jelsma 
<ma...@openindex.io>wrote:
> > What kind of scoring are you using?
> > 
> > > Hi,
> > > I am using nutch/solr to crawl and search some websites. A large number
> > 
> > of
> > 
> > > urls (all from the same domain) were injected directly, and the rest
> > 
> > (from
> > 
> > > different domains) were obtained by crawling. The problem is that the
> > > injected urls seem to be ranking higher than the others, even when they
> > > seem to match the search worse.
> > > 
> > > In nutch-default.xml it looks like the default scores for injected urls
> > 
> > are
> > 
> > > the same as everything else. Is something else driving the scores down
> > > to less than the default for the others? How can I correct this?
> > > Thanks,
> > > Harris Rappaport

Re: Ranking of injected urls vs crawled urls

[Harris Rappaport <hp...@gmail.com>]

I am using the default text search of solr, searching the text contents of
web pages.

As I try to find a good example to show this problem, though (where an
injected page is ranked above an uninjected one even though the uninjected
one fits the search better), I can't seem to find one. I thought I had
before, but it seems like I was mistaken. I think the apparent overscoring
of the pages from injected urls were just because there were so much more
of them than the crawled ones.

So unless you can think of a reason why injected urls would be scored
higher than crawled ones, I think we can assume this wasn't actually a
problem and was just my imagination.

Thanks,
Harris Rappaport

On Tue, Dec 6, 2011 at 4:18 AM, Markus Jelsma <ma...@openindex.io>wrote:

> What kind of scoring are you using?
>
> > Hi,
> > I am using nutch/solr to crawl and search some websites. A large number
> of
> > urls (all from the same domain) were injected directly, and the rest
> (from
> > different domains) were obtained by crawling. The problem is that the
> > injected urls seem to be ranking higher than the others, even when they
> > seem to match the search worse.
> >
> > In nutch-default.xml it looks like the default scores for injected urls
> are
> > the same as everything else. Is something else driving the scores down to
> > less than the default for the others? How can I correct this?
> > Thanks,
> > Harris Rappaport
>

Re: Ranking of injected urls vs crawled urls

[Markus Jelsma <ma...@openindex.io>]

What kind of scoring are you using? 

> Hi,
> I am using nutch/solr to crawl and search some websites. A large number of
> urls (all from the same domain) were injected directly, and the rest (from
> different domains) were obtained by crawling. The problem is that the
> injected urls seem to be ranking higher than the others, even when they
> seem to match the search worse.
> 
> In nutch-default.xml it looks like the default scores for injected urls are
> the same as everything else. Is something else driving the scores down to
> less than the default for the others? How can I correct this?
> Thanks,
> Harris Rappaport