Limiting search/crawl to specific language

[Byron Miller <by...@yahoo.com>]

I know you can enable language detect during
index-more however is there a method to doing this
during the crawl?

I'm interested in building an index as english only
right now. what is the theory behind that? anyone have
any experience?

would it be building a huge black list, ignoring tlds
until you find a computational method or??? thoughts anyone?

Re: Limiting search/crawl to specific language

[Ken Krugler <kk...@transpac.com>]

>Is it possible to use the webdb (link db) to add rules
>to as you recursively process your data based upon
>link analysis?
>
>Such as adding a field for language so you can either
>score based on language (using language as your
>identifying "top" to build your link analysis from) or
>simply ignoring a language/tld or some field all
>together from generate process (essentially not
>refetching or pragmatically refetching or scaling your
>fetch on some priority)?

1. As others have noted, you want to run analysis code on the fetched 
page contents to identify (guess at) the language. This kind of code 
typically uses statistical models to generate probabilities for a set 
of languages that it's been trained on.

There was a past thread about this on Nutch developer list.

2. We modified Nutch to do something similar, at the point where 
outlinks are harvested. We stuff this special score into the page's 
nextScore field, since this isn't used other than for the (too slow) 
link analysis tool. Note that this was with the 0.7 code base.

3. Then in FetchListTool we use nextScore to order links for 
fetching. So for you, if the language of the page was used to 
calculate this score, then you could focus your crawl on 
English-content pages.

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-470-9200

Re: Limiting search/crawl to specific language

[Byron Miller <by...@yahoo.com>]

Is it possible to use the webdb (link db) to add rules
to as you recursively process your data based upon
link analysis?

Such as adding a field for language so you can either
score based on language (using language as your
identifying "top" to build your link analysis from) or
simply ignoring a language/tld or some field all
together from generate process (essentially not
refetching or pragmatically refetching or scaling your
fetch on some priority)?

 
does google have a page rank based upon language or is
it based upon entire dataset?  From what i can find
out temoa says there are 2 billion english only pages
and the rest are non english; is that a safe
assumption or just what teoma has been able to
process? I ask that because it would mean simplifying
your processing to manageable datasets for many
languages compared to trying the "whole internet" in a
single dataset.

I also think it's more feasable for me to focus on
english based sites as a whole since the cultural
differences and laws are enough to shy me away from
getting into the legal messes other nations can
potentially enforce or imply :)

-byron

--- Byron Miller <by...@yahoo.com> wrote:

> I know you can enable language detect during
> index-more however is there a method to doing this
> during the crawl?
> 
> I'm interested in building an index as english only
> right now. what is the theory behind that? anyone
> have
> any experience?
> 
> would it be building a huge black list, ignoring
> tlds
> until you find a computational method or??? thoughts
> anyone?
> 

Fetching in multiple machines setup.

[Gal Nitzan <gn...@usa.net>]

Hi,

If I understand what happened correctly. there is only one task working
for a fetch job.

How can I increase the number of fetchers for a fetch job?

Re: [Nutch-general] Limiting search/crawl to specific language

[og...@yahoo.com]

Hello Byron,

I know you can enable language detect during
index-more however is there a method to doing this
during the crawl?

OG: I don't recall the name of the class/method (UrlFilter?), nor do I recall whether that is an extension point for a plugin, but wherever the point in the fetch exection is where you can get a hold of the page content (text), you can potentially use the language identification framework to guess the language of the page.

I'm interested in building an index as english only
right now. what is the theory behind that? anyone have
any experience?

The theory is that you need to train the lang id library to learn to recognize English, and then throw sufficiently long chunks of text at it, and let it guess the lingo.  I believe Jerome has the training files somewhere (and a page about all this on the Wiki).

would it be building a huge black list, ignoring tlds
until you find a computational method or??? thoughts anyone?


If you really want to distinguish by language, then you need to use language recognition, not TLD filtering. A .hr (Croatia) site/page could easily be all in English, for instance, and a .com site may easily be in Chinese.

Otis