Fetch only Blogs.

[Armando Gonçalves <ma...@gmail.com>]

Can Anyone tell-me if there is a way of nutch just fetch blogs during the
crawl process???
My current application has a white list of domains, any better idea ?

-- 
Armando Gonçalves
C.C 2005-2

Re: Fetch only Blogs.

[Laurent Laborde <ke...@gmail.com>]

On Thu, Feb 5, 2009 at 5:07 PM, Brian Ulicny <bu...@alum.mit.edu> wrote:
> Filtering by the blog host (blogspot.com, wordpress.com) is the most
> direct and efficient way to do about 90% of this, however it will miss
> self-hosted blogs.  Often these are the most important blogs in a
> locale: they self-host because they want total control of the traffic,
> advertising, etc.  The list of established blogger platforms is not very
> large.

Hi !
I'm sysadmin of over-blog.com
Please avoid running your test on this platform. :)
(you'll quickly hit our mod_limit anyway, so it's useless :p )

-- 
F4FQM
Kerunix Flan
Laurent Laborde

Re: Fetch only Blogs.

[Brian Ulicny <bu...@alum.mit.edu>]

Filtering by the blog host (blogspot.com, wordpress.com) is the most
direct and efficient way to do about 90% of this, however it will miss
self-hosted blogs.  Often these are the most important blogs in a
locale: they self-host because they want total control of the traffic,
advertising, etc.  The list of established blogger platforms is not very
large.

The presence of an RSS feed would get you most blogs, but also a lot of
non-blogs.

You could identify self-hosted blogs by identifying and parsing blog
rolls on the blogs identified using the domain filter above or look for
blog directory sources (DMOZ, Wikipedia) that would identify these in a
certain locale or topic domain.

A third option, is to start with a seed of known blogs, crawl outward,
then look for identifying phrases in the text of a web page that
identifies it as a blog, for example "Powered by Wordpress".  This again
will help identify self-hosted blogs, which are the main issue here.

In any case, there is no foolproof formula for identifying all blogs by
URL format only, so if you only want to crawl blogs, you'll have to do
some upfront work to identify the ones you want to crawl.

Brian Ulicny

On Thu, 5 Feb 2009 02:02:24 -0300, "Armando Gonçalves"
<ma...@gmail.com> said:
> Can Anyone tell-me if there is a way of nutch just fetch blogs during the
> crawl process???
> My current application has a white list of domains, any better idea ?
> 
> -- 
> Armando Gonçalves
> C.C 2005-2
-- 
  Brian Ulicny
  bulicny at alum dot mit dot edu
  home: 781-721-5746
  fax: 360-361-5746