[jira] Updated: (NUTCH-100) New plugin urlfilter-db

["Gal Nitzan (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/NUTCH-100?page=all ]

Gal Nitzan updated NUTCH-100:
-----------------------------

    Attachment: urlfilter-db.tar.gz
                AddedDbURLFilter.patch

Fixed some issue with swarm cache (removed loading as daemon).
Code cleanup and remarks
Added some logging

> New plugin urlfilter-db
> -----------------------
>
>          Key: NUTCH-100
>          URL: http://issues.apache.org/jira/browse/NUTCH-100
>      Project: Nutch
>         Type: New Feature
>   Components: fetcher
>     Versions: 0.8-dev
>  Environment: MapRed
>     Reporter: Gal Nitzan
>     Priority: Trivial
>  Attachments: AddedDbURLFilter.patch, urlfilter-db.tar.gz, urlfilter-db.tar.gz
>
> Hi,
> I have written (not much) a new plugin, based on the URLFilter interface: urlfilter-db .
> The purpose of this plugin is to filter domains, i.e. I would like to crawl the world but to fetch only certain domains.
> The plugin uses a caching system (SwarmCache, easier to deploy than JCS) and on the back-end a database.
> For each url
>    filter is called
> end for
> filter
>  get the domain name from url
>   call cache.get domain
>   if not in cache try the database
>   if in database cache it and return it
>   return null
> end filter
> The plugin reads the cache size, jdbc driver, connection string, table to use and domain field from nutch-site.xml

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira