[jira] Commented: (NUTCH-628) Host database to keep track of host-level information

["Otis Gospodnetic (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641059#action_12641059 ] 

Otis Gospodnetic commented on NUTCH-628:
----------------------------------------

After seeing NUTCH-650 I have a feeling this issue should be closed with "Won't Fix".  Thoughts?

Does it make sense to save and commit the Domain Statistics patch, though?
(to be ported to Hbase approach later, once Hbase stuff from NUTCH-650 is in)


> Host database to keep track of host-level information
> -----------------------------------------------------
>
>                 Key: NUTCH-628
>                 URL: https://issues.apache.org/jira/browse/NUTCH-628
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher, generator
>            Reporter: Otis Gospodnetic
>         Attachments: NUTCH-628-DomainStatistics.patch, NUTCH-628-HostDb.patch
>
>
> Nutch would benefit from having a DB with per-host/domain/TLD information.  For instance, Nutch could detect hosts that are timing out, store information about that in this DB.  Segment/fetchlist Generator could then skip such hosts, so they don't slow down the fetch job.  Another good use for such a DB is keeping track of various host scores, e.g. spam score.
> From the recent thread on nutch-user@lucene:
> Otis asked:
> > While we are at it, how would one go about implementing this DB, as far as its structures go?
> Andrzej said:
> The easiest I can imagine is to use something like <Text, MapWritable>.
> This way you could store arbitrary information under arbitrary keys.
> I.e. a single database then could keep track of aggregate statistics at
> different levels, e.g. TLD, domain, host, ip range, etc. The basic set
> of statistics could consist of a few predefined gauges, totals and averages.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.