[jira] Updated: (NUTCH-882) Design a Host table in GORA

["Julien Nioche (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche updated NUTCH-882:
--------------------------------

    Attachment: NUTCH-882-v1.patch

Here is an initial version of the Host table. Very minimalistic for now but we'll make it evolve as we go. 

* src/gora/host.avsc : avro schema for the host table 
* modified gora mapping for sql backend
* src/java/org/apache/nutch/host/HostDBReader.java : displays the info about an entry of the host table (or all of them)
* src/java/org/apache/nutch/host/HostInjectorJob.java : mapreduce job which takes a seed host list and populates the host table with it
* src/java/org/apache/nutch/host/HostMDApplierJob.java : mapreduce job which reads through the webtable and adds metadata taken from the host table

The key for the host table is reverted - just like the WebPages in the webtable. The hosts are represented with a URL in order to know about the protocol as well. The metadata are similar to the ones in the WebPages.

Obviously we'll need to add more things to it but this is already useful for projecting metadata by host onto the webtable. You can do that by : 
# Injecting the host metadata into the host table 

{code}
./nutch org.apache.nutch.host.HostInjectorJob hostlist
{code}

# Applying the metadata to the WebTable

{code}
./nutch org.apache.nutch.host.HostMDApplierJob
{code}

# Check on the Webtable using 

{code}
./nutch org.apache.nutch.crawl.WebTableReader
{code}

You can of course do the same thing by putting the metadata in the URL seedlist then propagating them using  custom URL filters but this won't work for instance if a page is found from a page which belongs to a different host. The approach described here is a bit cleaner and can be used in a larger number of situations, e.g. when the metadata values are not known at the time of the URL seeding. We have developed a plugin for the detection of adult content which works quite well but found that the results were better after aggregating the stats at the host level, marking them as adult via a metadata, then project back onto the webtable and let a custom indexer use the value coming from the host to override the detection at the page level. Anyway, that was just an example :-)

I am planning to add more to this code in the short term but would like to hear your comments on it. In particular I am planning to : 
* add a class which populate the host table given a webtable and (possibly) add statistics to it at the same time
* maybe create a new plugin endpoint so that such statistics could be achieved using custom user functions
* write an example of such an endpoint which would add stats per status
* look into the management of robots.txt and sitemaps
* see how we could leverage these host-related metadata to give specific instructions for the fetching (number of threads, time between calls) etc...

Julien
 

   


> Design a Host table in GORA
> ---------------------------
>
>                 Key: NUTCH-882
>                 URL: https://issues.apache.org/jira/browse/NUTCH-882
>             Project: Nutch
>          Issue Type: New Feature
>    Affects Versions: 2.0
>            Reporter: Julien Nioche
>            Assignee: Julien Nioche
>             Fix For: 2.0
>
>         Attachments: NUTCH-882-v1.patch
>
>
> Having a separate GORA table for storing information about hosts (and domains?) would be very useful for : 
> * customising the behaviour of the fetching on a host basis e.g. number of threads, min time between threads etc...
> * storing stats
> * keeping metadata and possibly propagate them to the webpages 
> * keeping a copy of the robots.txt and possibly use that later to filter the webtable
> * store sitemaps files and update the webtable accordingly
> I'll try to come up with a GORA schema for such a host table but any comments are of course already welcome 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.