[jira] Updated: (NUTCH-863) Benchmark and a testbed proxy server

["Andrzej Bialecki (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  updated NUTCH-863:
------------------------------------

    Attachment: proxy.patch

Patch with the tools. If there are no objections I'd like to commit this soon, and track improvements in other JIRA issues.

> Benchmark and a testbed proxy server
> ------------------------------------
>
>                 Key: NUTCH-863
>                 URL: https://issues.apache.org/jira/browse/NUTCH-863
>             Project: Nutch
>          Issue Type: New Feature
>    Affects Versions: 2.0
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>         Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.