You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2010/07/29 15:09:16 UTC
[jira] Created: (NUTCH-863) Benchmark and a testbed proxy server
Benchmark and a testbed proxy server
------------------------------------
Key: NUTCH-863
URL: https://issues.apache.org/jira/browse/NUTCH-863
Project: Nutch
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Andrzej Bialecki
Assignee: Andrzej Bialecki
This issue adds two components:
* a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
* a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
* 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated NUTCH-863:
------------------------------------
Attachment: proxy.patch
Patch with the tools. If there are no objections I'd like to commit this soon, and track improvements in other JIRA issues.
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (NUTCH-863) Benchmark and a testbed
proxy server
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894944#action_12894944 ]
Chris A. Mattmann edited comment on NUTCH-863 at 8/3/10 1:10 PM:
-----------------------------------------------------------------
Hey Andrzej,
Oddly enough, this test fails for me on a fresh checkout of Nutch (r981962 as of today):
{noformat}
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
[junit] Test org.apache.nutch.tools.proxy.TestbedProxy FAILED
{noformat}
{noformat}
[chipotle:~/src/nutch] mattmann% more build/test/TEST-org.apache.nutch.tools.proxy.TestbedProxy.txt
Testsuite: org.apache.nutch.tools.proxy.TestbedProxy
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
Testcase: initializationError took 0.006 sec
Caused an ERROR
No runnable methods
java.lang.Exception: No runnable methods
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[chipotle:~/src/nutch] mattmann% java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
[chipotle:~/src/nutch] mattmann%
{noformat}
Any clue as to why?
Cheers,
Chris
was (Author: chrismattmann):
Hey Andrzej,
Oddly enough, this test fails for me on a fresh checkout of Nutch (r as of today):
{noformat}
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
[junit] Test org.apache.nutch.tools.proxy.TestbedProxy FAILED
{noformat}
{noformat}
[chipotle:~/src/nutch] mattmann% more build/test/TEST-org.apache.nutch.tools.proxy.TestbedProxy.txt
Testsuite: org.apache.nutch.tools.proxy.TestbedProxy
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
Testcase: initializationError took 0.006 sec
Caused an ERROR
No runnable methods
java.lang.Exception: No runnable methods
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[chipotle:~/src/nutch] mattmann% java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
[chipotle:~/src/nutch] mattmann%
{noformat}
Any clue as to why?
Cheers,
Chris
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894944#action_12894944 ]
Chris A. Mattmann commented on NUTCH-863:
-----------------------------------------
Hey Andrzej,
Oddly enough, this test fails for me on a fresh checkout of Nutch (r as of today):
{noformat}
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
[junit] Test org.apache.nutch.tools.proxy.TestbedProxy FAILED
{noformat}
{noformat}
[chipotle:~/src/nutch] mattmann% more build/test/TEST-org.apache.nutch.tools.proxy.TestbedProxy.txt
Testsuite: org.apache.nutch.tools.proxy.TestbedProxy
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.063 sec
Testcase: initializationError took 0.006 sec
Caused an ERROR
No runnable methods
java.lang.Exception: No runnable methods
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[chipotle:~/src/nutch] mattmann% java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
[chipotle:~/src/nutch] mattmann%
{noformat}
Any clue as to why?
Cheers,
Chris
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895021#action_12895021 ]
Chris A. Mattmann commented on NUTCH-863:
-----------------------------------------
haha not your fault! I blame Junit! :)
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki resolved NUTCH-863.
-------------------------------------
Fix Version/s: 2.0
Resolution: Fixed
Committed in rev. 980932.
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895015#action_12895015 ]
Andrzej Bialecki commented on NUTCH-863:
-----------------------------------------
Yes, the combined naivety of JUnit and mine contributed to this error ... The class is NOT a JUnit test, its name doesn't even start with a properly CamelCase-d "Test" keyword... but apparently JUnit only does a substring comparison. Pity, I will have to rename this class.
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895259#action_12895259 ]
Chris A. Mattmann commented on NUTCH-863:
-----------------------------------------
Hey Guys:
Just wanted to let you know that I kind of jumped the gun on committing my solution to this and in the future I'll wait a bit longer before trying to resolve it since the issue wasn't really assigned to me anyways! :) Great work on the tool, Andrzej, I really like it!
Cheers,
Chris
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-863) Benchmark and a testbed proxy server
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895125#action_12895125 ]
Chris A. Mattmann commented on NUTCH-863:
-----------------------------------------
okey dok, I cleaned this up in r982102, r982103, r982104, and r982107 by moving the Benchmark and its classes into src/java instead of src/test.
> Benchmark and a testbed proxy server
> ------------------------------------
>
> Key: NUTCH-863
> URL: https://issues.apache.org/jira/browse/NUTCH-863
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: proxy.patch
>
>
> This issue adds two components:
> * a testbed proxy server that can serve various content: pre-fetched Nutch segments, forward requests to original URLs, or create a lot of unique but predictable fake content (with outlinks) on the fly.
> * a simple Benchmark class to measure the time taken to complete several crawl cycles using fake content.
> * 'ant proxy' and 'ant benchmark' targets to execute a benchmark run.
> Together these tools should provide a more or less objective method to measure the end-to-end crawl performance. This initial version can be further instrumented to collect statistics about various stages of data processing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.