[jira] Updated: (NUTCH-116) TestNDFS a JUnit test specifically for NDFS

["Paul Baclace (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/NUTCH-116?page=all ]

Paul Baclace updated NUTCH-116:
-------------------------------

    Attachment: required_by_TestNDFS.patch
                TestNDFS.java

Patch comments:  

src/java/org/apache/nutch/ipc/Server.java
  improved logging details, use the safer notifyAll() instead of notify(), added comments.

src/java/org/apache/nutch/ndfs/DataNode.java
  improved logging details, added comments, improved error message, 
  factored reuseable code into makeInstanceForDir(), added toString(),
  added properties ndfs.blockreport.intervalMsec and ndfs.datanode.startupMsec
  to allow the override of BLOCKREPORT_INTERVAL and DATANODE_STARTUP_PERIOD,
  respectively.

src/java/org/apache/nutch/ndfs/FSDataset.java
  added toString() methods.

src/java/org/apache/nutch/ndfs/FSNamesystem.java
  Changed chooseTarget() to behave as commented rather than as implemented (it
  says it fobids picking a target on the same host, but it was using
  host:port as the basis of comparison, TestNDFS needs host:port comparison,
  and normal operation just uses 'host' to better distribute replicants;
  simplified redundant conditional,
  added property test.ndfs.same.host.targets.allowed which allows target datanode
  selection to use same host (but not same port), added logging for previously
  silent errors, emit more info for some logging, change LOG.info() to 
  LOG.warning(), added javadoc comments.

src/java/org/apache/nutch/ndfs/NameNode.java
  Added a way to stop the daemon for JUnit testing, added javadoc comments.


> TestNDFS a JUnit test specifically for NDFS
> -------------------------------------------
>
>          Key: NUTCH-116
>          URL: http://issues.apache.org/jira/browse/NUTCH-116
>      Project: Nutch
>         Type: Test
>   Components: fetcher, indexer, searcher
>     Versions: 0.8-dev
>     Reporter: Paul Baclace
>  Attachments: TestNDFS.java, required_by_TestNDFS.patch
>
> TestNDFS is a JUnit test for NDFS using "pseudo multiprocessing" (or more strictly, pseudo distributed) meaning all daemons run in one process and sockets are used to communicate between daemons.  
> The test permutes various block sizes, number of files, file sizes, and number of datanodes.  After creating 1 or more files and filling them with random data, one datanode is shutdown, and then the files are verfified. Next, all the random test files are deleted and we test for leakage (non-deletion) by directly checking the real directories corresponding to the datanodes still running.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira