Re[2]: NDFS question

[Egor Chernodarov <eg...@zarinsk.dem.ru>]

Hello, Doug!

I try with "mapred" branch, but anyway get errors like this:
$./nutch ndfs -put ./test.txt /test.txt
=====================
050831 055936 Client connection to 192.168.0.170:9000: starting
050831 060245 Waiting to find target node
=====================
On namenode I see :
050831 055936 Server connection on port 9000 from 192.168.0.170: starting

At the same time $./nutch ndfs -report    works fine:
=====================
Total effective bytes: 0 (0.0 k)
Effective replication multiplier: Infinity
-------------------------------------------------
Datanodes available: 1

Name: server.domain.com:7000
Total raw bytes: 75487932416 (70.30 Gb)
Used raw bytes: 7289752863 (6.78 Gb)
% used: 9.65%
Last contact with namenode: Wed Aug 31 06:08:32 CDT 2005
=====================

What also I can try? I really interested in NDFS...

Thanks for any help.


Tuesday, August 30, 2005, 10:51:10 PM, you wrote:

Doug Cutting> It sounds like you're using a nightly
Doug Cutting> build of trunk.  The NDFS code in 
Doug Cutting> trunk is old.  The NDFS code is currently
Doug Cutting> maintained in a branch named 
Doug Cutting> "mapred".  Please check out the mapred branch and retry.

Doug Cutting> svn co
Doug Cutting> https://svn.apache.org/repos/asf/lucene/nutch/branches/mapred/

Doug Cutting> Doug

Doug Cutting> Egor Chernodarov wrote:
>> Hello!
>> 
>> I want to test NDFS on my nutch installation, but I have some problem.
>> I have started from wiki, where is quick demo for NDFS:
>> http://wiki.apache.org/nutch/NutchDistributedFileSystem
>> 
>> On "$ nutch ndfs -put local_file /test/testfile"(or ./nutch admin db
>> -create and etc.) I always have exception "Could not obtain new output block":
>> =======================================================================
>> 050830 061956 Waiting to find target node
>> Exception in thread "main" java.io.IOException: Could not obtain new
>> output block for file /test/testfile
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NameNodeCaller.getNewOutputBlock(NDFSClient.java:921)
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(NDFSClient.java:616)
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(NDFSClient.java:597)
>>         at
>> org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:85)
>>         at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:76)
>>         at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:71)
>>         at
>> org.apache.nutch.io.SequenceFile$Writer.<init>(SequenceFile.java:80)
>>         at
>> org.apache.nutch.io.MapFile$Writer.<init>(MapFile.java:94)
>>         at
>> org.apache.nutch.db.WebDBWriter.<init>(WebDBWriter.java:1507)
>>         at
>> org.apache.nutch.db.WebDBWriter.createWebDB(WebDBWriter.java:1438)
>>         at
>> org.apache.nutch.tools.WebDBAdminTool.main(WebDBAdminTool.java:172)
>> =======================================================================
>> 
>> On namenode I see something like this:
>> =======================================================================
>> 050830 061445 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061447 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061448 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061451 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061454 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061455 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061457 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061500 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> =======================================================================
>> 
>> But if I run datanode and namenode on the same server - all's ok!
>> 
>> On "$ nutch ndfs -report" I see list of my datanodes, but these
>> datanodes defined by external hostname. I think that namenode tries
>> to connect to datanodes by this NOT LOCAL hostnames. It is impossible
>> because firewall not allow incoming connection from external network
>> interfaces to this port(7000).
>> 
>> It's right? The error can be generated in this case?
>> 
>> So, can you tell me, please, what I can make to define namenode for use
>> local interfaces for data transfer?  I can't reconfigure firewall..
>> 
>> Red hat ES3.0, nutch-2005-08-25 (>nutch-0.7).
>> $ java -version
>> java version "1.4.2-01"
>> Java(TM) 2 Runtime Environment, Standard Edition (build Blackdown-1.4.2-01)
>> Java HotSpot(TM) 64-Bit Server VM (build Blackdown-1.4.2-01, mixed mode)
>> 
>> 
>> Thanks for your time!
>> 
>> 



-- 
Best regards,               
 Chernodarov Egor