You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Ilia S. Yatsenko" <sh...@yandex.ru> on 2005/06/30 11:17:14 UTC
Max Urls Per Server how to?
I find this property fetcher.server.maxurls, but how and when it works?
NDFS why
Posted by webmaster <sa...@www.poundwebhosting.com>.
ok I'm running suse 9.3 on 3 computers, a amd athlon 64 3500+ the x86_64
edition of course, and on a pentium 4 3.0ghz suse x86, and a athlon 1900+
x86 version. I am trying to setup ndfs across all the nodes, the athlon
1900+ is the namenode and a datanode, the athlon 64 and pentium 4 are
datanodes, I ssh from my amd64 to the namenode and start the namenode first,
then I start a datanode on it in another ssh shell, then I startup the amd64
and pentium 4 as datanodes, they all connect and I can run the basic
commands mkdir ls rm report du, but when I do a "put" which is also
copyFromLocal moveFromLocal. here is the stuff i did on my a64 testing it:
64bit:/ # cd nutch/
64bit:/nutch # cd nutch-nightly/
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-report
Error: JAVA_HOME is not set.
64bit:/nutch/nutch-nightly # export JAVA_HOME="/usr"
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-report
050707 040223 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040223 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040223 No FS indicated, using default:10.0.0.222:9000
050707 040223 Client connection to 10.0.0.222:9000: starting
Total raw bytes: 137920811008 (128.44 Gb)
Used raw bytes: 22408891270 (20.86 Gb)
% used: 16.24%
Total effective bytes: 0 (0.0 k)
Effective replication multiplier: Infinity
-------------------------------------------------
Datanodes available: 2
Name: 64bit:7000
Total raw bytes: 78955618304 (73.53 Gb)
Used raw bytes: 12090358170 (11.26 Gb)
% used: 15.31%
Last contact with namenode: Thu Jul 07 03:11:15 MDT 2005
Name: linux:7000
Total raw bytes: 58965192704 (54.91 Gb)
Used raw bytes: 10318533100 (9.60 Gb)
% used: 17.49%
Last contact with namenode: Thu Jul 07 03:11:14 MDT 2005
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-report\
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-report
050707 040243 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040243 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040243 No FS indicated, using default:10.0.0.222:9000
050707 040243 Client connection to 10.0.0.222:9000: starting
Total raw bytes: 254930059264 (237.42 Gb)
Used raw bytes: 32569607947 (30.33 Gb)
% used: 12.77%
Total effective bytes: 0 (0.0 k)
Effective replication multiplier: Infinity
-------------------------------------------------
Datanodes available: 3
Name: 64bit:7000
Total raw bytes: 78955618304 (73.53 Gb)
Used raw bytes: 12090358170 (11.26 Gb)
% used: 15.31%
Last contact with namenode: Thu Jul 07 03:11:36 MDT 2005
Name: linux:7000
Total raw bytes: 58965192704 (54.91 Gb)
Used raw bytes: 10318533100 (9.60 Gb)
% used: 17.49%
Last contact with namenode: Thu Jul 07 03:11:35 MDT 2005
Name: mikes:7000
Total raw bytes: 117009248256 (108.97 Gb)
Used raw bytes: 10160716677 (9.46 Gb)
% used: 8.68%
Last contact with namenode: Thu Jul 07 03:11:36 MDT 2005
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-ls /
050707 040406 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040406 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040406 No FS indicated, using default:10.0.0.222:9000
050707 040406 Client connection to 10.0.0.222:9000: starting
Found 0 items
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-mkdir /content
050707 040413 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040413 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040414 No FS indicated, using default:10.0.0.222:9000
050707 040414 Client connection to 10.0.0.222:9000: starting
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
-ls /
050707 040424 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040424 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040424 No FS indicated, using default:10.0.0.222:9000
050707 040424 Client connection to 10.0.0.222:9000: starting
Found 1 items
/content <dir>
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient -put
con
conf/ content.rdf.u8
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient -put
content.rdf.u8 /content
050707 040504 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040504 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040504 No FS indicated, using default:10.0.0.222:9000
050707 040504 Client connection to 10.0.0.222:9000: starting
Exception in thread "main" java.lang.NullPointerException
at java.net.Socket.<init>(Socket.java:357)
at java.net.Socket.<init>(Socket.java:207)
at
org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(NDFSClient.java:627)
at
org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(NDFSClient.java:597)
at org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:85)
at org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:74)
at org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:69)
at
org.apache.nutch.fs.NDFSFileSystem.doFromLocalFile(NDFSFileSystem.java:181)
at
org.apache.nutch.fs.NDFSFileSystem.copyFromLocalFile(NDFSFileSystem.java:156)
at org.apache.nutch.fs.TestClient.copyFromLocal(TestClient.java:46)
at org.apache.nutch.fs.TestClient.main(TestClient.java:234)
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient -put
content.rdf.u8 /content/content.rdf.u8
050707 040521 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040521 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040521 No FS indicated, using default:10.0.0.222:9000
050707 040521 Client connection to 10.0.0.222:9000: starting
Exception in thread "main" java.io.IOException: Could not obtain new output
block for file /content/content.rdf.u8
at
org.apache.nutch.ndfs.NDFSClient$NameNodeCaller.getNewOutputBlock(NDFSClient.java:921)
at
org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(NDFSClient.java:616)
at
org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(NDFSClient.java:597)
at org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:85)
at org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:74)
at org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:69)
at
org.apache.nutch.fs.NDFSFileSystem.doFromLocalFile(NDFSFileSystem.java:181)
at
org.apache.nutch.fs.NDFSFileSystem.copyFromLocalFile(NDFSFileSystem.java:156)
at org.apache.nutch.fs.TestClient.copyFromLocal(TestClient.java:46)
at org.apache.nutch.fs.TestClient.main(TestClient.java:234)
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient -put
content.rdf.u8 content.rdf.u8
050707 040534 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 040534 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 040534 No FS indicated, using default:10.0.0.222:9000
050707 040535 Client connection to 10.0.0.222:9000: starting
050707 040540 Waiting to find new output block node for 5007ms
050707 040541 Waiting to find new output block node for 5463ms
050707 040541 Waiting to find new output block node for 5918ms
050707 040542 Waiting to find new output block node for 6372ms
050707 040542 Waiting to find new output block node for 6827ms
050707 040543 Waiting to find new output block node for 7282ms
050707 040543 Waiting to find new output block node for 7737ms
050707 040544 Waiting to find new output block node for 8192ms
050707 040544 Waiting to find new output block node for 8647ms
050707 040545 Waiting to find new output block node for 9102ms
050707 040545 Waiting to find new output block node for 9557ms
050707 040545 Waiting to find new output block node for 10012ms
050707 040546 Waiting to find new output block node for 10467ms
050707 040546 Waiting to find new output block node for 10922ms
050707 040547 Waiting to find new output block node for 11377ms
050707 040547 Waiting to find new output block node for 11832ms
050707 040548 Waiting to find new output block node for 12287ms
050707 040548 Waiting to find new output block node for 12741ms
050707 040549 Waiting to find new output block node for 13196ms
050707 040549 Waiting to find new output block node for 13651ms
050707 040550 Waiting to find new output block node for 14108ms
050707 040550 Waiting to find new output block node for 14563ms
050707 040550 Waiting to find new output block node for 15018ms
050707 040551 Waiting to find new output block node for 15473ms
64bit:/nutch/nutch-nightly # ./bin/nutch org.apache.nutch.fs.TestClient
Usage: java NDFSClient [-local | -ndfs <namenode:port>] [-ls <path>] [-du
<path>] [-mv <src> <dst>] [-cp <src> <dst>] [-rm <src>] [-put <localsrc>
<dst>] [-copyFromLocal <localsrc> <dst>] [-moveFromLocal <localsrc> <dst>]
[-get <src> <localdst>] [-copyToLocal <src> <localdst>] [-moveToLocal <src>
<localdst>] [-mkdir <path>] [-report]
64bit:/nutch/nutch-nightly #
here is what the namenode said while my a64 was doing those commands
linux:~ # cd /
linux:/ # cd tmp/
linux:/tmp # cd nutch/
linux:/tmp/nutch # cd ndfs/
linux:/tmp/nutch/ndfs # ls
. .. data name
linux:/tmp/nutch/ndfs # cd name
linux:/tmp/nutch/ndfs/name # ls
. .. edits image
linux:/tmp/nutch/ndfs/name # del edits
-bash: del: command not found
linux:/tmp/nutch/ndfs/name # rm edits
linux:/tmp/nutch/ndfs/name # ls
. .. image
linux:/tmp/nutch/ndfs/name # rm image
rm: cannot remove `image': Is a directory
linux:/tmp/nutch/ndfs/name # rd image/
rmdir: `image/': Directory not empty
linux:/tmp/nutch/ndfs/name # cd image/
linux:/tmp/nutch/ndfs/name/image # ls
. .. fsimage
linux:/tmp/nutch/ndfs/name/image # rm fsimage
linux:/tmp/nutch/ndfs/name/image # cd ..
linux:/tmp/nutch/ndfs/name # rd image/
linux:/tmp/nutch/ndfs/name # ls
. ..
linux:/tmp/nutch/ndfs/name # cd /
linux:/ # cd nutch/
linux:/nutch # cd nutch-nightly/
linux:/nutch/nutch-nightly # ./bin/nutch namenode
050707 030736 parsing file:/nutch/nutch-nightly/conf/nutch-default.xml
050707 030736 parsing file:/nutch/nutch-nightly/conf/nutch-site.xml
050707 030736 Server listener on port 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030736 Server handler on 9000: starting
050707 030759 Server connection on port 9000 from 10.0.0.222: starting
050707 030943 Server connection on port 9000 from 10.0.0.69: starting
050707 031004 Server connection on port 9000 from 10.0.0.69: exiting
050707 031015 Lost heartbeat for 64bit:7000
050707 031018 Server connection on port 9000 from 10.0.0.69: starting
050707 031116 Server connection on port 9000 from 10.0.0.69: starting
050707 031117 Server connection on port 9000 from 10.0.0.69: exiting
050707 031124 Server connection on port 9000 from 10.0.0.112: starting
050707 031136 Server connection on port 9000 from 10.0.0.69: starting
050707 031137 Server connection on port 9000 from 10.0.0.69: exiting
050707 031259 Server connection on port 9000 from 10.0.0.69: starting
050707 031259 Server connection on port 9000 from 10.0.0.69: exiting
050707 031307 Server connection on port 9000 from 10.0.0.69: starting
050707 031307 Server connection on port 9000 from 10.0.0.69: exiting
050707 031317 Server connection on port 9000 from 10.0.0.69: starting
050707 031317 Server connection on port 9000 from 10.0.0.69: exiting
050707 031357 Server connection on port 9000 from 10.0.0.69: starting
050707 031357 New lease, holder NDFSClient_-1418582750
050707 031400 Server connection on port 9000 from 10.0.0.69: exiting
050707 031414 Server connection on port 9000 from 10.0.0.69: starting
050707 031414 Removing lease [Lease. Holder: NDFSClient_-1418582750,
heldlocks: 0, pendingcreates: 0], leases remaining: 1
050707 031416 Server connection on port 9000 from 10.0.0.69: exiting
050707 031427 Server connection on port 9000 from 10.0.0.69: starting
050707 031427 New lease, holder NDFSClient_1090559415
050707 031428 Renewed lease [Lease. Holder: NDFSClient_1090559415,
heldlocks: 0, pendingcreates: 1]
050707 031428 File progress failure for /user/root/content.rdf.u8
050707 031428 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031428 File progress failure for /user/root/content.rdf.u8
050707 031428 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031429 File progress failure for /user/root/content.rdf.u8
050707 031429 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031429 File progress failure for /user/root/content.rdf.u8
050707 031429 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031429 File progress failure for /user/root/content.rdf.u8
050707 031429 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031429 File progress failure for /user/root/content.rdf.u8
050707 031429 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031430 File progress failure for /user/root/content.rdf.u8
050707 031430 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031430 File progress failure for /user/root/content.rdf.u8
050707 031430 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031430 File progress failure for /user/root/content.rdf.u8
050707 031430 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031430 File progress failure for /user/root/content.rdf.u8
050707 031430 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031431 File progress failure for /user/root/content.rdf.u8
050707 031431 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031431 File progress failure for /user/root/content.rdf.u8
050707 031431 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031431 File progress failure for /user/root/content.rdf.u8
050707 031431 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031431 File progress failure for /user/root/content.rdf.u8
050707 031431 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031432 File progress failure for /user/root/content.rdf.u8
050707 031432 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031432 File progress failure for /user/root/content.rdf.u8
050707 031432 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031432 File progress failure for /user/root/content.rdf.u8
050707 031432 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031432 File progress failure for /user/root/content.rdf.u8
050707 031432 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031433 File progress failure for /user/root/content.rdf.u8
050707 031433 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031434 File progress failure for /user/root/content.rdf.u8
050707 031434 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031434 File progress failure for /user/root/content.rdf.u8
050707 031434 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031434 File progress failure for /user/root/content.rdf.u8
050707 031434 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031434 File progress failure for /user/root/content.rdf.u8
050707 031434 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031435 File progress failure for /user/root/content.rdf.u8
050707 031435 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031435 File progress failure for /user/root/content.rdf.u8
050707 031435 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031435 File progress failure for /user/root/content.rdf.u8
050707 031435 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031435 File progress failure for /user/root/content.rdf.u8
050707 031435 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031436 Renewed lease [Lease. Holder: NDFSClient_1090559415,
heldlocks: 0, pendingcreates: 1]
050707 031436 File progress failure for /user/root/content.rdf.u8
050707 031436 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031436 File progress failure for /user/root/content.rdf.u8
050707 031436 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031436 File progress failure for /user/root/content.rdf.u8
050707 031436 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031436 File progress failure for /user/root/content.rdf.u8
050707 031436 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031437 File progress failure for /user/root/content.rdf.u8
050707 031437 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031437 File progress failure for /user/root/content.rdf.u8
050707 031437 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031437 File progress failure for /user/root/content.rdf.u8
050707 031437 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031437 File progress failure for /user/root/content.rdf.u8
050707 031437 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031438 File progress failure for /user/root/content.rdf.u8
050707 031438 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031439 File progress failure for /user/root/content.rdf.u8
050707 031439 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031439 File progress failure for /user/root/content.rdf.u8
050707 031439 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031439 File progress failure for /user/root/content.rdf.u8
050707 031439 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031439 File progress failure for /user/root/content.rdf.u8
050707 031439 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031440 File progress failure for /user/root/content.rdf.u8
050707 031440 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031440 File progress failure for /user/root/content.rdf.u8
050707 031440 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031440 File progress failure for /user/root/content.rdf.u8
050707 031440 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031440 File progress failure for /user/root/content.rdf.u8
050707 031440 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031441 File progress failure for /user/root/content.rdf.u8
050707 031441 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031441 File progress failure for /user/root/content.rdf.u8
050707 031441 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031441 File progress failure for /user/root/content.rdf.u8
050707 031441 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031441 File progress failure for /user/root/content.rdf.u8
050707 031441 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031442 File progress failure for /user/root/content.rdf.u8
050707 031442 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031442 File progress failure for /user/root/content.rdf.u8
050707 031442 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031442 File progress failure for /user/root/content.rdf.u8
050707 031442 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031442 File progress failure for /user/root/content.rdf.u8
050707 031442 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031443 File progress failure for /user/root/content.rdf.u8
050707 031443 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031444 Renewed lease [Lease. Holder: NDFSClient_1090559415,
heldlocks: 0, pendingcreates: 1]
050707 031444 File progress failure for /user/root/content.rdf.u8
050707 031444 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031444 File progress failure for /user/root/content.rdf.u8
050707 031444 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031444 File progress failure for /user/root/content.rdf.u8
050707 031444 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031444 File progress failure for /user/root/content.rdf.u8
050707 031444 Problem with block blk_-686619750383293943, with 0 nodes
reporting in.
050707 031444 Server connection on port 9000 from 10.0.0.69: exiting
050707 031500 Removing lease [Lease. Holder: NDFSClient_1090559415,
heldlocks: 0, pendingcreates: 0], leases remaining: 1
any ideas??? also before I ran it this time I trashed all the data and
folders out of my /tmp/nutch/ndfs/data and name directories!!!
Does anyone also know what the limitations of ndfs are, max size and max
datanodes etc...
thanks for your help
-Jay Pound
Re: Nutch Fatal Exception when trying to search
Posted by qu...@webmail.co.za.
Hi Sebastien
I thought I'd give you some feedback on this issue. As you
are aware (or not) the problem was being experienced with
file descriptors (file handles in windows) and as I was
running win2k adv server I couldnt find any reference on
how to up this value and using ulimit in cygwin still didnt
sort out the problem. So for the 1st time I gave up on
windows and installed a nice copy of Fedora FC4 running
under VMWARE - I'm happy to say that on doing this even
though having to upgrade to J2RE 1.5 due to problems with
the socket calls on FC4/JAVA nutch is outperforming its
previous install on a windows machine (even though its
still on the same machine running windows 2k with vmware
and fedora core 4) - With 100 threads I'm totalling about
2-3x as many pages per second when fetching as before as
well as no lock ups or any other problems whatsoever to
date. I wouldn't recommend using nutch on windows with java
if you're planning on having a very large index.
Thanks!
On Tue, 5 Jul 2005 11:37:39 +0200 (CEST)
Sébastien LE CALLONNEC <sl...@yahoo.ie> wrote:
>Hi quovadis,
>
>
>I don't think this is a Nutch-related problem. It is most
>certainly
>due to the underlying OS which limits the number of open
>file
>descriptors.
>
>AFAIK, you can run the command "ulimit -a" to see what
>this limit is
>and "ulimit -n <new_limit>" to modify it.
>
>
>Regards,
>Sebastien.
>
>
>--- quovadis@webmail.co.za a écrit :
>
>> Hi there
>>
>> I wonder if anyone can help. I restarted the webserver
>for
>> nutch this morning and opened the front page at
>> http://127.0.0.1:8080 which displayed the search bar
>etc.
>> When searching however I get a HTTP Status 500 Exception
>> with the error:
>>
>> org.apache.jasper.JasperException: Too many open files
>>
>> Any ideas?
>>
>> Thanks
>>
>_____________________________________________________________________
>> For super low premiums, click here
>http://www.dialdirect.co.za/quote
>>
>
>
>
>
>
>
>
>___________________________________________________________________________
>
>Appel audio GRATUIT partout dans le monde avec le nouveau
>Yahoo! Messenger
>Téléchargez cette version sur
>http://fr.messenger.yahoo.com
_____________________________________________________________________
For super low premiums, click here http://www.dialdirect.co.za/quote
RE: Nutch Fatal Exception when trying to search
Posted by Sébastien LE CALLONNEC <sl...@yahoo.ie>.
Hi quovadis,
I don't think this is a Nutch-related problem. It is most certainly
due to the underlying OS which limits the number of open file
descriptors.
AFAIK, you can run the command "ulimit -a" to see what this limit is
and "ulimit -n <new_limit>" to modify it.
Regards,
Sebastien.
--- quovadis@webmail.co.za a écrit :
> Hi there
>
> I wonder if anyone can help. I restarted the webserver for
> nutch this morning and opened the front page at
> http://127.0.0.1:8080 which displayed the search bar etc.
> When searching however I get a HTTP Status 500 Exception
> with the error:
>
> org.apache.jasper.JasperException: Too many open files
>
> Any ideas?
>
> Thanks
> _____________________________________________________________________
> For super low premiums, click here http://www.dialdirect.co.za/quote
>
___________________________________________________________________________
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger
Téléchargez cette version sur http://fr.messenger.yahoo.com
Nutch Fatal Exception when trying to search
Posted by qu...@webmail.co.za.
Hi there
I wonder if anyone can help. I restarted the webserver for
nutch this morning and opened the front page at
http://127.0.0.1:8080 which displayed the search bar etc.
When searching however I get a HTTP Status 500 Exception
with the error:
org.apache.jasper.JasperException: Too many open files
Any ideas?
Thanks
_____________________________________________________________________
For super low premiums, click here http://www.dialdirect.co.za/quote