You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Hamza Kaya <ha...@gmail.com> on 2005/12/06 14:48:08 UTC
NDFS problem on mapred branch
Hi,
We hava a mapred setup on 4 machines. (1 namenode and 3 datanodes)
I can access the file system from these machines without any problem.
However, when I tried to write a file to the NDFS on a machine other than
these 4 machines
I got the following error:
~/nutch-mapred/bin$ ./nutch ndfs -put nutch nutch
051206 152157 parsing file:/home/agmlab/nutch-mapred/conf/nutch-default.xml
051206 152158 parsing file:/home/agmlab/nutch-mapred/conf/nutch-site.xml
051206 152158 No FS indicated, using default:192.168.15.118:9001
051206 152158 Client connection to 192.168.15.118:9001: starting
Exception in thread "main" java.io.IOException: Cannot create file
/user/agmlab/nutch on client NDFSClient_1904460956
at org.apache.nutch.ipc.Client.call(Client.java:294)
at org.apache.nutch.ipc.RPC$Invoker.invoke(RPC.java:128)
at $Proxy0.create(Unknown Source)
at
org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(
NDFSClient.java:537)
at org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(
NDFSClient.java:512)
at org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:74)
at org.apache.nutch.fs.NDFSFileSystem.createRaw(NDFSFileSystem.java
:67)
at org.apache.nutch.fs.NFSDataOutputStream$Summer.<init>(
NFSDataOutputStream.java:41)
at org.apache.nutch.fs.NFSDataOutputStream.<init>(
NFSDataOutputStream.java:129)
at org.apache.nutch.fs.NutchFileSystem.create(NutchFileSystem.java
:175)
at org.apache.nutch.fs.NutchFileSystem.create(NutchFileSystem.java
:162)
at org.apache.nutch.fs.NDFSFileSystem.doFromLocalFile(
NDFSFileSystem.java:174)
at org.apache.nutch.fs.NDFSFileSystem.copyFromLocalFile(
NDFSFileSystem.java:149)
at org.apache.nutch.fs.NDFSShell.copyFromLocal(NDFSShell.java:46)
at org.apache.nutch.fs.NDFSShell.main(NDFSShell.java:234)
>From the same machine I was able to list the files and create directories.
What may be the problem?
Thanks.
--
Hamza Kaya
Re: NDFS problem on mapred branch
Posted by Stefan Groschupf <sg...@media-style.com>.
I had the same problem, you will find it in the mail archive.
In my case it one box was unable to connect to the other.
There can be 2 cases, first may a fire wall block the ports or a
common case is that the network dns name and the name the box uses to
identify itself against other boxes is different.
Check that you can telnet on the port or / and ping all boxes from
all other boxes by using the names that are setuped in the host.conf
Let me please know if this was also the problem in your case, since
other people had this problem as well and we may should add this to
the faq.
Stefan
Am 06.12.2005 um 14:48 schrieb Hamza Kaya:
> Hi,
> We hava a mapred setup on 4 machines. (1 namenode and 3 datanodes)
> I can access the file system from these machines without any problem.
> However, when I tried to write a file to the NDFS on a machine
> other than
> these 4 machines
> I got the following error:
>
> ~/nutch-mapred/bin$ ./nutch ndfs -put nutch nutch
> 051206 152157 parsing file:/home/agmlab/nutch-mapred/conf/nutch-
> default.xml
> 051206 152158 parsing file:/home/agmlab/nutch-mapred/conf/nutch-
> site.xml
> 051206 152158 No FS indicated, using default:192.168.15.118:9001
> 051206 152158 Client connection to 192.168.15.118:9001: starting
> Exception in thread "main" java.io.IOException: Cannot create file
> /user/agmlab/nutch on client NDFSClient_1904460956
> at org.apache.nutch.ipc.Client.call(Client.java:294)
> at org.apache.nutch.ipc.RPC$Invoker.invoke(RPC.java:128)
> at $Proxy0.create(Unknown Source)
> at
> org.apache.nutch.ndfs.NDFSClient
> $NDFSOutputStream.nextBlockOutputStream(
> NDFSClient.java:537)
> at org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(
> NDFSClient.java:512)
> at org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:74)
> at org.apache.nutch.fs.NDFSFileSystem.createRaw
> (NDFSFileSystem.java
> :67)
> at org.apache.nutch.fs.NFSDataOutputStream$Summer.<init>(
> NFSDataOutputStream.java:41)
> at org.apache.nutch.fs.NFSDataOutputStream.<init>(
> NFSDataOutputStream.java:129)
> at org.apache.nutch.fs.NutchFileSystem.create
> (NutchFileSystem.java
> :175)
> at org.apache.nutch.fs.NutchFileSystem.create
> (NutchFileSystem.java
> :162)
> at org.apache.nutch.fs.NDFSFileSystem.doFromLocalFile(
> NDFSFileSystem.java:174)
> at org.apache.nutch.fs.NDFSFileSystem.copyFromLocalFile(
> NDFSFileSystem.java:149)
> at org.apache.nutch.fs.NDFSShell.copyFromLocal
> (NDFSShell.java:46)
> at org.apache.nutch.fs.NDFSShell.main(NDFSShell.java:234)
>
> From the same machine I was able to list the files and create
> directories.
> What may be the problem?
>
> Thanks.
>
> --
> Hamza Kaya
---------------------------------------------------------------
company: http://www.media-style.com
forum: http://www.text-mining.org
blog: http://www.find23.net
Re: NDFS problem on mapred branch
Posted by Hamza Kaya <ha...@gmail.com>.
Hi,
The problem was solved after I added the names to the host.conf!
Thanks,
--
Hamza
I had the same problem, you will find it in the mail archive.
> In my case it one box was unable to connect to the other.
> There can be 2 cases, first may a fire wall block the ports or a
> common case is that the network dns name and the name the box uses to
>
> identify itself against other boxes is different.
> Check that you can telnet on the port or / and ping all boxes from
> all other boxes by using the names that are setuped in the host.conf
> Let me please know if this was also the problem in your case, since
>
> other people had this problem as well and we may should add this to
> the faq.
>
>
> Stefan
>
>
>
> On 12/6/05, Hamza Kaya <ha...@gmail.com> wrote:
> >
> > Hi,
> > We hava a mapred setup on 4 machines. (1 namenode and 3 datanodes)
> > I can access the file system from these machines without any problem.
> > However, when I tried to write a file to the NDFS on a machine other
> > than these 4 machines
> > I got the following error:
> >
> > ~/nutch-mapred/bin$ ./nutch ndfs -put nutch nutch
> > 051206 152157 parsing file:/home/agmlab/nutch-mapred/conf/nutch-
> > default.xml
> > 051206 152158 parsing file:/home/agmlab/nutch-mapred/conf/nutch-site.xml
> > 051206 152158 No FS indicated, using default:192.168.15.118:9001
> > 051206 152158 Client connection to 192.168.15.118:9001: starting
> > Exception in thread "main" java.io.IOException: Cannot create file
> > /user/agmlab/nutch on client NDFSClient_1904460956
> > at org.apache.nutch.ipc.Client.call(Client.java:294)
> > at org.apache.nutch.ipc.RPC$Invoker.invoke(RPC.java:128)
> > at $Proxy0.create(Unknown Source)
> > at
> > org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(
> > NDFSClient.java:537)
> > at org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(
> > NDFSClient.java:512)
> > at org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:74)
> > at org.apache.nutch.fs.NDFSFileSystem.createRaw(
> > NDFSFileSystem.java:67)
> > at org.apache.nutch.fs.NFSDataOutputStream$Summer.<init>(
> > NFSDataOutputStream.java:41)
> > at org.apache.nutch.fs.NFSDataOutputStream.<init>(
> > NFSDataOutputStream.java:129)
> > at org.apache.nutch.fs.NutchFileSystem.create(
> > NutchFileSystem.java:175)
> > at org.apache.nutch.fs.NutchFileSystem.create(
> > NutchFileSystem.java:162)
> > at org.apache.nutch.fs.NDFSFileSystem.doFromLocalFile(
> > NDFSFileSystem.java:174)
> > at org.apache.nutch.fs.NDFSFileSystem.copyFromLocalFile(
> > NDFSFileSystem.java:149)
> > at org.apache.nutch.fs.NDFSShell.copyFromLocal(NDFSShell.java
> > :46)
> > at org.apache.nutch.fs.NDFSShell.main(NDFSShell.java:234)
> >
> > From the same machine I was able to list the files and create
> > directories. What may be the problem?
> >
> > Thanks.
> >
> > --
> > Hamza Kaya
> >
>
>
Re: NDFS problem on mapred branch
Posted by Andrzej Bialecki <ab...@getopt.org>.
Hamza Kaya wrote:
>Hi,
>We hava a mapred setup on 4 machines. (1 namenode and 3 datanodes)
>I can access the file system from these machines without any problem.
>However, when I tried to write a file to the NDFS on a machine other than
>these 4 machines
>I got the following error:
>
>~/nutch-mapred/bin$ ./nutch ndfs -put nutch nutch
>
>
Could you try the same, but using absolute paths? NDFS client has no
notion of relative or current directory, so the file names must always
be absolute, i.e. starting with the leading / .
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com