You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Ayon Sinha <ay...@yahoo.com> on 2010/08/09 23:12:15 UTC
HDFS client java access
I have a HDFS java client on the same network domain as the cluster which needs
to read the dfs file with permissions rw-r--r--
I get
java.io.IOException
at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:169)
at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
Caused by: javax.security.auth.login.LoginException: Login failed: Expect one
token as the result of whoami: nt authority\\system
at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:167)\r\n\t... 20
more\r\n&msg=2010-08-06 16:25:38,069 ERROR -
com.ebay.integ.dal.cache2.CacheLoaderFailurenull\r\n
Can I use
job.set("dfs.permissions", "false");
My question is: Is it a matter of setting the right property values in the conf
object? like user.name or dfs.web.ugi or does the clinet need to actually have
a Unix login or passwordless ssh setup? -Ayon
Re: HDFS java client connect retry count
Posted by Ayon Sinha <ay...@yahoo.com>.
Looks like Client.java has the following code:
} catch (SocketTimeoutException toe) {
/* The max number of retries is 45,
* which amounts to 20s*45 = 15 minutes retries.
*/
handleConnectionFailure(timeoutFailures++, 45, toe);
} catch (IOException ie) {
handleConnectionFailure(ioFailures++, maxRetries, ie);
}
So it doesnt use the maxRetries variable when it gets a SocketTimeoutException.
Our client is 0.18.3 (I know its ancient) but is this fixed in a later release?
-Ayon
________________________________
From: Ayon Sinha <ay...@yahoo.com>
To: hdfs-user@hadoop.apache.org
Sent: Tue, August 17, 2010 3:49:56 PM
Subject: HDFS java client connect retry count
Hi,
I have a java HDFS client which connects to a production cluster and gets data.
On our staging environment we see that since it cannot connect to the namenode
(expected) it keeps retrying for 45 times. I am looking for a way to set the
retry count to much much lower.
This is what we see in server logs:
2010-08-17 15:15:06,973 INFO [Client] Retrying connect to server:
xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 0 time(s).
2010-08-17 15:15:27,979 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 1 time(s).
2010-08-17 15:15:48,984 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 2 time(s).
2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 3 time(s).
..
..
..
2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 44 time(s).
I have tried the client config
job.set("ipc.client.connect.max.retries", "5");
which doesn't seem to work.
Where is this 45 reties coming from?
Thanks in advance. -Ayon
HDFS java client connect retry count
Posted by Ayon Sinha <ay...@yahoo.com>.
Hi,
I have a java HDFS client which connects to a production cluster and gets data.
On our staging environment we see that since it cannot connect to the namenode
(expected) it keeps retrying for 45 times. I am looking for a way to set the
retry count to much much lower.
This is what we see in server logs:
2010-08-17 15:15:06,973 INFO [Client] Retrying connect to server:
xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 0 time(s).
2010-08-17 15:15:27,979 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 1 time(s).
2010-08-17 15:15:48,984 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 2 time(s).
2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 3 time(s).
..
..
..
2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 44 time(s).
I have tried the client config
job.set("ipc.client.connect.max.retries", "5");
which doesn't seem to work.
Where is this 45 reties coming from?
Thanks in advance. -Ayon
Re: HDFS client java access
Posted by Josh Patterson <jo...@cloudera.com>.
Ayon,
When you have a java client that you want to use to simply access hdfs
you need to make sure that you let your hdfs client know where to look
to connect to hdfs and the namenode; You'll want to create a new
Configuration object (which will automatically try and load a copy of
your core-size.xml conf file if its around).
conf = new Configuration();
If you dont want to move a copy of it locally, you can manually set
the parameter for the namenode:
conf.set("fs.default.name", "hdfs://localhost:PORT");
and then get a reference to hdfs.
hdfs_ref = FileSystem.get(conf);
from there, you can do basic FS operations like get a dir listing,
create a file, delete a file, etc.
Josh Patterson
Cloudera
On Mon, Aug 9, 2010 at 5:12 PM, Ayon Sinha <ay...@yahoo.com> wrote:
> I have a HDFS java client on the same network domain as the cluster which
> needs to read the dfs file with permissions rw-r--r--
> I get
> java.io.IOException
> at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:169)
> at
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
>
> Caused by: javax.security.auth.login.LoginException: Login failed: Expect
> one token as the result of whoami: nt authority\\system
> at
> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
> at
> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
> at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:167)\r\n\t... 20
> more\r\n&msg=2010-08-06 16:25:38,069 ERROR -
> com.ebay.integ.dal.cache2.CacheLoaderFailurenull\r\n
>
> Can I use
> job.set("dfs.permissions", "false");
> My question is: Is it a matter of setting the right property values in the
> conf object? like user.name or dfs.web.ugi or does the clinet need to
> actually have a Unix login or passwordless ssh setup?
>
> -Ayon
>
>
>