You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mahesh Shinde <Ma...@persistent.co.in> on 2011/07/22 08:20:29 UTC
FW: Question about property "fs.default.name"
Hi,
I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set fs.default.name=file:/// .
If we define not use HDFS as our file system then how hadoop deals with File system of local?..
Please reply.
Mahesh Shinde | Systems Engineer
mahesh_shinde@persistent.co.in<ma...@persistent.co.in>| Cell: +918308321501 | Tel: +91-20-30235194
Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: FW: Question about property "fs.default.name"
Posted by Harsh J <ha...@cloudera.com>.
Mahesh,
The NameNode is a component of HDFS. It does not apply to other FSes
Hadoop can use. You can browse file:/// (are you planning to use NFS,
etc.?) with a regular file browser.
Besides this, am not sure I even got your question. Why don't you try
out how Hadoop deals with a local fs by running a few MR jobs with
that config?
On Fri, Jul 22, 2011 at 11:50 AM, Mahesh Shinde
<Ma...@persistent.co.in> wrote:
>
>
> Hi,
>
> I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set fs.default.name=file:/// .
> If we define not use HDFS as our file system then how hadoop deals with File system of local?..
> Please reply.
>
> Mahesh Shinde | Systems Engineer
> mahesh_shinde@persistent.co.in<ma...@persistent.co.in>| Cell: +918308321501 | Tel: +91-20-30235194
> Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
>
--
Harsh J
Re: FW: Question about property "fs.default.name"
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
Hi Mahesh,
When starting the NN, it will throw exception with your provided configuration. please check the code snippet below where exactly validation will happen.
in NameNode:
public static InetSocketAddress getAddress(URI filesystemURI) {
String authority = filesystemURI.getAuthority();
if (authority == null) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): %s has no authority.",
FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString()));
}
if (!FSConstants.HDFS_URI_SCHEME.equalsIgnoreCase(
filesystemURI.getScheme())) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): %s is not of scheme '%s'.",
FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString(),
FSConstants.HDFS_URI_SCHEME));
}
Since NN is specific to HDFS, it will expect scheme as hdfs. Otherwise it will throw the exception.
If you develop your own file system, then clients will have intelligence to connect to that file system, based on your provided configuration for fs.default.name.
Coming to clients side, if you pass file:/// as fs uri, it will not try to connect to DFS because your passed fs is related to local. So, it will create LocalFileSystem instead of DistributedFilesystem.
Regards,
Uma
Regards,
Uma
******************************************************************************************
This email and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained here in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it!
*****************************************************************************************
----- Original Message -----
From: Mahesh Shinde <Ma...@persistent.co.in>
Date: Saturday, July 23, 2011 4:17 pm
Subject: FW: Question about property "fs.default.name"
To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
>
>
> Hi,
>
> I have basic question on property "fs.default.name" is that I am
> not able to open NameNode URL when I set fs.default.name=file:/// .
> If we define not use HDFS as our file system then how hadoop deals
> with File system of local?..
> Please reply.
>
> Mahesh Shinde | Systems Engineer
> mahesh_shinde@persistent.co.in<ma...@persistent.co.in>| Cell: +918308321501 | Tel: +91-20-30235194
> Persistent Systems Ltd. | 20 Glorious Years |
> www.persistentsys.com<http://www.persistentsys.com/>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information
> which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
>