You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by deepya <m_...@yahoo.co.in> on 2009/03/29 07:29:24 UTC
hadoop-a small doubt
Hi,
I am SreeDeepya doing MTech in IIIT.I am working on a project named cost
effective and scalable storage server.I configured a small hadoop cluster
with only two nodes one namenode and one datanode.I am new to hadoop.
I have a small doubt.
Can a system not in the hadoop cluster access the namenode or the
datanode????If yes,then can you please tell me the necessary configurations
that has to be done.
Thanks in advance.
SreeDeepya
--
View this message in context: http://www.nabble.com/hadoop-a-small-doubt-tp22764615p22764615.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: hadoop-a small doubt
Posted by Parul Kudtarkar <Pa...@hms.harvard.edu>.
What is the exact purpose that you want a system not in hadoop cluster to
access the namenode or datanode? If it is simply to write data to HDFS from
local system and then to copy back data from HDFS to local system simply use
hadoop file system's shell commands.
Hope this helps!
deepya wrote:
>
> Hi,
> I am SreeDeepya doing MTech in IIIT.I am working on a project named
> cost effective and scalable storage server.I configured a small hadoop
> cluster with only two nodes one namenode and one datanode.I am new to
> hadoop.
> I have a small doubt.
>
> Can a system not in the hadoop cluster access the namenode or the
> datanode????If yes,then can you please tell me the necessary
> configurations that has to be done.
>
> Thanks in advance.
>
> SreeDeepya
>
--
View this message in context: http://www.nabble.com/hadoop-a-small-doubt-tp22764615p23149249.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: hadoop-a small doubt
Posted by nitesh bhatia <ni...@gmail.com>.
Hi
You can ssh to them from any PC in your domain.
--nitesh
On Sun, Mar 29, 2009 at 10:59 AM, deepya <m_...@yahoo.co.in> wrote:
>
> Hi,
> I am SreeDeepya doing MTech in IIIT.I am working on a project named cost
> effective and scalable storage server.I configured a small hadoop cluster
> with only two nodes one namenode and one datanode.I am new to hadoop.
> I have a small doubt.
>
> Can a system not in the hadoop cluster access the namenode or the
> datanode????If yes,then can you please tell me the necessary configurations
> that has to be done.
>
> Thanks in advance.
>
> SreeDeepya
> --
> View this message in context:
> http://www.nabble.com/hadoop-a-small-doubt-tp22764615p22764615.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: hadoop-a small doubt
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Mar 30, 2009, at 3:53 AM, deepya wrote:
>
> Do you mean to say the node from which we want to access hdfs should
> also
> have hadoop installed on it??If that is the case then doesnt that
> node also
> become apart of the cluster??
Yes. You need the Hadoop client installed to access HDFS. You also
need direct access from the client node to the data node - or you need
to use some other method to bridge the internal cluster and the
external client. We use a GridFTP server to set between HDFS and
remote WAN clients and achieve multiple Gbps.
Brian
>
>
> Can you please be a bit more clear
>
> Sagar Naik-3 wrote:
>>
>> Yes u can
>> Java Client :
>> Copy the conf dir (same as one on namenode/datanode) and hadoop jars
>> shud be in the classpath of client
>> Non Java Client :
>> http://wiki.apache.org/hadoop/MountableHDFS
>>
>>
>>
>> -Sagar
>>
>> -Sagar
>>
>> deepya wrote:
>>> Hi,
>>> I am SreeDeepya doing MTech in IIIT.I am working on a project
>>> named
>>> cost
>>> effective and scalable storage server.I configured a small hadoop
>>> cluster
>>> with only two nodes one namenode and one datanode.I am new to
>>> hadoop.
>>> I have a small doubt.
>>>
>>> Can a system not in the hadoop cluster access the namenode or the
>>> datanode????If yes,then can you please tell me the necessary
>>> configurations
>>> that has to be done.
>>>
>>> Thanks in advance.
>>>
>>> SreeDeepya
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Re%3A-hadoop-a-small-doubt-tp22765160p22778753.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: hadoop-a small doubt
Posted by deepya <m_...@yahoo.co.in>.
Do you mean to say the node from which we want to access hdfs should also
have hadoop installed on it??If that is the case then doesnt that node also
become apart of the cluster??
Can you please be a bit more clear
Sagar Naik-3 wrote:
>
> Yes u can
> Java Client :
> Copy the conf dir (same as one on namenode/datanode) and hadoop jars
> shud be in the classpath of client
> Non Java Client :
> http://wiki.apache.org/hadoop/MountableHDFS
>
>
>
> -Sagar
>
> -Sagar
>
> deepya wrote:
>> Hi,
>> I am SreeDeepya doing MTech in IIIT.I am working on a project named
>> cost
>> effective and scalable storage server.I configured a small hadoop cluster
>> with only two nodes one namenode and one datanode.I am new to hadoop.
>> I have a small doubt.
>>
>> Can a system not in the hadoop cluster access the namenode or the
>> datanode????If yes,then can you please tell me the necessary
>> configurations
>> that has to be done.
>>
>> Thanks in advance.
>>
>> SreeDeepya
>>
>
>
--
View this message in context: http://www.nabble.com/Re%3A-hadoop-a-small-doubt-tp22765160p22778753.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: hadoop-a small doubt
Posted by Aaron Kimball <aa...@cloudera.com>.
"Becoming a part of the cluster" implies that you're running the daemons on
the node. You need the Hadoop JARs on the client machine so that you can use
FileSystem.open(), etc. And the conf/hadoop-site.xml file should indicate
the NameNode's address in its fs.default.name parameter -- that's how the
client knows which HDFS server to connect to.
But that machine will not "be part of the cluster" and store HDFS blocks or
compute MapReduce jobs unless you explicitly start the datanode and
tasktracker services on that machine (or do it implicitly by adding the
machine to the slaves file and running bin/start-all.sh, etc.)
- Aaron
On Mon, Mar 30, 2009 at 5:01 AM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
> On Mar 30, 2009, at 3:59 AM, W wrote:
>
> I already try the mountable HDFS, both webDav and FUSE approach, it
>> seem both of it is not
>> production ready ..
>>
>>
> Depends on what you define to be "production ready"; for a business serving
> HDFS to external customers directly, no. But then again, it's questionable
> whether Hadoop itself has that level of "production ready". For internal
> batch/data processing, the uptime requirements are far less.
>
> We move 10's of TB a day through FUSE (yesterday looks like it was a 50TB
> day), and I have seen it move up to 300TB in 12 hours.
>
> Brian
>
>
> CMIIW
>>
>> Best Regards,
>> Wildan
>>
>> ---
>> OpenThink Labs
>> www.tobethink.com
>>
>> Aligning IT and Education
>>
>> 021-99325243
>>>>
>>> Y! : hawking_123
>> Linkedln : http://www.linkedin.com/in/wildanmaulana
>>
>>
>>
>> On Sun, Mar 29, 2009 at 2:52 PM, Sagar Naik <sn...@attributor.com> wrote:
>>
>>> Yes u can
>>> Java Client :
>>> Copy the conf dir (same as one on namenode/datanode) and hadoop jars shud
>>> be
>>> in the classpath of client
>>> Non Java Client :
>>> http://wiki.apache.org/hadoop/MountableHDFS
>>>
>>>
>>>
>>> -Sagar
>>>
>>> -Sagar
>>>
>>> deepya wrote:
>>>
>>>>
>>>> Hi,
>>>> I am SreeDeepya doing MTech in IIIT.I am working on a project named
>>>> cost
>>>> effective and scalable storage server.I configured a small hadoop
>>>> cluster
>>>> with only two nodes one namenode and one datanode.I am new to hadoop.
>>>> I have a small doubt.
>>>>
>>>> Can a system not in the hadoop cluster access the namenode or the
>>>> datanode????If yes,then can you please tell me the necessary
>>>> configurations
>>>> that has to be done.
>>>>
>>>> Thanks in advance.
>>>>
>>>> SreeDeepya
>>>>
>>>>
>>>
>
Re: hadoop-a small doubt
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Mar 30, 2009, at 3:59 AM, W wrote:
> I already try the mountable HDFS, both webDav and FUSE approach, it
> seem both of it is not
> production ready ..
>
Depends on what you define to be "production ready"; for a business
serving HDFS to external customers directly, no. But then again, it's
questionable whether Hadoop itself has that level of "production
ready". For internal batch/data processing, the uptime requirements
are far less.
We move 10's of TB a day through FUSE (yesterday looks like it was a
50TB day), and I have seen it move up to 300TB in 12 hours.
Brian
> CMIIW
>
> Best Regards,
> Wildan
>
> ---
> OpenThink Labs
> www.tobethink.com
>
> Aligning IT and Education
>
>>> 021-99325243
> Y! : hawking_123
> Linkedln : http://www.linkedin.com/in/wildanmaulana
>
>
>
> On Sun, Mar 29, 2009 at 2:52 PM, Sagar Naik <sn...@attributor.com>
> wrote:
>> Yes u can
>> Java Client :
>> Copy the conf dir (same as one on namenode/datanode) and hadoop
>> jars shud be
>> in the classpath of client
>> Non Java Client :
>> http://wiki.apache.org/hadoop/MountableHDFS
>>
>>
>>
>> -Sagar
>>
>> -Sagar
>>
>> deepya wrote:
>>>
>>> Hi,
>>> I am SreeDeepya doing MTech in IIIT.I am working on a project
>>> named cost
>>> effective and scalable storage server.I configured a small hadoop
>>> cluster
>>> with only two nodes one namenode and one datanode.I am new to
>>> hadoop.
>>> I have a small doubt.
>>>
>>> Can a system not in the hadoop cluster access the namenode or the
>>> datanode????If yes,then can you please tell me the necessary
>>> configurations
>>> that has to be done.
>>>
>>> Thanks in advance.
>>>
>>> SreeDeepya
>>>
>>
Re: hadoop-a small doubt
Posted by W <wi...@gmail.com>.
I already try the mountable HDFS, both webDav and FUSE approach, it
seem both of it is not
production ready ..
CMIIW
Best Regards,
Wildan
---
OpenThink Labs
www.tobethink.com
Aligning IT and Education
>> 021-99325243
Y! : hawking_123
Linkedln : http://www.linkedin.com/in/wildanmaulana
On Sun, Mar 29, 2009 at 2:52 PM, Sagar Naik <sn...@attributor.com> wrote:
> Yes u can
> Java Client :
> Copy the conf dir (same as one on namenode/datanode) and hadoop jars shud be
> in the classpath of client
> Non Java Client :
> http://wiki.apache.org/hadoop/MountableHDFS
>
>
>
> -Sagar
>
> -Sagar
>
> deepya wrote:
>>
>> Hi,
>> I am SreeDeepya doing MTech in IIIT.I am working on a project named cost
>> effective and scalable storage server.I configured a small hadoop cluster
>> with only two nodes one namenode and one datanode.I am new to hadoop.
>> I have a small doubt.
>>
>> Can a system not in the hadoop cluster access the namenode or the
>> datanode????If yes,then can you please tell me the necessary
>> configurations
>> that has to be done.
>>
>> Thanks in advance.
>>
>> SreeDeepya
>>
>
Re: hadoop-a small doubt
Posted by Sagar Naik <sn...@attributor.com>.
Yes u can
Java Client :
Copy the conf dir (same as one on namenode/datanode) and hadoop jars
shud be in the classpath of client
Non Java Client :
http://wiki.apache.org/hadoop/MountableHDFS
-Sagar
-Sagar
deepya wrote:
> Hi,
> I am SreeDeepya doing MTech in IIIT.I am working on a project named cost
> effective and scalable storage server.I configured a small hadoop cluster
> with only two nodes one namenode and one datanode.I am new to hadoop.
> I have a small doubt.
>
> Can a system not in the hadoop cluster access the namenode or the
> datanode????If yes,then can you please tell me the necessary configurations
> that has to be done.
>
> Thanks in advance.
>
> SreeDeepya
>
Re: hadoop-a small doubt
Posted by Pankil Doshi <fo...@gmail.com>.
Hey ,
You can do that.That system should have same usrname like those of cluster
and ofcourse it should be able to ssh name node.Also it should have hadoop
and its hadoop-site.xml should be similar .Then u can access namenode,hdfs
etc.
if you are willing to see the web interface that can be done easily using
any system.
deepya wrote:
>
> Hi,
> I am SreeDeepya doing MTech in IIIT.I am working on a project named
> cost effective and scalable storage server.I configured a small hadoop
> cluster with only two nodes one namenode and one datanode.I am new to
> hadoop.
> I have a small doubt.
>
> Can a system not in the hadoop cluster access the namenode or the
> datanode????If yes,then can you please tell me the necessary
> configurations that has to be done.
>
> Thanks in advance.
>
> SreeDeepya
>
--
View this message in context: http://www.nabble.com/hadoop-a-small-doubt-tp22764615p23061794.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.