question about hadoop

[dechao bu <de...@gmail.com>]

hello,
     I want to deploy Hadoop on a cluster. In this cluster, different nodes
share same file system. If I make changes to files on node1. then other
nodes will have the same changes. (The file system of this cluster is
perhaps called NFS ).
     I don't know whether this cluster is fit for deploying Hadoop.


     Look forward to you reply. Thank you.

Re: question about hadoop

[Edson Ramiro <er...@gmail.com>]

Abhishek,

I'm running Hadoop here and the cluster admin had mounted the NFS with
nolocks option.

So, I was getting "No locks available" message.

I said it just to pay attention in these kind of config ; )


--
Edson Ramiro Lucas Filho
http://www.inf.ufpr.br/erlf07/


On 12 May 2010 14:42, abhishek sharma <ab...@usc.edu> wrote:

> Edson,
>
>
> > Pay attention that running Hadoop on NFS you can get problems with locks.
>
> What locks are you referring to? I ran Hadoop on NFS and never ran
> into any problems. I had a small cluster with 10 servers all connected
> to the same switch.
>
> >
> > And if you're looking for process large files, your network will probably
> be
> > a bottleneck.
>
> However, the network did become a bottleneck.
>
> Abhishek
>

Re: question about hadoop

[abhishek sharma <ab...@usc.edu>]

Edson,


> Pay attention that running Hadoop on NFS you can get problems with locks.

What locks are you referring to? I ran Hadoop on NFS and never ran
into any problems. I had a small cluster with 10 servers all connected
to the same switch.

>
> And if you're looking for process large files, your network will probably be
> a bottleneck.

However, the network did become a bottleneck.

Abhishek

Re: question about hadoop

[Edson Ramiro <er...@gmail.com>]

Dechao bu,

Pay attention that running Hadoop on NFS you can get problems with locks.

And if you're looking for process large files, your network will probably be
a bottleneck.


--
Edson Ramiro Lucas Filho
http://www.inf.ufpr.br/erlf07/


On 12 May 2010 12:47, abhishek sharma <ab...@usc.edu> wrote:

> If the HDFS uses the NFS to store files, then all I/O during the
> execution of map and reduce tasks will use the NFS instead of the
> local disks on each machine in the cluster (if they have one). This
> can become a bottleneck if you have lots of tasks running
> simultaneously.
>
> However, even with the NFS, you can still use Hadoop to run multiple
> map and reduce tasks in parallel.
>
> Abhishek
>
> On Wed, May 12, 2010 at 8:31 AM, dechao bu <de...@gmail.com> wrote:
> > hello,
> >     I want to deploy Hadoop on a cluster. In this cluster, different
> nodes
> > share same file system. If I make changes to files on node1. then other
> > nodes will have the same changes. (The file system of this cluster is
> > perhaps called NFS ).
> >     I don't know whether this cluster is fit for deploying Hadoop.
> >
> >
> >     Look forward to you reply. Thank you.
> >
>

Re: question about hadoop

[abhishek sharma <ab...@usc.edu>]

If the HDFS uses the NFS to store files, then all I/O during the
execution of map and reduce tasks will use the NFS instead of the
local disks on each machine in the cluster (if they have one). This
can become a bottleneck if you have lots of tasks running
simultaneously.

However, even with the NFS, you can still use Hadoop to run multiple
map and reduce tasks in parallel.

Abhishek

On Wed, May 12, 2010 at 8:31 AM, dechao bu <de...@gmail.com> wrote:
> hello,
>     I want to deploy Hadoop on a cluster. In this cluster, different nodes
> share same file system. If I make changes to files on node1. then other
> nodes will have the same changes. (The file system of this cluster is
> perhaps called NFS ).
>     I don't know whether this cluster is fit for deploying Hadoop.
>
>
>     Look forward to you reply. Thank you.
>