You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mehmet Belgin <me...@oit.gatech.edu> on 2013/02/18 19:00:44 UTC
Using NFS mounted volume for Hadoop installation/configuration
Hi Everyone,
Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
Thanks!
-Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
Cloudera manager or Zettaset can be a choice if you like easy
configuration. This type of software will do the rysnc for you.
On Mon, Feb 18, 2013 at 12:53 PM, Mehmet Belgin <
mehmet.belgin@oit.gatech.edu> wrote:
> It looks like the NFS stability and performance are two main concerns.
> Since my cluster is still experimental, I will continue to use NFS for now.
> In the future, when we have a larger production cluster, I will consider
> local configurations.
>
> Thank you all for your replies!
>
> -Mehmet
>
>
>
> On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
>
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
Cloudera manager or Zettaset can be a choice if you like easy
configuration. This type of software will do the rysnc for you.
On Mon, Feb 18, 2013 at 12:53 PM, Mehmet Belgin <
mehmet.belgin@oit.gatech.edu> wrote:
> It looks like the NFS stability and performance are two main concerns.
> Since my cluster is still experimental, I will continue to use NFS for now.
> In the future, when we have a larger production cluster, I will consider
> local configurations.
>
> Thank you all for your replies!
>
> -Mehmet
>
>
>
> On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
>
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
Cloudera manager or Zettaset can be a choice if you like easy
configuration. This type of software will do the rysnc for you.
On Mon, Feb 18, 2013 at 12:53 PM, Mehmet Belgin <
mehmet.belgin@oit.gatech.edu> wrote:
> It looks like the NFS stability and performance are two main concerns.
> Since my cluster is still experimental, I will continue to use NFS for now.
> In the future, when we have a larger production cluster, I will consider
> local configurations.
>
> Thank you all for your replies!
>
> -Mehmet
>
>
>
> On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
>
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
Cloudera manager or Zettaset can be a choice if you like easy
configuration. This type of software will do the rysnc for you.
On Mon, Feb 18, 2013 at 12:53 PM, Mehmet Belgin <
mehmet.belgin@oit.gatech.edu> wrote:
> It looks like the NFS stability and performance are two main concerns.
> Since my cluster is still experimental, I will continue to use NFS for now.
> In the future, when we have a larger production cluster, I will consider
> local configurations.
>
> Thank you all for your replies!
>
> -Mehmet
>
>
>
> On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
>
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Mehmet Belgin <me...@oit.gatech.edu>.
It looks like the NFS stability and performance are two main concerns. Since my cluster is still experimental, I will continue to use NFS for now. In the future, when we have a larger production cluster, I will consider local configurations.
Thank you all for your replies!
-Mehmet
On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Mehmet Belgin <me...@oit.gatech.edu>.
It looks like the NFS stability and performance are two main concerns. Since my cluster is still experimental, I will continue to use NFS for now. In the future, when we have a larger production cluster, I will consider local configurations.
Thank you all for your replies!
-Mehmet
On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop
installation/configuration
Posted by Nan Zhu <zh...@gmail.com>.
I'm also maintaining an experimental Hadoop cluster, and I need to modify the Hadoop source code and test it,
so just use NFS to deploy the latest version of code, no problem found yet
Best,
--
Nan Zhu
School of Computer Science,
McGill University
On Monday, 18 February, 2013 at 1:09 PM, Chris Embree wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu (mailto:mehmet.belgin@oit.gatech.edu)> wrote:
> > Hi Everyone,
> >
> > Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
> >
> > On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
> >
> > Thanks!
> > -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
To use NFS as datanode fs may bring performance problems. Millions of
requests may block your NFS server.
On Mon, Feb 18, 2013 at 12:09 PM, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop
installation/configuration
Posted by Nan Zhu <zh...@gmail.com>.
I'm also maintaining an experimental Hadoop cluster, and I need to modify the Hadoop source code and test it,
so just use NFS to deploy the latest version of code, no problem found yet
Best,
--
Nan Zhu
School of Computer Science,
McGill University
On Monday, 18 February, 2013 at 1:09 PM, Chris Embree wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu (mailto:mehmet.belgin@oit.gatech.edu)> wrote:
> > Hi Everyone,
> >
> > Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
> >
> > On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
> >
> > Thanks!
> > -Mehmet
Re: Using NFS mounted volume for Hadoop
installation/configuration
Posted by Nan Zhu <zh...@gmail.com>.
I'm also maintaining an experimental Hadoop cluster, and I need to modify the Hadoop source code and test it,
so just use NFS to deploy the latest version of code, no problem found yet
Best,
--
Nan Zhu
School of Computer Science,
McGill University
On Monday, 18 February, 2013 at 1:09 PM, Chris Embree wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu (mailto:mehmet.belgin@oit.gatech.edu)> wrote:
> > Hi Everyone,
> >
> > Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
> >
> > On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
> >
> > Thanks!
> > -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
Just for clarification, we only use NFS for binaries and config files.
HDFS and MarpRed write to local disk. We just don't install an OS there.
:)
On Mon, Feb 18, 2013 at 1:44 PM, Paul Wilkinson
<pa...@gmail.com>wrote:
> That requirement for 100% availability is the issue. If NFS goes down, you
> lose all sorts of things that are critical. This will work for a dev
> cluster, but strongly isn't recommended for production.
>
> As a first step, consider rsync - that way everything is local, so fewer
> external dependencies. After that, consider not managing boxes by hand :)
>
> Paul
>
>
> On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
>
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
Just for clarification, we only use NFS for binaries and config files.
HDFS and MarpRed write to local disk. We just don't install an OS there.
:)
On Mon, Feb 18, 2013 at 1:44 PM, Paul Wilkinson
<pa...@gmail.com>wrote:
> That requirement for 100% availability is the issue. If NFS goes down, you
> lose all sorts of things that are critical. This will work for a dev
> cluster, but strongly isn't recommended for production.
>
> As a first step, consider rsync - that way everything is local, so fewer
> external dependencies. After that, consider not managing boxes by hand :)
>
> Paul
>
>
> On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
>
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
Just for clarification, we only use NFS for binaries and config files.
HDFS and MarpRed write to local disk. We just don't install an OS there.
:)
On Mon, Feb 18, 2013 at 1:44 PM, Paul Wilkinson
<pa...@gmail.com>wrote:
> That requirement for 100% availability is the issue. If NFS goes down, you
> lose all sorts of things that are critical. This will work for a dev
> cluster, but strongly isn't recommended for production.
>
> As a first step, consider rsync - that way everything is local, so fewer
> external dependencies. After that, consider not managing boxes by hand :)
>
> Paul
>
>
> On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
>
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
Just for clarification, we only use NFS for binaries and config files.
HDFS and MarpRed write to local disk. We just don't install an OS there.
:)
On Mon, Feb 18, 2013 at 1:44 PM, Paul Wilkinson
<pa...@gmail.com>wrote:
> That requirement for 100% availability is the issue. If NFS goes down, you
> lose all sorts of things that are critical. This will work for a dev
> cluster, but strongly isn't recommended for production.
>
> As a first step, consider rsync - that way everything is local, so fewer
> external dependencies. After that, consider not managing boxes by hand :)
>
> Paul
>
>
> On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
>
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Paul Wilkinson <pa...@gmail.com>.
That requirement for 100% availability is the issue. If NFS goes down, you lose all sorts of things that are critical. This will work for a dev cluster, but strongly isn't recommended for production.
As a first step, consider rsync - that way everything is local, so fewer external dependencies. After that, consider not managing boxes by hand :)
Paul
On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <me...@oit.gatech.edu> wrote:
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>>
>> Thanks!
>> -Mehmet
>
Re: Using NFS mounted volume for Hadoop
installation/configuration
Posted by Nan Zhu <zh...@gmail.com>.
I'm also maintaining an experimental Hadoop cluster, and I need to modify the Hadoop source code and test it,
so just use NFS to deploy the latest version of code, no problem found yet
Best,
--
Nan Zhu
School of Computer Science,
McGill University
On Monday, 18 February, 2013 at 1:09 PM, Chris Embree wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu (mailto:mehmet.belgin@oit.gatech.edu)> wrote:
> > Hi Everyone,
> >
> > Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
> >
> > On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
> >
> > Thanks!
> > -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Paul Wilkinson <pa...@gmail.com>.
That requirement for 100% availability is the issue. If NFS goes down, you lose all sorts of things that are critical. This will work for a dev cluster, but strongly isn't recommended for production.
As a first step, consider rsync - that way everything is local, so fewer external dependencies. After that, consider not managing boxes by hand :)
Paul
On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <me...@oit.gatech.edu> wrote:
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>>
>> Thanks!
>> -Mehmet
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Paul Wilkinson <pa...@gmail.com>.
That requirement for 100% availability is the issue. If NFS goes down, you lose all sorts of things that are critical. This will work for a dev cluster, but strongly isn't recommended for production.
As a first step, consider rsync - that way everything is local, so fewer external dependencies. After that, consider not managing boxes by hand :)
Paul
On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <me...@oit.gatech.edu> wrote:
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>>
>> Thanks!
>> -Mehmet
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
To use NFS as datanode fs may bring performance problems. Millions of
requests may block your NFS server.
On Mon, Feb 18, 2013 at 12:09 PM, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Paul Wilkinson <pa...@gmail.com>.
That requirement for 100% availability is the issue. If NFS goes down, you lose all sorts of things that are critical. This will work for a dev cluster, but strongly isn't recommended for production.
As a first step, consider rsync - that way everything is local, so fewer external dependencies. After that, consider not managing boxes by hand :)
Paul
On 18 Feb 2013, at 18:09, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100% solid no problems. I've seen mtab get messed up and refuse to remount if NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <me...@oit.gatech.edu> wrote:
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>>
>> Thanks!
>> -Mehmet
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
To use NFS as datanode fs may bring performance problems. Millions of
requests may block your NFS server.
On Mon, Feb 18, 2013 at 12:09 PM, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chen He <ai...@gmail.com>.
To use NFS as datanode fs may bring performance problems. Millions of
requests may block your NFS server.
On Mon, Feb 18, 2013 at 12:09 PM, Chris Embree <ce...@gmail.com> wrote:
> I'm doing that currently. No problems to report so far.
>
> The only pitfall I've found is around NFS stability. If your NAS is 100%
> solid no problems. I've seen mtab get messed up and refuse to remount if
> NFS has any hiccups.
>
> If you want to really crazy, consider NFS for your datanode root fs. See
> the oneSIS project for details. http://onesis.sourceforge.net
>
> Enjoy.
>
>
> On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <
> mehmet.belgin@oit.gatech.edu> wrote:
>
>> Hi Everyone,
>>
>> Will it be any problem if I put the hadoop executables and configuration
>> on a NFS volume, which is shared by all masters and slaves? This way the
>> configuration changes will be available for all nodes, without need for
>> synching any files. While this looks almost like a no-brainer, I am
>> wondering if there are any pitfalls I need to be aware of.
>>
>> On a related question, is there a best practices (do's and don'ts )
>> document that you can suggest other than the regular documentation by
>> Apache?
>>
>> Thanks!
>> -Mehmet
>
>
>
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
I'm doing that currently. No problems to report so far.
The only pitfall I've found is around NFS stability. If your NAS is 100%
solid no problems. I've seen mtab get messed up and refuse to remount if
NFS has any hiccups.
If you want to really crazy, consider NFS for your datanode root fs. See
the oneSIS project for details. http://onesis.sourceforge.net
Enjoy.
On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu
> wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Mehmet Belgin <me...@oit.gatech.edu>.
It looks like the NFS stability and performance are two main concerns. Since my cluster is still experimental, I will continue to use NFS for now. In the future, when we have a larger production cluster, I will consider local configurations.
Thank you all for your replies!
-Mehmet
On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
I'm doing that currently. No problems to report so far.
The only pitfall I've found is around NFS stability. If your NAS is 100%
solid no problems. I've seen mtab get messed up and refuse to remount if
NFS has any hiccups.
If you want to really crazy, consider NFS for your datanode root fs. See
the oneSIS project for details. http://onesis.sourceforge.net
Enjoy.
On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu
> wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
I'm doing that currently. No problems to report so far.
The only pitfall I've found is around NFS stability. If your NAS is 100%
solid no problems. I've seen mtab get messed up and refuse to remount if
NFS has any hiccups.
If you want to really crazy, consider NFS for your datanode root fs. See
the oneSIS project for details. http://onesis.sourceforge.net
Enjoy.
On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu
> wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Chris Embree <ce...@gmail.com>.
I'm doing that currently. No problems to report so far.
The only pitfall I've found is around NFS stability. If your NAS is 100%
solid no problems. I've seen mtab get messed up and refuse to remount if
NFS has any hiccups.
If you want to really crazy, consider NFS for your datanode root fs. See
the oneSIS project for details. http://onesis.sourceforge.net
Enjoy.
On Mon, Feb 18, 2013 at 1:00 PM, Mehmet Belgin <mehmet.belgin@oit.gatech.edu
> wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration
> on a NFS volume, which is shared by all masters and slaves? This way the
> configuration changes will be available for all nodes, without need for
> synching any files. While this looks almost like a no-brainer, I am
> wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts )
> document that you can suggest other than the regular documentation by
> Apache?
>
> Thanks!
> -Mehmet
Re: Using NFS mounted volume for Hadoop installation/configuration
Posted by Mehmet Belgin <me...@oit.gatech.edu>.
It looks like the NFS stability and performance are two main concerns. Since my cluster is still experimental, I will continue to use NFS for now. In the future, when we have a larger production cluster, I will consider local configurations.
Thank you all for your replies!
-Mehmet
On Feb 18, 2013, at 1:00 PM, Mehmet Belgin wrote:
> Hi Everyone,
>
> Will it be any problem if I put the hadoop executables and configuration on a NFS volume, which is shared by all masters and slaves? This way the configuration changes will be available for all nodes, without need for synching any files. While this looks almost like a no-brainer, I am wondering if there are any pitfalls I need to be aware of.
>
> On a related question, is there a best practices (do's and don'ts ) document that you can suggest other than the regular documentation by Apache?
>
> Thanks!
> -Mehmet