You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Prachi Sharma <pr...@gmail.com> on 2016/07/20 07:30:48 UTC
Windows and Linux hadoop cluster
Hi All,
Please let me know if it’s feasible to have hadoop cluster with data nodes
running on multiple Operating systems. For instance few data nodes running
on windows server and others on linux based OS (RHEL,centOS).
If above scenario is feasible then please provide configuration settings
required in various xml
files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data
nodes and namenode.
Thanks !
Prachi
RE: Windows and Linux hadoop cluster
Posted by Ashish Kumar9 <as...@in.ibm.com>.
Only linux distros.
From: Prachi Sharma <pr...@gmail.com>
To: Ashish Kumar9/India/IBM@IBMIN
Cc: user@hadoop.apache.org, Alexander Alten-Lorenz
<wg...@gmail.com>
Date: 07/21/2016 08:45 AM
Subject: RE: Windows and Linux hadoop cluster
Hi Ashish,
Did you also try the heterogeneous cluster with windows and Linux nodes ?
Thanks!
Prachi
On 20-Jul-2016 8:13 PM, "Ashish Kumar9" <as...@in.ibm.com> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux
distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I
did not see any technical challenge .
From: Alexander Alten-Lorenz <wg...@gmail.com>
To: Prachi Sharma <pr...@gmail.com>, "
user@hadoop.apache.org" <us...@hadoop.apache.org>
Date: 07/20/2016 04:42 PM
Subject: RE: Windows and Linux hadoop cluster
Hi,
That should be possible, but will have performance impacts / additional
configurations and potential misbehavior. But in general, it should work
for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
cheers,
--alex
--
b: mapredit.blogspot.com
From: Prachi Sharma
Sent: Wednesday, July 20, 2016 9:31 AM
To: user@hadoop.apache.org
Subject: Windows and Linux hadoop cluster
Hi All,
Please let me know if it’s feasible to have hadoop cluster with data nodes
running on multiple Operating systems. For instance few data nodes running
on windows server and others on linux based OS (RHEL,centOS).
If above scenario is feasible then please provide configuration settings
required in various xml
files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data
nodes and namenode.
Thanks !
Prachi
RE: Windows and Linux hadoop cluster
Posted by Prachi Sharma <pr...@gmail.com>.
Hi Ashish,
Did you also try the heterogeneous cluster with windows and Linux nodes ?
Thanks!
Prachi
On 20-Jul-2016 8:13 PM, "Ashish Kumar9" <as...@in.ibm.com> wrote:
> I have tried heterogeneous hadoop 2.6 cluster across multiple linux
> distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did
> not see any technical challenge .
>
>
>
> From: Alexander Alten-Lorenz <wg...@gmail.com>
> To: Prachi Sharma <pr...@gmail.com>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: 07/20/2016 04:42 PM
> Subject: RE: Windows and Linux hadoop cluster
> ------------------------------
>
>
>
> Hi,
>
> That should be possible, but will have performance impacts / additional
> configurations and potential misbehavior. But in general, it should work
> for Yarn, but not for MRv1.
>
> *https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html*
> <https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html>
>
> cheers,
> --alex
>
> --
> b: mapredit.blogspot.com
>
> *From: **Prachi Sharma* <pr...@gmail.com>
> *Sent: *Wednesday, July 20, 2016 9:31 AM
> *To: **user@hadoop.apache.org* <us...@hadoop.apache.org>
> *Subject: *Windows and Linux hadoop cluster
>
> Hi All,
>
> Please let me know if it’s feasible to have hadoop cluster with data nodes
> running on multiple Operating systems. For instance few data nodes running
> on windows server and others on linux based OS (RHEL,centOS).
>
> If above scenario is feasible then please provide configuration settings
> required in various xml
> files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
> environment files(*hadoop-env.sh/hadoop-cmd.sh*
> <http://hadoop-env.sh/hadoop-cmd.sh>) for windows and linux data nodes
> and namenode.
>
> Thanks !
> Prachi
>
>
>
Re: Windows and Linux hadoop cluster
Posted by pi...@gmail.com.
For "trial" it should just work but for production maintaining multiple environment will be painful. This holds true not only for Hadoop but for languages such as python(in case you use it as a programming language). Not all Additional Hadoop components can run natively on windows.
> On 20-Jul-2016, at 8:13 PM, Ashish Kumar9 <as...@in.ibm.com> wrote:
>
> I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .
>
>
>
> From: Alexander Alten-Lorenz <wg...@gmail.com>
> To: Prachi Sharma <pr...@gmail.com>, "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: 07/20/2016 04:42 PM
> Subject: RE: Windows and Linux hadoop cluster
>
>
>
> Hi,
>
> That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
>
> cheers,
> --alex
>
> --
> b: mapredit.blogspot.com
>
> From: Prachi Sharma
> Sent: Wednesday, July 20, 2016 9:31 AM
> To: user@hadoop.apache.org
> Subject: Windows and Linux hadoop cluster
>
> Hi All,
>
> Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
>
> If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
>
> Thanks !
> Prachi
>
>
RE: Windows and Linux hadoop cluster
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I had used community edition . It was PoC and not production . I had hive
use-case .
From: "sreebalineni ." <sr...@gmail.com>
To: Ashish Kumar9/India/IBM@IBMIN
Cc: Prachi Sharma <pr...@gmail.com>,
user@hadoop.apache.org, Alexander Alten-Lorenz <wg...@gmail.com>
Date: 07/20/2016 10:24 PM
Subject: RE: Windows and Linux hadoop cluster
Was it in production and good amount of workload? That's interesting.
Which distribution was used
On Jul 20, 2016 8:13 PM, "Ashish Kumar9" <as...@in.ibm.com> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux
distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I
did not see any technical challenge .
From: Alexander Alten-Lorenz <wg...@gmail.com>
To: Prachi Sharma <pr...@gmail.com>, "
user@hadoop.apache.org" <us...@hadoop.apache.org>
Date: 07/20/2016 04:42 PM
Subject: RE: Windows and Linux hadoop cluster
Hi,
That should be possible, but will have performance impacts / additional
configurations and potential misbehavior. But in general, it should work
for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
cheers,
--alex
--
b: mapredit.blogspot.com
From: Prachi Sharma
Sent: Wednesday, July 20, 2016 9:31 AM
To: user@hadoop.apache.org
Subject: Windows and Linux hadoop cluster
Hi All,
Please let me know if it’s feasible to have hadoop cluster with data nodes
running on multiple Operating systems. For instance few data nodes running
on windows server and others on linux based OS (RHEL,centOS).
If above scenario is feasible then please provide configuration settings
required in various xml
files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data
nodes and namenode.
Thanks !
Prachi
RE: Windows and Linux hadoop cluster
Posted by "sreebalineni ." <sr...@gmail.com>.
Was it in production and good amount of workload? That's interesting. Which
distribution was used
On Jul 20, 2016 8:13 PM, "Ashish Kumar9" <as...@in.ibm.com> wrote:
> I have tried heterogeneous hadoop 2.6 cluster across multiple linux
> distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did
> not see any technical challenge .
>
>
>
> From: Alexander Alten-Lorenz <wg...@gmail.com>
> To: Prachi Sharma <pr...@gmail.com>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: 07/20/2016 04:42 PM
> Subject: RE: Windows and Linux hadoop cluster
> ------------------------------
>
>
>
> Hi,
>
> That should be possible, but will have performance impacts / additional
> configurations and potential misbehavior. But in general, it should work
> for Yarn, but not for MRv1.
>
> *https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html*
> <https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html>
>
> cheers,
> --alex
>
> --
> b: mapredit.blogspot.com
>
> *From: **Prachi Sharma* <pr...@gmail.com>
> *Sent: *Wednesday, July 20, 2016 9:31 AM
> *To: **user@hadoop.apache.org* <us...@hadoop.apache.org>
> *Subject: *Windows and Linux hadoop cluster
>
> Hi All,
>
> Please let me know if it’s feasible to have hadoop cluster with data nodes
> running on multiple Operating systems. For instance few data nodes running
> on windows server and others on linux based OS (RHEL,centOS).
>
> If above scenario is feasible then please provide configuration settings
> required in various xml
> files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
> environment files(*hadoop-env.sh/hadoop-cmd.sh*
> <http://hadoop-env.sh/hadoop-cmd.sh>) for windows and linux data nodes
> and namenode.
>
> Thanks !
> Prachi
>
>
>
RE: Windows and Linux hadoop cluster
Posted by Ashish Kumar9 <as...@in.ibm.com>.
I have tried heterogeneous hadoop 2.6 cluster across multiple linux
distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I
did not see any technical challenge .
From: Alexander Alten-Lorenz <wg...@gmail.com>
To: Prachi Sharma <pr...@gmail.com>,
"user@hadoop.apache.org" <us...@hadoop.apache.org>
Date: 07/20/2016 04:42 PM
Subject: RE: Windows and Linux hadoop cluster
Hi,
That should be possible, but will have performance impacts / additional
configurations and potential misbehavior. But in general, it should work
for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
cheers,
--alex
--
b: mapredit.blogspot.com
From: Prachi Sharma
Sent: Wednesday, July 20, 2016 9:31 AM
To: user@hadoop.apache.org
Subject: Windows and Linux hadoop cluster
Hi All,
Please let me know if it’s feasible to have hadoop cluster with data nodes
running on multiple Operating systems. For instance few data nodes running
on windows server and others on linux based OS (RHEL,centOS).
If above scenario is feasible then please provide configuration settings
required in various xml
files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data
nodes and namenode.
Thanks !
Prachi
RE: Windows and Linux hadoop cluster
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Security:
Windows doesn’t have a working OpenSSL implementation
Malware, Virus and other typical windows based threads
Disk level permission or encryption
Performance:
Different thread handling per OS
Yarn implementation differs, can cause negative performance
Windows CPU / Core scaling isn’t the same as Linux
I would not go with a mixed environment in production, and I see no sense behind. Stable solutions are often use CentOS, since the TCO is much smaller than in Windows environments. If you’re a Windows shop, go with Azure.
Cheers,
--alex
From: Prachi Sharma
Re: Windows and Linux hadoop cluster
Posted by Prachi Sharma <pr...@gmail.com>.
Hi Alex,
what potential misbehavior and performance impact do you foresee . can you
please give a brief . Any other security threats that could be present in
such cluster or any other constraints in such environment . Please let me
know I need to check the mixed cluster feasibility for production
environment.
Thanks
Prachi
On Wed, Jul 20, 2016 at 4:42 PM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:
> Hi,
>
>
>
> That should be possible, but will have performance impacts / additional
> configurations and potential misbehavior. But in general, it should work
> for Yarn, but not for MRv1.
>
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
>
>
>
> cheers,
>
> --alex
>
>
>
> --
>
> b: mapredit.blogspot.com
>
>
>
> *From: *Prachi Sharma <pr...@gmail.com>
> *Sent: *Wednesday, July 20, 2016 9:31 AM
> *To: *user@hadoop.apache.org
> *Subject: *Windows and Linux hadoop cluster
>
>
>
> Hi All,
>
>
>
> Please let me know if it’s feasible to have hadoop cluster with data nodes
> running on multiple Operating systems. For instance few data nodes running
> on windows server and others on linux based OS (RHEL,centOS).
>
>
>
> If above scenario is feasible then please provide configuration settings
> required in various xml
> files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and
> environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data
> nodes and namenode.
>
>
>
> Thanks !
>
> Prachi
>
>
>
RE: Windows and Linux hadoop cluster
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi,
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
cheers,
--alex
--
b: mapredit.blogspot.com
From: Prachi Sharma