Is windows Hadoop cluster feasible for production ?

[pulkit khandelwal <kh...@gmail.com>]

Hi team,

I am trying to setup a Hadoop cluster using 5 windows nodes. While doing so
I faced some challenges for which it either takes a lot of time to resolve
or I don’t find any support on Internet. And even after the setup was done
the cluster wasn’t robust enough.

I am not sure if there’s something I am doing wrong or windows cluster in
production is not recommended?

If it’s not recommended to use the windows cluster in production, then can
you please state some reasons for it, as I need some evidence to present at
my workplace.

Please let me know what do you think about this.

Looking forward to hearing from you.


FYI, this cluster is going to be used in production for running spark jobs
on HDFS storage.

Thanks and regards
Pulkit Khandelwal

Re: Is windows Hadoop cluster feasible for production ?

[Chris Nauroth <cn...@apache.org>]

Speaking as a developer formerly involved heavily on Windows compatibility,
I can say I have not participated on any significant Windows cluster
deployment in multiple years.  I do occasionally field client-side
questions (e.g. running the CLI on Windows), but not full cluster
deployments.

Unfortunately, I no longer have easy access to a Windows development
environment, so I can't offer as much help as I did in the past.  Lacking
critical mass of committers with dedication to Windows compatibility, I
would approach with caution.

--Chris

On Fri, Sep 24, 2021, 3:45 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:

> I think it's been a long time since anyone did this.
>
> The engineering was done with microsoft heavily involved, but with their
> focus on Azure *and linux VMs in Azure compute*, I don't think anyone has
> tested -let alone deployed- a windows-based hadoop cluster on recent 3.x
> releases. Certainly (a) our QE teams don't and (b) nobody else files bug
> reports about issues surfacing in testing.
>
> Active Directory is heavily used for kerberos auth.
>
>
>
>
> On Wed, 22 Sept 2021 at 10:25, pulkit khandelwal <
> khandelwalpulkit007@gmail.com> wrote:
>
> > Hi team,
> >
> > I am trying to setup a Hadoop cluster using 5 windows nodes. While doing
> so
> > I faced some challenges for which it either takes a lot of time to
> resolve
> > or I don’t find any support on Internet. And even after the setup was
> done
> > the cluster wasn’t robust enough.
> >
> > I am not sure if there’s something I am doing wrong or windows cluster in
> > production is not recommended?
> >
> > If it’s not recommended to use the windows cluster in production, then
> can
> > you please state some reasons for it, as I need some evidence to present
> at
> > my workplace.
> >
> > Please let me know what do you think about this.
> >
> > Looking forward to hearing from you.
> >
> >
> > FYI, this cluster is going to be used in production for running spark
> jobs
> > on HDFS storage.
> >
> > Thanks and regards
> > Pulkit Khandelwal
> >
>

Re: Is windows Hadoop cluster feasible for production ?

[Steve Loughran <st...@cloudera.com.INVALID>]

I think it's been a long time since anyone did this.

The engineering was done with microsoft heavily involved, but with their
focus on Azure *and linux VMs in Azure compute*, I don't think anyone has
tested -let alone deployed- a windows-based hadoop cluster on recent 3.x
releases. Certainly (a) our QE teams don't and (b) nobody else files bug
reports about issues surfacing in testing.

Active Directory is heavily used for kerberos auth.




On Wed, 22 Sept 2021 at 10:25, pulkit khandelwal <
khandelwalpulkit007@gmail.com> wrote:

> Hi team,
>
> I am trying to setup a Hadoop cluster using 5 windows nodes. While doing so
> I faced some challenges for which it either takes a lot of time to resolve
> or I don’t find any support on Internet. And even after the setup was done
> the cluster wasn’t robust enough.
>
> I am not sure if there’s something I am doing wrong or windows cluster in
> production is not recommended?
>
> If it’s not recommended to use the windows cluster in production, then can
> you please state some reasons for it, as I need some evidence to present at
> my workplace.
>
> Please let me know what do you think about this.
>
> Looking forward to hearing from you.
>
>
> FYI, this cluster is going to be used in production for running spark jobs
> on HDFS storage.
>
> Thanks and regards
> Pulkit Khandelwal
>