Contributing Spark Infrastructure Configuration Docs

[Stephen Watt <sw...@redhat.com>]

Hi Folks

My name is Steve Watt and I work in the CTO Office at Red Hat. I've recently spent quite a bit of time designing single rack and multi-rack infrastructures for Spark for our own hardware procurement at Red Hat and I thought the diagrams and server specs for both Dell and HP would be useful to the broader community as well. Even if folks don't want to go with my exact design, having the designs as a starting point should save  quite a bit of time. I think I can fold this quite easily into http://spark.apache.org/docs/latest/hardware-provisioning.html

However, before submitting a pull request for the modified page I thought I'd send this note up front and see if anyone wanted to provide some feedback first. If there is none, I'll submit the pull request early next week.

Regards
Steve Watt

Re: Contributing Spark Infrastructure Configuration Docs

[Andrew Ash <an...@andrewash.com>]

I would appreciate seeing the specs you came up with as well but don't need
to particularly quickly.  I'll wait until seeing the PR to comment on the
specifics, but have some questions about the thought process that went into
configuring the hardware.

Is the idea to see how you spec'd out memory/disk/processor for each
physical machine?  How did you optimize the ratios between the three for
Spark specifically?  Were you able to test this configuration against
others to optimize for price per performance?

Sorry for the barrage of questions -- I know talking hardware tends to
bring out the crazy in people.

Cheers,
Andrew


On Thu, Jun 5, 2014 at 4:04 PM, Krishna Sankar <ks...@gmail.com> wrote:

> Stephen,
>     We are working thru Dell configurations; would be happy to review your
> diagrams and offer feedback from our experience. Let me know the URLs.
> Cheers
> <k/>
>
>
> On Thu, Jun 5, 2014 at 2:51 PM, Stephen Watt <sw...@redhat.com> wrote:
>
> > Hi Folks
> >
> > My name is Steve Watt and I work in the CTO Office at Red Hat. I've
> > recently spent quite a bit of time designing single rack and multi-rack
> > infrastructures for Spark for our own hardware procurement at Red Hat
> and I
> > thought the diagrams and server specs for both Dell and HP would be
> useful
> > to the broader community as well. Even if folks don't want to go with my
> > exact design, having the designs as a starting point should save  quite a
> > bit of time. I think I can fold this quite easily into
> > http://spark.apache.org/docs/latest/hardware-provisioning.html
> >
> > However, before submitting a pull request for the modified page I thought
> > I'd send this note up front and see if anyone wanted to provide some
> > feedback first. If there is none, I'll submit the pull request early next
> > week.
> >
> > Regards
> > Steve Watt
> >
>

Re: Contributing Spark Infrastructure Configuration Docs

[Krishna Sankar <ks...@gmail.com>]

Stephen,
    We are working thru Dell configurations; would be happy to review your
diagrams and offer feedback from our experience. Let me know the URLs.
Cheers
<k/>


On Thu, Jun 5, 2014 at 2:51 PM, Stephen Watt <sw...@redhat.com> wrote:

> Hi Folks
>
> My name is Steve Watt and I work in the CTO Office at Red Hat. I've
> recently spent quite a bit of time designing single rack and multi-rack
> infrastructures for Spark for our own hardware procurement at Red Hat and I
> thought the diagrams and server specs for both Dell and HP would be useful
> to the broader community as well. Even if folks don't want to go with my
> exact design, having the designs as a starting point should save  quite a
> bit of time. I think I can fold this quite easily into
> http://spark.apache.org/docs/latest/hardware-provisioning.html
>
> However, before submitting a pull request for the modified page I thought
> I'd send this note up front and see if anyone wanted to provide some
> feedback first. If there is none, I'll submit the pull request early next
> week.
>
> Regards
> Steve Watt
>