Introduction

[Darren Shepherd <da...@gmail.com>]

 All,

I want to introduce myself to the Apache CloudStack community.  I've have
had a long love/hate relationship with CloudStack over the years, but
recently I've decided to fully dedicate myself to working on this platform
(again).

Just a little about me, well okay, this might be a little long.  I've been
a Linux user for a good amount of time.  I've ran Linux as my primary OS
for over 15 years.  Both personally and professionally, my desktops/laptops
have always ran Linux.  As soon as I discovered Linux I also fell in love
with virtualization.  I've ran the gamut in terms of virtualization
technologies.  Type 1, Type 2, containers, etc.  I have a particular
fondness for Xen though.  I've been a Xen user since the 2.x days.

Couple years after I got into Linux I discovered programming.  After so
many failed "./configure && make" I figured I'd should probably learn C/C++
to figure out why the heck nothing ever "make'd" right.  I worked for
awhile running a lab for developers that were writing stuff in Java.  Again
I learned Java to figure out why the heck their stuff kept failing to build
and run.  After I learned Java I took the professional route of leaving the
sysadmin stuff and started programming distributed systems.

As means to an end I developed a system that would take code builds, deploy
them to Xen thin provisioned VMs.  So a developer could go to a portal,
click some stuff and get a clean VM that optionally had a code build on
it.  We would then running automated integration tests against it.  This
was around 2006.  About a year or 2 later I discovered EC2 and was like
"holy crap, that's exactly what I want."  Then I found eucalpytus and
realized there was a whole industry around this stuff.  IaaS is basically
the trifecta for me.  I get to combine my love of Linux, virtualization,
and programming distributed system.   Basically since then I've been
absolutely obsessed with IaaS.

Around 2010 I got a job at Go Daddy specifically to build their public
Cloud product.  Before I even got there they had already decided to go with
vmops/cloud.com.  I'm pretty sure they were already cloud.com when I
started.  At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java
guys, 2 sysengs) we built three generations of clouds.  The first
generation of cloud was basically CloudStack 2.2++.  It was CloudStack 2.2
with a new UI, billing system, storage subsystem, and a tons of operational
tools.  That was the initial public launch of our Cloud.

The second generation was "CloudStack like."  Basically we had a lot of
problems with CloudStack at the time.  First, it was just a complete pain
to operationalize.  Trying to hand off CloudStack to our ops and support
teams was practically impossible.  Basically our level 1 and 2 support
would be very efficient at escalating all issues to level 3 (my team).
Typically the sysadmins would see something is stuck, they'd look at a log
(that made no sense to them at all), see a 100 line stack trace and then
they'd just toss the issue to the developers.  Also, their was no
visibility to the general health of CloudStack.  If you asked ops, "How's
production doing right now?"  Their answer would be, "Well I can ping the
UI and the java processes are still running."  In developer terms that is
kinda like saying, "Well it compiles so it must work fine."

Secondly, we had a hard time extending the platform.  We always seemed to
be a little ahead of the curve in how we wanted to implement the cloud.
How we wanted to run storage and networking didn't match at all what
CloudStack was at that time.  So we embarked on improving it.  It was just
way too difficult.  Basically we felt like 70% of our dev time was just
fighting the framework.  So we made the choice to abandon the platform.  We
started rewriting the the entire platform.  We did this piece meal over
time.  We'd redo a subsystem, put that in production, and then move onto
the next subsystem.  This ended up being very tricky because we had mixture
of CloudStack and our own stuff.  We also always tried to do no downtime
deployments.  So we would almost never lock people out of the UI and most
deployments were during business hours.  We basically finished most of this
work around summer last year.  I gave a talk at OSCON 2012 [1] regarding
what we built.  A couple months later, solely based on business direction,
we decided to stop selling the public cloud (it really wasn't shutdown
until about May this year.  I'm proud to say it ran for almost 8 months
with live customers, some quite large, with no incident and no dedicated
ops team).

After we shutdown the public cloud, we turned our effort to building a
private cloud to run Go Daddy on top of it.  This marked the third
generation of the cloud.  At this point the IaaS system was 100% our code.
The problem was it inherited the CloudStack DB schema and with it a lot of
cruft in dealing with being compatible with CloudStack.  So we started a
clean up of the code, which ended up being a huge refactor.  Our goal was
to be able to support all of Go Daddy running on it.  That meant we needed
a very flexible and extensible platform to support a very heterogeneous
environment.  (We had a very scalable L3 network design that I've yet to
see any IaaS system be able to support too.)  Additionally it needed to
plug into all other systems in Go Daddy for monitor, tracking, and network
management (IPAM, etc).  This was by far my favorite cloud to build.
Really, really complex.  The goal was to build a system that could deploy
over 1 million VMs in less than a day.  (I think we did it too,
unfortunately I quit before I could fully prove it.  We got the system to
the point of deploying 1000 real VMs in 1 minute.  I just never had the
time to fully run the 1m test).

So I've done a lot with cloud provisioning systems.  I was given an
incredible opportunity at Go Daddy that most people don't get.  I was
allowed to build an IaaS system from scratch and given millions and
millions in hardware to build and test it out and run real production work
loads on it.  I very much liked my cloud I built.  Unfortunately, despite
my best efforts, it was all proprietary.  I've decided to abandon all the
proprietary stuff I've done and try to do my best to help out the open
source IaaS landscape.  After looking at all the stacks out there, I
honestly believe CloudStack is the best one.  I reached out to Citrix and
they have been so gracious as to employee me full time to work on
CloudStack.

So expect to hear a lot from me.  I'm initially very interested at
improving the core architecture of CloudStack.  Just a couple random
things.  1) I'm here to code.  Talking is great, collaboration and
information sharing is important, but I'm not here just to talk.  I'm here
to actually get stuff done.  2)  I've only ran production clouds.  None of
this dev/test/poc "optimize for innovation" crap.  I want a system I can
run in production.  So stability, upgrades, making sysadmins happy, is very
important to me.  3)  I'm very pragmatic and practical about how I
implement stuff.  I always have the 2 year vision, but I'm very interested
in what I can deliver in the next 3-6 months.  I enjoy delivering solutions
far more than coding. 4) I'm really bad at typing emails and make tons of
typos and forget important words like "not" or "no" all the time.

I really look forward to gaining the trust and respect of this community
and, more importantly, improving CloudStack.  I'm based out of Phoenix, AZ
area, if anybody happens to be in this fabulous part of the country hit me
up and we'll get together.

Darren Shepherd

[1] You can see my crappy slides for my OSCON talk at
http://www.oscon.com/oscon2012/public/schedule/detail/24035 .  They were
really bad, basically I created them the night before.  Did I mention I'm
good at procrastination too.  Also that has the only known picture of me on
the internet.

Re: Introduction

[Hugo Trippaers <HT...@schubergphilis.com>]

Good to have you back :-)

Cheers,

Hugo

Sent from my iPhone

On 21 aug. 2013, at 21:16, Darren Shepherd <da...@gmail.com> wrote:

> All,
> 
> I want to introduce myself to the Apache CloudStack community.  I've have
> had a long love/hate relationship with CloudStack over the years, but
> recently I've decided to fully dedicate myself to working on this platform
> (again).
> 
> Just a little about me, well okay, this might be a little long.  I've been
> a Linux user for a good amount of time.  I've ran Linux as my primary OS
> for over 15 years.  Both personally and professionally, my desktops/laptops
> have always ran Linux.  As soon as I discovered Linux I also fell in love
> with virtualization.  I've ran the gamut in terms of virtualization
> technologies.  Type 1, Type 2, containers, etc.  I have a particular
> fondness for Xen though.  I've been a Xen user since the 2.x days.
> 
> Couple years after I got into Linux I discovered programming.  After so
> many failed "./configure && make" I figured I'd should probably learn C/C++
> to figure out why the heck nothing ever "make'd" right.  I worked for
> awhile running a lab for developers that were writing stuff in Java.  Again
> I learned Java to figure out why the heck their stuff kept failing to build
> and run.  After I learned Java I took the professional route of leaving the
> sysadmin stuff and started programming distributed systems.
> 
> As means to an end I developed a system that would take code builds, deploy
> them to Xen thin provisioned VMs.  So a developer could go to a portal,
> click some stuff and get a clean VM that optionally had a code build on
> it.  We would then running automated integration tests against it.  This
> was around 2006.  About a year or 2 later I discovered EC2 and was like
> "holy crap, that's exactly what I want."  Then I found eucalpytus and
> realized there was a whole industry around this stuff.  IaaS is basically
> the trifecta for me.  I get to combine my love of Linux, virtualization,
> and programming distributed system.   Basically since then I've been
> absolutely obsessed with IaaS.
> 
> Around 2010 I got a job at Go Daddy specifically to build their public
> Cloud product.  Before I even got there they had already decided to go with
> vmops/cloud.com.  I'm pretty sure they were already cloud.com when I
> started.  At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java
> guys, 2 sysengs) we built three generations of clouds.  The first
> generation of cloud was basically CloudStack 2.2++.  It was CloudStack 2.2
> with a new UI, billing system, storage subsystem, and a tons of operational
> tools.  That was the initial public launch of our Cloud.
> 
> The second generation was "CloudStack like."  Basically we had a lot of
> problems with CloudStack at the time.  First, it was just a complete pain
> to operationalize.  Trying to hand off CloudStack to our ops and support
> teams was practically impossible.  Basically our level 1 and 2 support
> would be very efficient at escalating all issues to level 3 (my team).
> Typically the sysadmins would see something is stuck, they'd look at a log
> (that made no sense to them at all), see a 100 line stack trace and then
> they'd just toss the issue to the developers.  Also, their was no
> visibility to the general health of CloudStack.  If you asked ops, "How's
> production doing right now?"  Their answer would be, "Well I can ping the
> UI and the java processes are still running."  In developer terms that is
> kinda like saying, "Well it compiles so it must work fine."
> 
> Secondly, we had a hard time extending the platform.  We always seemed to
> be a little ahead of the curve in how we wanted to implement the cloud.
> How we wanted to run storage and networking didn't match at all what
> CloudStack was at that time.  So we embarked on improving it.  It was just
> way too difficult.  Basically we felt like 70% of our dev time was just
> fighting the framework.  So we made the choice to abandon the platform.  We
> started rewriting the the entire platform.  We did this piece meal over
> time.  We'd redo a subsystem, put that in production, and then move onto
> the next subsystem.  This ended up being very tricky because we had mixture
> of CloudStack and our own stuff.  We also always tried to do no downtime
> deployments.  So we would almost never lock people out of the UI and most
> deployments were during business hours.  We basically finished most of this
> work around summer last year.  I gave a talk at OSCON 2012 [1] regarding
> what we built.  A couple months later, solely based on business direction,
> we decided to stop selling the public cloud (it really wasn't shutdown
> until about May this year.  I'm proud to say it ran for almost 8 months
> with live customers, some quite large, with no incident and no dedicated
> ops team).
> 
> After we shutdown the public cloud, we turned our effort to building a
> private cloud to run Go Daddy on top of it.  This marked the third
> generation of the cloud.  At this point the IaaS system was 100% our code.
> The problem was it inherited the CloudStack DB schema and with it a lot of
> cruft in dealing with being compatible with CloudStack.  So we started a
> clean up of the code, which ended up being a huge refactor.  Our goal was
> to be able to support all of Go Daddy running on it.  That meant we needed
> a very flexible and extensible platform to support a very heterogeneous
> environment.  (We had a very scalable L3 network design that I've yet to
> see any IaaS system be able to support too.)  Additionally it needed to
> plug into all other systems in Go Daddy for monitor, tracking, and network
> management (IPAM, etc).  This was by far my favorite cloud to build.
> Really, really complex.  The goal was to build a system that could deploy
> over 1 million VMs in less than a day.  (I think we did it too,
> unfortunately I quit before I could fully prove it.  We got the system to
> the point of deploying 1000 real VMs in 1 minute.  I just never had the
> time to fully run the 1m test).
> 
> So I've done a lot with cloud provisioning systems.  I was given an
> incredible opportunity at Go Daddy that most people don't get.  I was
> allowed to build an IaaS system from scratch and given millions and
> millions in hardware to build and test it out and run real production work
> loads on it.  I very much liked my cloud I built.  Unfortunately, despite
> my best efforts, it was all proprietary.  I've decided to abandon all the
> proprietary stuff I've done and try to do my best to help out the open
> source IaaS landscape.  After looking at all the stacks out there, I
> honestly believe CloudStack is the best one.  I reached out to Citrix and
> they have been so gracious as to employee me full time to work on
> CloudStack.
> 
> So expect to hear a lot from me.  I'm initially very interested at
> improving the core architecture of CloudStack.  Just a couple random
> things.  1) I'm here to code.  Talking is great, collaboration and
> information sharing is important, but I'm not here just to talk.  I'm here
> to actually get stuff done.  2)  I've only ran production clouds.  None of
> this dev/test/poc "optimize for innovation" crap.  I want a system I can
> run in production.  So stability, upgrades, making sysadmins happy, is very
> important to me.  3)  I'm very pragmatic and practical about how I
> implement stuff.  I always have the 2 year vision, but I'm very interested
> in what I can deliver in the next 3-6 months.  I enjoy delivering solutions
> far more than coding. 4) I'm really bad at typing emails and make tons of
> typos and forget important words like "not" or "no" all the time.
> 
> I really look forward to gaining the trust and respect of this community
> and, more importantly, improving CloudStack.  I'm based out of Phoenix, AZ
> area, if anybody happens to be in this fabulous part of the country hit me
> up and we'll get together.
> 
> Darren Shepherd
> 
> [1] You can see my crappy slides for my OSCON talk at
> http://www.oscon.com/oscon2012/public/schedule/detail/24035 .  They were
> really bad, basically I created them the night before.  Did I mention I'm
> good at procrastination too.  Also that has the only known picture of me on
> the internet.

Re: Introduction

[Chip Childers <ch...@sungard.com>]

On Wed, Aug 21, 2013 at 12:16:49PM -0700, Darren Shepherd wrote:
>  All,
> 
> I want to introduce myself to the Apache CloudStack community.

Welcome (back?) Darren!

RE: Introduction

["Musayev, Ilya" <im...@webmd.net>]

Welcome back Darren!

Seems like we have a lot in common, I tend to leave "no's and not's" quite often! 

Since your experience was based on older CS 2.x system, I wonder how different (better or worse) it is now? Are we closer to GoDaddy hybrid solution?

-----Original Message-----
From: Frank Zhang [mailto:Frank.Zhang@citrix.com] 
Sent: Wednesday, August 21, 2013 6:59 PM
To: dev@cloudstack.apache.org
Subject: RE: Introduction

Welcome back!

> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> Sent: Wednesday, August 21, 2013 12:17 PM
> To: dev@cloudstack.apache.org
> Subject: Introduction
> 
>  All,
> 
> I want to introduce myself to the Apache CloudStack community.  I've 
> have had a long love/hate relationship with CloudStack over the years, 
> but recently I've decided to fully dedicate myself to working on this platform (again).
> 
> Just a little about me, well okay, this might be a little long.  I've 
> been a Linux user for a good amount of time.  I've ran Linux as my 
> primary OS for over 15 years.  Both personally and professionally, my 
> desktops/laptops have always ran Linux.  As soon as I discovered Linux I also fell in love with virtualization.
> I've ran the gamut in terms of virtualization technologies.  Type 1, 
> Type 2, containers, etc.  I have a particular fondness for Xen though.  
> I've been a Xen user since the 2.x days.
> 
> Couple years after I got into Linux I discovered programming.  After 
> so many failed "./configure && make" I figured I'd should probably 
> learn C/C++ to figure out why the heck nothing ever "make'd" right.  I 
> worked for awhile running a lab for developers that were writing stuff 
> in Java.  Again I learned Java to figure out why the heck their stuff 
> kept failing to build and run.  After I learned Java I took the 
> professional route of leaving the sysadmin stuff and started programming distributed systems.
> 
> As means to an end I developed a system that would take code builds, 
> deploy them to Xen thin provisioned VMs.  So a developer could go to a 
> portal, click some stuff and get a clean VM that optionally had a code 
> build on it.  We would then running automated integration tests against it.  This was around 2006.
> About a year or 2 later I discovered EC2 and was like "holy crap, 
> that's exactly what I want."  Then I found eucalpytus and realized 
> there was a whole industry around this stuff.  IaaS is basically the 
> trifecta for me.  I get to combine my love of Linux, virtualization,
> and programming distributed system.   Basically since then I've been
> absolutely obsessed with IaaS.
> 
> Around 2010 I got a job at Go Daddy specifically to build their public 
> Cloud product.  Before I even got there they had already decided to go 
> with vmops/cloud.com.  I'm pretty sure they were already cloud.com when I started.
> At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java guys, 
> 2 sysengs) we built three generations of clouds.  The first generation 
> of cloud was basically CloudStack 2.2++.  It was CloudStack 2.2 with a 
> new UI, billing system, storage subsystem, and a tons of operational 
> tools.  That was the initial public launch of our Cloud.
> 
> The second generation was "CloudStack like."  Basically we had a lot 
> of problems with CloudStack at the time.  First, it was just a 
> complete pain to operationalize.  Trying to hand off CloudStack to our 
> ops and support teams was practically impossible.  Basically our level 
> 1 and 2 support would be very efficient at escalating all issues to level 3 (my team).
> Typically the sysadmins would see something is stuck, they'd look at a 
> log (that made no sense to them at all), see a 100 line stack trace 
> and then they'd just toss the issue to the developers.  Also, their 
> was no visibility to the general health of CloudStack.  If you asked ops, "How's production doing right now?"
> Their answer would be, "Well I can ping the UI and the java processes 
> are still running."  In developer terms that is kinda like saying, 
> "Well it compiles so it must work fine."
> 
> Secondly, we had a hard time extending the platform.  We always seemed 
> to be a little ahead of the curve in how we wanted to implement the cloud.
> How we wanted to run storage and networking didn't match at all what 
> CloudStack was at that time.  So we embarked on improving it.  It was 
> just way too difficult.  Basically we felt like 70% of our dev time 
> was just fighting the framework.  So we made the choice to abandon the 
> platform.  We started rewriting the the entire platform.  We did this 
> piece meal over time.  We'd redo a subsystem, put that in production, 
> and then move onto the next subsystem.  This ended up being very 
> tricky because we had mixture of CloudStack and our own stuff.  We 
> also always tried to do no downtime deployments.  So we would almost 
> never lock people out of the UI and most deployments were during 
> business hours.  We basically finished most of this work around summer 
> last year.  I gave a talk at OSCON 2012 [1] regarding what we built.  
> A couple months later, solely based on business direction, we decided 
> to stop selling the public cloud (it really wasn't shutdown until 
> about May this year.  I'm proud to say it ran for almost 8 months with live customers, some quite large, with no incident and no dedicated ops team).
> 
> After we shutdown the public cloud, we turned our effort to building a 
> private cloud to run Go Daddy on top of it.  This marked the third 
> generation of the cloud.  At this point the IaaS system was 100% our code.
> The problem was it inherited the CloudStack DB schema and with it a 
> lot of cruft in dealing with being compatible with CloudStack.  So we 
> started a clean up of the code, which ended up being a huge refactor.  
> Our goal was to be able to support all of Go Daddy running on it.  
> That meant we needed a very flexible and extensible platform to 
> support a very heterogeneous environment.  (We had a very scalable L3 
> network design that I've yet to see any IaaS system be able to support 
> too.)  Additionally it needed to plug into all other systems in Go 
> Daddy for monitor, tracking, and network management (IPAM, etc).  This was by far my favorite cloud to build.
> Really, really complex.  The goal was to build a system that could 
> deploy over 1 million VMs in less than a day.  (I think we did it too, 
> unfortunately I quit before I could fully prove it.  We got the system 
> to the point of deploying 1000 real VMs in 1 minute.  I just never had the time to fully run the 1m test).
> 
> So I've done a lot with cloud provisioning systems.  I was given an 
> incredible opportunity at Go Daddy that most people don't get.  I was 
> allowed to build an IaaS system from scratch and given millions and 
> millions in hardware to build and test it out and run real production 
> work loads on it.  I very much liked my cloud I built.  Unfortunately, 
> despite my best efforts, it was all proprietary.  I've decided to 
> abandon all the proprietary stuff I've done and try to do my best to 
> help out the open source IaaS landscape.  After looking at all the 
> stacks out there, I honestly believe CloudStack is the best one.  I reached out to Citrix and they have been so gracious as to employee me full time to work on CloudStack.
> 
> So expect to hear a lot from me.  I'm initially very interested at 
> improving the core architecture of CloudStack.  Just a couple random 
> things.  1) I'm here to code.  Talking is great, collaboration and 
> information sharing is important, but I'm not here just to talk.  I'm 
> here to actually get stuff done.  2)  I've only ran production clouds.  
> None of this dev/test/poc "optimize for innovation" crap.  I want a 
> system I can run in production.  So stability, upgrades, making 
> sysadmins happy, is very important to me.  3)  I'm very pragmatic and 
> practical about how I implement stuff.  I always have the 2 year 
> vision, but I'm very interested in what I can deliver in the next 3-6 
> months.  I enjoy delivering solutions far more than coding. 4) I'm really bad at typing emails and make tons of typos and forget important words like "not" or "no" all the time.
> 
> I really look forward to gaining the trust and respect of this 
> community and, more importantly, improving CloudStack.  I'm based out 
> of Phoenix, AZ area, if anybody happens to be in this fabulous part of 
> the country hit me up and we'll get together.
> 
> Darren Shepherd
> 
> [1] You can see my crappy slides for my OSCON talk at
> http://www.oscon.com/oscon2012/public/schedule/detail/24035 .  They 
> were really bad, basically I created them the night before.  Did I 
> mention I'm good at procrastination too.  Also that has the only known picture of me on the internet.

RE: Introduction

[Frank Zhang <Fr...@citrix.com>]

Welcome back!

> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> Sent: Wednesday, August 21, 2013 12:17 PM
> To: dev@cloudstack.apache.org
> Subject: Introduction
> 
>  All,
> 
> I want to introduce myself to the Apache CloudStack community.  I've have had
> a long love/hate relationship with CloudStack over the years, but recently I've
> decided to fully dedicate myself to working on this platform (again).
> 
> Just a little about me, well okay, this might be a little long.  I've been a Linux
> user for a good amount of time.  I've ran Linux as my primary OS for over 15
> years.  Both personally and professionally, my desktops/laptops have always
> ran Linux.  As soon as I discovered Linux I also fell in love with virtualization.
> I've ran the gamut in terms of virtualization technologies.  Type 1, Type 2,
> containers, etc.  I have a particular fondness for Xen though.  I've been a Xen
> user since the 2.x days.
> 
> Couple years after I got into Linux I discovered programming.  After so many
> failed "./configure && make" I figured I'd should probably learn C/C++ to figure
> out why the heck nothing ever "make'd" right.  I worked for awhile running a
> lab for developers that were writing stuff in Java.  Again I learned Java to figure
> out why the heck their stuff kept failing to build and run.  After I learned Java I
> took the professional route of leaving the sysadmin stuff and started
> programming distributed systems.
> 
> As means to an end I developed a system that would take code builds, deploy
> them to Xen thin provisioned VMs.  So a developer could go to a portal, click
> some stuff and get a clean VM that optionally had a code build on it.  We would
> then running automated integration tests against it.  This was around 2006.
> About a year or 2 later I discovered EC2 and was like "holy crap, that's exactly
> what I want."  Then I found eucalpytus and realized there was a whole industry
> around this stuff.  IaaS is basically the trifecta for me.  I get to combine my love
> of Linux, virtualization,
> and programming distributed system.   Basically since then I've been
> absolutely obsessed with IaaS.
> 
> Around 2010 I got a job at Go Daddy specifically to build their public Cloud
> product.  Before I even got there they had already decided to go with
> vmops/cloud.com.  I'm pretty sure they were already cloud.com when I started.
> At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java guys, 2 sysengs)
> we built three generations of clouds.  The first generation of cloud was
> basically CloudStack 2.2++.  It was CloudStack 2.2 with a new UI, billing system,
> storage subsystem, and a tons of operational tools.  That was the initial public
> launch of our Cloud.
> 
> The second generation was "CloudStack like."  Basically we had a lot of
> problems with CloudStack at the time.  First, it was just a complete pain to
> operationalize.  Trying to hand off CloudStack to our ops and support teams was
> practically impossible.  Basically our level 1 and 2 support would be very
> efficient at escalating all issues to level 3 (my team).
> Typically the sysadmins would see something is stuck, they'd look at a log (that
> made no sense to them at all), see a 100 line stack trace and then they'd just
> toss the issue to the developers.  Also, their was no visibility to the general
> health of CloudStack.  If you asked ops, "How's production doing right now?"
> Their answer would be, "Well I can ping the UI and the java processes are still
> running."  In developer terms that is kinda like saying, "Well it compiles so it
> must work fine."
> 
> Secondly, we had a hard time extending the platform.  We always seemed to
> be a little ahead of the curve in how we wanted to implement the cloud.
> How we wanted to run storage and networking didn't match at all what
> CloudStack was at that time.  So we embarked on improving it.  It was just way
> too difficult.  Basically we felt like 70% of our dev time was just fighting the
> framework.  So we made the choice to abandon the platform.  We started
> rewriting the the entire platform.  We did this piece meal over time.  We'd
> redo a subsystem, put that in production, and then move onto the next
> subsystem.  This ended up being very tricky because we had mixture of
> CloudStack and our own stuff.  We also always tried to do no downtime
> deployments.  So we would almost never lock people out of the UI and most
> deployments were during business hours.  We basically finished most of this
> work around summer last year.  I gave a talk at OSCON 2012 [1] regarding
> what we built.  A couple months later, solely based on business direction, we
> decided to stop selling the public cloud (it really wasn't shutdown until about
> May this year.  I'm proud to say it ran for almost 8 months with live customers,
> some quite large, with no incident and no dedicated ops team).
> 
> After we shutdown the public cloud, we turned our effort to building a private
> cloud to run Go Daddy on top of it.  This marked the third generation of the
> cloud.  At this point the IaaS system was 100% our code.
> The problem was it inherited the CloudStack DB schema and with it a lot of
> cruft in dealing with being compatible with CloudStack.  So we started a clean
> up of the code, which ended up being a huge refactor.  Our goal was to be able
> to support all of Go Daddy running on it.  That meant we needed a very flexible
> and extensible platform to support a very heterogeneous environment.  (We
> had a very scalable L3 network design that I've yet to see any IaaS system be
> able to support too.)  Additionally it needed to plug into all other systems in Go
> Daddy for monitor, tracking, and network management (IPAM, etc).  This was
> by far my favorite cloud to build.
> Really, really complex.  The goal was to build a system that could deploy over 1
> million VMs in less than a day.  (I think we did it too, unfortunately I quit before
> I could fully prove it.  We got the system to the point of deploying 1000 real
> VMs in 1 minute.  I just never had the time to fully run the 1m test).
> 
> So I've done a lot with cloud provisioning systems.  I was given an incredible
> opportunity at Go Daddy that most people don't get.  I was allowed to build an
> IaaS system from scratch and given millions and millions in hardware to build
> and test it out and run real production work loads on it.  I very much liked my
> cloud I built.  Unfortunately, despite my best efforts, it was all proprietary.  I've
> decided to abandon all the proprietary stuff I've done and try to do my best to
> help out the open source IaaS landscape.  After looking at all the stacks out
> there, I honestly believe CloudStack is the best one.  I reached out to Citrix and
> they have been so gracious as to employee me full time to work on CloudStack.
> 
> So expect to hear a lot from me.  I'm initially very interested at improving the
> core architecture of CloudStack.  Just a couple random things.  1) I'm here to
> code.  Talking is great, collaboration and information sharing is important, but
> I'm not here just to talk.  I'm here to actually get stuff done.  2)  I've only ran
> production clouds.  None of this dev/test/poc "optimize for innovation" crap.  I
> want a system I can run in production.  So stability, upgrades, making
> sysadmins happy, is very important to me.  3)  I'm very pragmatic and practical
> about how I implement stuff.  I always have the 2 year vision, but I'm very
> interested in what I can deliver in the next 3-6 months.  I enjoy delivering
> solutions far more than coding. 4) I'm really bad at typing emails and make
> tons of typos and forget important words like "not" or "no" all the time.
> 
> I really look forward to gaining the trust and respect of this community and,
> more importantly, improving CloudStack.  I'm based out of Phoenix, AZ area, if
> anybody happens to be in this fabulous part of the country hit me up and we'll
> get together.
> 
> Darren Shepherd
> 
> [1] You can see my crappy slides for my OSCON talk at
> http://www.oscon.com/oscon2012/public/schedule/detail/24035 .  They were
> really bad, basically I created them the night before.  Did I mention I'm good at
> procrastination too.  Also that has the only known picture of me on the internet.