name change of cloud- to cloudstack- explosion

[Marcus Sorensen <sh...@gmail.com>]

 I (and others) have been working through a ton of issues revolving
around this name change, and while I think it's a good move in
general, I wanted to bring up two points to consider for similar
changes in the future. I've been primarily working in the 4.1 branch.

1)  This is showing me just how tied the code is to the installation.
Mostly on the agent side, but a bit on the server side as well.
Someone mentioned in the future being able to download an archive and
just run it wherever you extract it, I think there's going to be a bit
of work there in making paths compatible.  It has also been mentioned
that we should make some globals such that we don't have paths
sprinkled all over the code, that's probably worthwhile as well.

2) I'm beginning to think that this sort of thing should be done in a
branch, thoroughly tested, and then merged in. I'm not sure what the
thinking was here as-is. If someone moves a file elsewhere, are they
thinking "this needs to be done, we'll just do it, see what breaks and
fix it later", or are they just unaware of item #1 above and have no
idea what code might be relying on that file being there in the right
place. I would hope that someone would at least do a grep through the
code or something. This is perhaps a subtle thing because changing
paths doesn't necessarily cause builds to break or some tests to fail.

Sorry for the rant. I understand that we didn't have packages at all a
few weeks ago, so this should be a step up. Ideally, changing the way
something is packaged shouldn't require changes to the code, or at
worst minimal changes, but we're not there and I think it caught us
off guard. Perhaps I don't understand all of the subtleties, but the
lesson I take from this is that we should have first made packaging
work as it did before, instead of introducing sweeping changes at the
same time.

RE: name change of cloud- to cloudstack- explosion

[Hugo Trippaers <HT...@schubergphilis.com>]

Heya guys,

When we started discussing packaging for the 4.1 branch (discussiong which actually started when we wanted to ditch waf) we aimed for some fundamental changes. The packaging at that time was based on the idea that CloudStack would be an integrated solution with the OS it would be deployed on. So building was a process of taking the source code and run a few commands and you end up with the RedHat or CentOS server with CloudStack installed on it. The tree really reflected this idea as it was setup to go from source code to installed rpm in a single pass. 

One of the ideas with the new build system (maven) was to separate this into separate stages. Source to Binary (using) maven, Binary to Package (using rpmbuild or dpkg). In the end the idea is that we move distribution specific stuff out of the "main" code directories and separate them in the packaging directory. This would make it more easy to get CloudStack into distros. 

Changing the name to cloudstack is a bit of marketing; cloud is very specific and cloudstack really indicates what you are installing and running. And the name was very tied in with the whole build system. A lot of stuff in the package files relates to the name of the package. It might have been easier to leave the name at cloud, but now we can identify all issues and all places where we have hardcoded assumptions on paths and names. Ideally these would be removed from the code and replaced by property references where possible. That would also make it easy for distros to change the way cloudstack is installed without having to write a custom patch to change the sourcecode. 

So I agree, lot of work going on to make it work, but I think it's a worthwhile effort and sysadmins/distros will thank us for this.


Cheers,

Hugo

> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Thursday, February 14, 2013 11:23 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: name change of cloud- to cloudstack- explosion
> 
> For everyone's information, we may end up hitting another issue where the
> code is being changed to use the new paths per how the packaging works,
> but people who have installations that don't rely on packaging (say dev
> environments) may miss out on that config change that the packages
> provide. Which will resurface identical bugs. What I'm concerned about is
> people rebuilding and running jars in their dev environment that actually rely
> also on things that were installed to the filesystem and not updated at the
> time of building the new code.
> 
> On Thu, Feb 14, 2013 at 10:32 AM, Rohit Yadav <bh...@apache.org>
> wrote:
> > Let me rant again: A symlink would just have worked if they dislike
> > cloud-, cloudstack- prefix opens just another can of worms.

I think a symlink hides the problem instead of actually fixing the problem

> >
> > Regards.
> >
> > On Thu, Feb 14, 2013 at 10:56 PM, Marcus Sorensen
> <sh...@gmail.com> wrote:
> >> Ideally, we could just install our whole tree somewhere, and leave it
> >> up to the packagers to do system specific stuff. For example, with
> >> CentOS we could dump a whole tree in /opt, then in the packaging
> >> create symlinks to /opt/whatever in the places an admin would expect
> >> to see config files, copy an init script that starts cloudstack in
> >> the location we chose for the tree, etc.
> >>
> >> As far as the recent fixes, I've fixed things like:
> >>
> >> change places where cloud-agent and cloud-setup-agent scripts were
> >> called in java code to use cloudstack-agent
> >>
> >> various things that were looking in /var/lib/cloud to
> >> /var/cloudstack, fixed things such as ssh key generation and unable
> >> to log in with default admin/password upon install
> >>
> >> change the path to systemvm.iso in the kvm agent from
> >> /usr/lib64/cloud to /usr/share/cloudstack-common  (this issue was
> >> only apparent after noticing that dhcp wasn't working on the routers,
> >> then noticing that the systemvm.iso was out of date, then finding the
> >> path was bad)
> >>
> >> Were being told to install vhd-util in /usr/share/cloud rather than
> >> /usr/share/cloudstack on mgt server
> >>
> >> On Thu, Feb 14, 2013 at 10:06 AM, Alex Huang <Al...@citrix.com>
> wrote:
> >>> Marcus,
> >>>
> >>> It's a good rant!  I brought up the whole download an archive and run it
> and partly because I figure there's problems like this since CloudStack has
> always been installed one way and one way only.
> >>>
> >>> But I don't know the specifics.  Do you have any examples you can
> share?  That way we can show people why this should not be done.  Will be a
> good lesson for us all.  Also if you show some examples, other developers
> can help in locating similar problems so you don't have to do all the work
> alone?
> >>>
> >>> Thanks!
> >>> --Alex
> >>>
> >>>> -----Original Message-----
> >>>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >>>> Sent: Thursday, February 14, 2013 8:38 AM
> >>>> To: cloudstack-dev@incubator.apache.org
> >>>> Subject: name change of cloud- to cloudstack- explosion
> >>>>
> >>>>  I (and others) have been working through a ton of issues revolving
> >>>> around this name change, and while I think it's a good move in
> >>>> general, I wanted to bring up two points to consider for similar
> >>>> changes in the future. I've been primarily working in the 4.1 branch.
> >>>>
> >>>> 1)  This is showing me just how tied the code is to the installation.
> >>>> Mostly on the agent side, but a bit on the server side as well.
> >>>> Someone mentioned in the future being able to download an archive
> >>>> and just run it wherever you extract it, I think there's going to
> >>>> be a bit of work there in making paths compatible.  It has also
> >>>> been mentioned that we should make some globals such that we don't
> >>>> have paths sprinkled all over the code, that's probably worthwhile as
> well.
> >>>>
> >>>> 2) I'm beginning to think that this sort of thing should be done in
> >>>> a branch, thoroughly tested, and then merged in. I'm not sure what
> >>>> the thinking was here as-is. If someone moves a file elsewhere, are
> >>>> they thinking "this needs to be done, we'll just do it, see what
> >>>> breaks and fix it later", or are they just unaware of item #1 above
> >>>> and have no idea what code might be relying on that file being
> >>>> there in the right place. I would hope that someone would at least
> >>>> do a grep through the code or something. This is perhaps a subtle
> >>>> thing because changing paths doesn't necessarily cause builds to break
> or some tests to fail.
> >>>>
> >>>> Sorry for the rant. I understand that we didn't have packages at
> >>>> all a few weeks ago, so this should be a step up. Ideally, changing
> >>>> the way something is packaged shouldn't require changes to the
> >>>> code, or at worst minimal changes, but we're not there and I think
> >>>> it caught us off guard. Perhaps I don't understand all of the
> >>>> subtleties, but the lesson I take from this is that we should have
> >>>> first made packaging work as it did before, instead of introducing
> >>>> sweeping changes at the same time.

Re: name change of cloud- to cloudstack- explosion

[Marcus Sorensen <sh...@gmail.com>]

For everyone's information, we may end up hitting another issue where
the code is being changed to use the new paths per how the packaging
works, but people who have installations that don't rely on packaging
(say dev environments) may miss out on that config change that the
packages provide. Which will resurface identical bugs. What I'm
concerned about is people rebuilding and running jars in their dev
environment that actually rely also on things that were installed to
the filesystem and not updated at the time of building the new code.

On Thu, Feb 14, 2013 at 10:32 AM, Rohit Yadav <bh...@apache.org> wrote:
> Let me rant again: A symlink would just have worked if they dislike
> cloud-, cloudstack- prefix opens just another can of worms.
>
> Regards.
>
> On Thu, Feb 14, 2013 at 10:56 PM, Marcus Sorensen <sh...@gmail.com> wrote:
>> Ideally, we could just install our whole tree somewhere, and leave it
>> up to the packagers to do system specific stuff. For example, with
>> CentOS we could dump a whole tree in /opt, then in the packaging
>> create symlinks to /opt/whatever in the places an admin would expect
>> to see config files, copy an init script that starts cloudstack in the
>> location we chose for the tree, etc.
>>
>> As far as the recent fixes, I've fixed things like:
>>
>> change places where cloud-agent and cloud-setup-agent scripts were
>> called in java code to use cloudstack-agent
>>
>> various things that were looking in /var/lib/cloud to /var/cloudstack,
>> fixed things such as ssh key generation and unable to log in with
>> default admin/password upon install
>>
>> change the path to systemvm.iso in the kvm agent from /usr/lib64/cloud
>> to /usr/share/cloudstack-common  (this issue was only apparent after
>> noticing that dhcp wasn't working on the routers, then noticing that
>> the systemvm.iso was out of date, then finding the path was bad)
>>
>> Were being told to install vhd-util in /usr/share/cloud rather than
>> /usr/share/cloudstack on mgt server
>>
>> On Thu, Feb 14, 2013 at 10:06 AM, Alex Huang <Al...@citrix.com> wrote:
>>> Marcus,
>>>
>>> It's a good rant!  I brought up the whole download an archive and run it and partly because I figure there's problems like this since CloudStack has always been installed one way and one way only.
>>>
>>> But I don't know the specifics.  Do you have any examples you can share?  That way we can show people why this should not be done.  Will be a good lesson for us all.  Also if you show some examples, other developers can help in locating similar problems so you don't have to do all the work alone?
>>>
>>> Thanks!
>>> --Alex
>>>
>>>> -----Original Message-----
>>>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>>> Sent: Thursday, February 14, 2013 8:38 AM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: name change of cloud- to cloudstack- explosion
>>>>
>>>>  I (and others) have been working through a ton of issues revolving
>>>> around this name change, and while I think it's a good move in
>>>> general, I wanted to bring up two points to consider for similar
>>>> changes in the future. I've been primarily working in the 4.1 branch.
>>>>
>>>> 1)  This is showing me just how tied the code is to the installation.
>>>> Mostly on the agent side, but a bit on the server side as well.
>>>> Someone mentioned in the future being able to download an archive and
>>>> just run it wherever you extract it, I think there's going to be a bit
>>>> of work there in making paths compatible.  It has also been mentioned
>>>> that we should make some globals such that we don't have paths
>>>> sprinkled all over the code, that's probably worthwhile as well.
>>>>
>>>> 2) I'm beginning to think that this sort of thing should be done in a
>>>> branch, thoroughly tested, and then merged in. I'm not sure what the
>>>> thinking was here as-is. If someone moves a file elsewhere, are they
>>>> thinking "this needs to be done, we'll just do it, see what breaks and
>>>> fix it later", or are they just unaware of item #1 above and have no
>>>> idea what code might be relying on that file being there in the right
>>>> place. I would hope that someone would at least do a grep through the
>>>> code or something. This is perhaps a subtle thing because changing
>>>> paths doesn't necessarily cause builds to break or some tests to fail.
>>>>
>>>> Sorry for the rant. I understand that we didn't have packages at all a
>>>> few weeks ago, so this should be a step up. Ideally, changing the way
>>>> something is packaged shouldn't require changes to the code, or at
>>>> worst minimal changes, but we're not there and I think it caught us
>>>> off guard. Perhaps I don't understand all of the subtleties, but the
>>>> lesson I take from this is that we should have first made packaging
>>>> work as it did before, instead of introducing sweeping changes at the
>>>> same time.

Re: name change of cloud- to cloudstack- explosion

[Rohit Yadav <bh...@apache.org>]

Let me rant again: A symlink would just have worked if they dislike
cloud-, cloudstack- prefix opens just another can of worms.

Regards.

On Thu, Feb 14, 2013 at 10:56 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> Ideally, we could just install our whole tree somewhere, and leave it
> up to the packagers to do system specific stuff. For example, with
> CentOS we could dump a whole tree in /opt, then in the packaging
> create symlinks to /opt/whatever in the places an admin would expect
> to see config files, copy an init script that starts cloudstack in the
> location we chose for the tree, etc.
>
> As far as the recent fixes, I've fixed things like:
>
> change places where cloud-agent and cloud-setup-agent scripts were
> called in java code to use cloudstack-agent
>
> various things that were looking in /var/lib/cloud to /var/cloudstack,
> fixed things such as ssh key generation and unable to log in with
> default admin/password upon install
>
> change the path to systemvm.iso in the kvm agent from /usr/lib64/cloud
> to /usr/share/cloudstack-common  (this issue was only apparent after
> noticing that dhcp wasn't working on the routers, then noticing that
> the systemvm.iso was out of date, then finding the path was bad)
>
> Were being told to install vhd-util in /usr/share/cloud rather than
> /usr/share/cloudstack on mgt server
>
> On Thu, Feb 14, 2013 at 10:06 AM, Alex Huang <Al...@citrix.com> wrote:
>> Marcus,
>>
>> It's a good rant!  I brought up the whole download an archive and run it and partly because I figure there's problems like this since CloudStack has always been installed one way and one way only.
>>
>> But I don't know the specifics.  Do you have any examples you can share?  That way we can show people why this should not be done.  Will be a good lesson for us all.  Also if you show some examples, other developers can help in locating similar problems so you don't have to do all the work alone?
>>
>> Thanks!
>> --Alex
>>
>>> -----Original Message-----
>>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> Sent: Thursday, February 14, 2013 8:38 AM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: name change of cloud- to cloudstack- explosion
>>>
>>>  I (and others) have been working through a ton of issues revolving
>>> around this name change, and while I think it's a good move in
>>> general, I wanted to bring up two points to consider for similar
>>> changes in the future. I've been primarily working in the 4.1 branch.
>>>
>>> 1)  This is showing me just how tied the code is to the installation.
>>> Mostly on the agent side, but a bit on the server side as well.
>>> Someone mentioned in the future being able to download an archive and
>>> just run it wherever you extract it, I think there's going to be a bit
>>> of work there in making paths compatible.  It has also been mentioned
>>> that we should make some globals such that we don't have paths
>>> sprinkled all over the code, that's probably worthwhile as well.
>>>
>>> 2) I'm beginning to think that this sort of thing should be done in a
>>> branch, thoroughly tested, and then merged in. I'm not sure what the
>>> thinking was here as-is. If someone moves a file elsewhere, are they
>>> thinking "this needs to be done, we'll just do it, see what breaks and
>>> fix it later", or are they just unaware of item #1 above and have no
>>> idea what code might be relying on that file being there in the right
>>> place. I would hope that someone would at least do a grep through the
>>> code or something. This is perhaps a subtle thing because changing
>>> paths doesn't necessarily cause builds to break or some tests to fail.
>>>
>>> Sorry for the rant. I understand that we didn't have packages at all a
>>> few weeks ago, so this should be a step up. Ideally, changing the way
>>> something is packaged shouldn't require changes to the code, or at
>>> worst minimal changes, but we're not there and I think it caught us
>>> off guard. Perhaps I don't understand all of the subtleties, but the
>>> lesson I take from this is that we should have first made packaging
>>> work as it did before, instead of introducing sweeping changes at the
>>> same time.

Re: name change of cloud- to cloudstack- explosion

[Marcus Sorensen <sh...@gmail.com>]

Ideally, we could just install our whole tree somewhere, and leave it
up to the packagers to do system specific stuff. For example, with
CentOS we could dump a whole tree in /opt, then in the packaging
create symlinks to /opt/whatever in the places an admin would expect
to see config files, copy an init script that starts cloudstack in the
location we chose for the tree, etc.

As far as the recent fixes, I've fixed things like:

change places where cloud-agent and cloud-setup-agent scripts were
called in java code to use cloudstack-agent

various things that were looking in /var/lib/cloud to /var/cloudstack,
fixed things such as ssh key generation and unable to log in with
default admin/password upon install

change the path to systemvm.iso in the kvm agent from /usr/lib64/cloud
to /usr/share/cloudstack-common  (this issue was only apparent after
noticing that dhcp wasn't working on the routers, then noticing that
the systemvm.iso was out of date, then finding the path was bad)

Were being told to install vhd-util in /usr/share/cloud rather than
/usr/share/cloudstack on mgt server

On Thu, Feb 14, 2013 at 10:06 AM, Alex Huang <Al...@citrix.com> wrote:
> Marcus,
>
> It's a good rant!  I brought up the whole download an archive and run it and partly because I figure there's problems like this since CloudStack has always been installed one way and one way only.
>
> But I don't know the specifics.  Do you have any examples you can share?  That way we can show people why this should not be done.  Will be a good lesson for us all.  Also if you show some examples, other developers can help in locating similar problems so you don't have to do all the work alone?
>
> Thanks!
> --Alex
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Thursday, February 14, 2013 8:38 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: name change of cloud- to cloudstack- explosion
>>
>>  I (and others) have been working through a ton of issues revolving
>> around this name change, and while I think it's a good move in
>> general, I wanted to bring up two points to consider for similar
>> changes in the future. I've been primarily working in the 4.1 branch.
>>
>> 1)  This is showing me just how tied the code is to the installation.
>> Mostly on the agent side, but a bit on the server side as well.
>> Someone mentioned in the future being able to download an archive and
>> just run it wherever you extract it, I think there's going to be a bit
>> of work there in making paths compatible.  It has also been mentioned
>> that we should make some globals such that we don't have paths
>> sprinkled all over the code, that's probably worthwhile as well.
>>
>> 2) I'm beginning to think that this sort of thing should be done in a
>> branch, thoroughly tested, and then merged in. I'm not sure what the
>> thinking was here as-is. If someone moves a file elsewhere, are they
>> thinking "this needs to be done, we'll just do it, see what breaks and
>> fix it later", or are they just unaware of item #1 above and have no
>> idea what code might be relying on that file being there in the right
>> place. I would hope that someone would at least do a grep through the
>> code or something. This is perhaps a subtle thing because changing
>> paths doesn't necessarily cause builds to break or some tests to fail.
>>
>> Sorry for the rant. I understand that we didn't have packages at all a
>> few weeks ago, so this should be a step up. Ideally, changing the way
>> something is packaged shouldn't require changes to the code, or at
>> worst minimal changes, but we're not there and I think it caught us
>> off guard. Perhaps I don't understand all of the subtleties, but the
>> lesson I take from this is that we should have first made packaging
>> work as it did before, instead of introducing sweeping changes at the
>> same time.

RE: name change of cloud- to cloudstack- explosion

[Alex Huang <Al...@citrix.com>]

Marcus,

It's a good rant!  I brought up the whole download an archive and run it and partly because I figure there's problems like this since CloudStack has always been installed one way and one way only.  

But I don't know the specifics.  Do you have any examples you can share?  That way we can show people why this should not be done.  Will be a good lesson for us all.  Also if you show some examples, other developers can help in locating similar problems so you don't have to do all the work alone?

Thanks!
--Alex

> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Thursday, February 14, 2013 8:38 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: name change of cloud- to cloudstack- explosion
> 
>  I (and others) have been working through a ton of issues revolving
> around this name change, and while I think it's a good move in
> general, I wanted to bring up two points to consider for similar
> changes in the future. I've been primarily working in the 4.1 branch.
> 
> 1)  This is showing me just how tied the code is to the installation.
> Mostly on the agent side, but a bit on the server side as well.
> Someone mentioned in the future being able to download an archive and
> just run it wherever you extract it, I think there's going to be a bit
> of work there in making paths compatible.  It has also been mentioned
> that we should make some globals such that we don't have paths
> sprinkled all over the code, that's probably worthwhile as well.
> 
> 2) I'm beginning to think that this sort of thing should be done in a
> branch, thoroughly tested, and then merged in. I'm not sure what the
> thinking was here as-is. If someone moves a file elsewhere, are they
> thinking "this needs to be done, we'll just do it, see what breaks and
> fix it later", or are they just unaware of item #1 above and have no
> idea what code might be relying on that file being there in the right
> place. I would hope that someone would at least do a grep through the
> code or something. This is perhaps a subtle thing because changing
> paths doesn't necessarily cause builds to break or some tests to fail.
> 
> Sorry for the rant. I understand that we didn't have packages at all a
> few weeks ago, so this should be a step up. Ideally, changing the way
> something is packaged shouldn't require changes to the code, or at
> worst minimal changes, but we're not there and I think it caught us
> off guard. Perhaps I don't understand all of the subtleties, but the
> lesson I take from this is that we should have first made packaging
> work as it did before, instead of introducing sweeping changes at the
> same time.