Suggestions and pitfalls for a development/deployment setup.

[Walter Hop <su...@walter.transip.nl>]

Hi all,

I am currently designing a SVN setup to use for our website and backend.
This is the first time I am going to use SVN for a serious project, so
I'd like to get it right the first time! I hope somebody with experience
can give me some comments on the viability and possible pitfalls of a
setup like this. I'll try to be as brief as possible!

My goals:
- Develop on a test server, and do controlled deployments to the
production server.
- Do periodic releases where all changes from the test server are merged
into a production release.
- Support for doing "hotfixes" on the production server and backporting
them to the development machine; this should be as easy and painless as
possible.

I was thinking of the following:
- Set up a SVN repository with a trunk and a release branch.
- Check out a local copy of the trunk on the test server.
- Check out a local copy of the release branch on the production server.
- Development use cases:
  1. the developer works on local copy on the test server, and commits
  this to the trunk.
  2. when an urgent bug is noticed that cannot wait for a release, work
  on local copy on the production server, and commit this (changes go
  into the release branch), then merge these changes back into the trunk.
- Release use case: merge all changes from trunk into the release
  branch, then do an 'svn update' on the production server.

Possible problems: 

- The various scripts and libraries under version control live in
various locations on the production/test server. For instance, some
files live in /usr/local/, some go into /www, some in /home et cetera.
It seems I should make subdirectories in the repository, checkout the
relevant parts of the repository at all the root points, and make a
script to walk through all these directories on the production/test
server and do a 'svn update' there. But can I also do the same for a
commit, keeping all the created changes in one singular commit (instead
of executing a 'svn ci' for every dir, creating several commits)?

- I would like to have developers use their own login/password when
committing. But at the same time, they may be working from the
test/production servers which always have an existing local copy (not
checked out by the developer account itself). Someone gave me the
suggestion to have the test/production servers checked out anonymously,
so users would then be asked for their login while committing. Will this
work? Is it also possible with a non-anonymously created local copy (for
instance with a 'testserver' account)?

- Merging changes to a different branch (releasing or backporting a
hotfix) seems to require knowing the revision number of the first change
after the previous merge. I think it's probably easiest to create a
merge-script which merges in the specified direction and then stores the
resulting revision number in a text file, so that the next merge in this
direction will be from this point. So I'd have to keep track of two
revision numbers for the latest trunk->release merge and the latest
release->trunk merge. Is my thinking on this correct?

- Are there any pitfalls in doing this kind of setup, or is there an
easier way to go abou this than the scenario?

Sigh. I guess this message is long after all. Anyway, I've tried to
google a bit for this kind of scenario, but I wasn't very succesful. Any
tips would be welcome!

Kind regards,
Walter Hop
Transip BV

-- 
  Transip BV | http://www.transip.nl/
  Hoogwaardige Innovatie | Aangename Zekerheid

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: web application deployment with(out) subversion (was: Suggestions and pitfalls for a development/deployment setup.)

["Daniele P." <da...@interline.it>]

On Friday 08 September 2006 23:49, Walter Hop wrote:
> Hi all,
>
> I am currently designing a SVN setup to use for our website and backend.
> This is the first time I am going to use SVN for a serious project, so
> I'd like to get it right the first time! I hope somebody with experience
> can give me some comments on the viability and possible pitfalls of a
> setup like this. I'll try to be as brief as possible!

Remember that subversion isn't designed to do "application deployment".
So it's a bit off-topic.

> My goals:
> 1) Develop on a test server, and do controlled deployments to the
> production server.
> 2) Do periodic releases where all changes from the test server are merged
> into a production release.
> 3) Support for doing "hotfixes" on the production server and backporting
> them to the development machine; this should be as easy and painless as
> possible.

We share the same goals, but we have other limitations:
  a) the branwith between the repository and the live server is limited
  b) the live site uses a database and you can't merge the development/test
     database with the live database, so the database modifications need to
     be done on the live site and then deployed on development/test server.
     (actually using dump + rsync)
  c) we use some external libraries but sometimes we need to fix bugs in
     the external libraries, so we import those libraries in an external
     subversion repository to get things less coupled
  d) some files are dynamically generated by the application, and we have to
     sync them without using subversion (using rsync)

In my experience you should forgot the goal number 3. Use a test server and
some script to write the glue for deploying to the live site starting from
the test server, and a script for sync the live database with the
developement/test database.

> - The various scripts and libraries under version control live in
> various locations on the production/test server. For instance, some
> files live in /usr/local/, some go into /www, some in /home et cetera.
> It seems I should make subdirectories in the repository, checkout the
> relevant parts of the repository at all the root points, and make a
> script to walk through all these directories on the production/test
> server and do a 'svn update' there. But can I also do the same for a
> commit, keeping all the created changes in one singular commit (instead
> of executing a 'svn ci' for every dir, creating several commits)?

No, AFAIKS the directories that are going to be checked in should have a 
common parent that must be a directory under version control.
And what about user/permission management? Are you going to checkout all the
files as root?
By the way somethin seems wrong in your application layout, you shouldn't
use /home for your application files.

> - I would like to have developers use their own login/password when
> committing. But at the same time, they may be working from the
> test/production servers which always have an existing local copy (not
> checked out by the developer account itself). Someone gave me the
> suggestion to have the test/production servers checked out anonymously,
> so users would then be asked for their login while committing. Will this
> work? Is it also possible with a non-anonymously created local copy (for
> instance with a 'testserver' account)?

No, remind that the .svn directories belong to the user who made the co,
and that subversion place a .subversion directory in the user home directory 
to remember user settings, sometimes too many settings (clear text password, 
for example).

> - Merging changes to a different branch (releasing or backporting a
> hotfix) seems to require knowing the revision number of the first change
> after the previous merge. I think it's probably easiest to create a
> merge-script which merges in the specified direction and then stores the
> resulting revision number in a text file, so that the next merge in this
> direction will be from this point. So I'd have to keep track of two
> revision numbers for the latest trunk->release merge and the latest
> release->trunk merge. Is my thinking on this correct?

Yes, usually you should keep note of merging in the log message.
There is also a contrib script that helps, search for svnmerge.py

> - Are there any pitfalls in doing this kind of setup, or is there an
> easier way to go abou this than the scenario?

Start thinking about adding a test server and custom script(s) in your 
deployment process.

> Sigh. I guess this message is long after all. Anyway, I've tried to
> google a bit for this kind of scenario, but I wasn't very succesful. Any
> tips would be welcome!

Yes, because it's really complicated and often application specific.
You could start looking at best pratices for well established 
application/framework like ruby on rails, django etc..
Maybe others could share their best pratices, or simply arguments against 
using subversion for application deployment.

Daniele

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Suggestions and pitfalls for a development/deployment setup.

[Grant Rettke <gr...@acm.org>]

Quoting Walter Hop <su...@walter.transip.nl>:

> Hi all,
>
> I am currently designing a SVN setup to use for our website and backend.
> This is the first time I am going to use SVN for a serious project, so
> I'd like to get it right the first time! I hope somebody with experience
> can give me some comments on the viability and possible pitfalls of a
> setup like this. I'll try to be as brief as possible!
>
> My goals:
> - Develop on a test server, and do controlled deployments to the
> production server.
> - Do periodic releases where all changes from the test server are merged
> into a production release.
> - Support for doing "hotfixes" on the production server and backporting
> them to the development machine; this should be as easy and painless as
> possible.

One option would be to set up a DEV, TEST, QA, and PROD machine.

Checkout your (development) trunk to DEV and do all of your work there. 
Do all of your trunk testing on the TEST box.

Checkout your release branch to QA. *Export* your release branch to PROD.
Then, rsync PROD to QA. All of the changes you make on PROD would get captured
in your release branch.

This sounds way too easy...it probably isn't.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Suggestions and pitfalls for a development/deployment setup.

[Ryan Schmidt <su...@ryandesign.com>]

On Sep 8, 2006, at 23:49, Walter Hop wrote:

> I am currently designing a SVN setup to use for our website and  
> backend.
> This is the first time I am going to use SVN for a serious project, so
> I'd like to get it right the first time! I hope somebody with  
> experience
> can give me some comments on the viability and possible pitfalls of a
> setup like this. I'll try to be as brief as possible!

> - Are there any pitfalls in doing this kind of setup, or is there an
> easier way to go abou this than the scenario?

Hi Walter. I'll give you some answers based on what we did in the web  
development shop where I worked. I can't say whether it's the best  
strategy, but it's at least not completely horrible.


> My goals:
> - Develop on a test server, and do controlled deployments to the
> production server.
> - Do periodic releases where all changes from the test server are  
> merged
> into a production release.
> - Support for doing "hotfixes" on the production server and  
> backporting
> them to the development machine; this should be as easy and  
> painless as
> possible.
>
> I was thinking of the following:
> - Set up a SVN repository with a trunk and a release branch.
> - Check out a local copy of the trunk on the test server.
> - Check out a local copy of the release branch on the production  
> server.

You can point your web server directly at a working copy, yes, but be  
aware that "svn up" is neither instant nor atomic. This means that if  
you update the working copy, between the time the update starts and  
the time it finishes, your web site is in an inconsistent state, and  
may blow up, depending on your programming. And if there is a network  
problem such that the update does not complete, then your working  
copy may be half updated and half not. To avoid these problems, we  
used a lengthier process which is described here:

http://svn.haxx.se/users/archive-2005-05/1117.shtml

In most of our projects we did indeed have a trunk and a release  
branch (say "1.0"). Smaller changes that should go into 1.0 right  
away could be made in either the trunk or the branch and then merged  
to the other place. Bigger changes that would wait for version 1.1  
would happen only in trunk. When the team mutually agrees that 1.1 is  
ready, trunk is copied to a new branch "1.1", it is pushed to the  
production server, and the 1.0 branch is moved to an "old_branches"  
directory so everyone knows not to use it anymore [1].

But the reality of doing customer-driven projects, which is what we  
did, is that most changes the customer asks for are "really  
important" changes that must go online immediately, so the vast  
majority of changes ended up having to be done in both places (trunk  
and branch) and a release cut immediately. The merging got tedious.  
I'm now of the opinion that one should only have a trunk, and cut  
releases directly from that. This eliminates the need to merge in the  
vast majority of cases (of our cases anyway). In the rare  
circumstance that a larger feature is being developed that will take  
some time and can't go online right away (perhaps because the  
customer hasn't paid for it yet), you can either then mutually agree  
with the team to cut a release branch before the development of this  
new feature starts on trunk, or else you can develop it in a feature  
branch and then merge it back to trunk when it's ready to go online.

To be honest, the frequency of our releases also made our release  
deployment process tedious. (Create a new tag of the release branch.  
ssh to the development server. cd to the project directory. Run the  
upgrade script. Remove the old symlink and add the new one.) Had I  
stayed at the company I likely would have investigated ways of  
automating this. Maybe do just have a single release branch like you  
propose. Or even define that the trunk is the release branch. Make it  
clear to developers that anything committed to this sacred release  
branch will appear on the production site immediately. Write a post- 
commit hook script which makes this possible (which ssh's to the  
production server and updates the production working copy, or else  
does the export and symlink switcheroo). The only problem with this  
idea is database updates, which we never attempted to automate. If  
there were database updates that needed to happen for a particular  
feature, we either did this on the live server immediately, if they  
posed no danger to the rest of the system (for example, adding a new  
table, or adding a new column or index in an existing table), or we  
wrote down the necessary SQL statements in our ticket system and ran  
these at the time that we deployed a new version of the site. You'd  
need to think about how you want to handle this situation.


> - Development use cases:
>   1. the developer works on local copy on the test server, and commits
>   this to the trunk.
>   2. when an urgent bug is noticed that cannot wait for a release,  
> work
>   on local copy on the production server, and commit this (changes go
>   into the release branch), then merge these changes back into the  
> trunk.

The developers should not work in the central working copies on the  
test server or the production server. They should each work in their  
own working copies in their home directories. Your web site should be  
programmed so that it works regardless of where it is located on  
disk. To avoid having to configure a web server and database server  
on each developer's machine, we stored all our working copies in the  
home directories on the central development server and accessed the  
files via Samba. This put a bit of strain on the development server,  
but if your working copies are small or you have few developers or  
your server has a lot of RAM then it should be ok. Some things were  
still shared between developers: the MySQL database, for example, and  
the directories where photos and things were uploaded. The photo  
directory was located in a central place on the server, and its  
location defined in the web site code as an absolute path in a  
constant so that it could be found regardless of where the working  
copy is.

The point is that during development you break stuff, often  
deliberately. If Bob is fixing problems in the sitewide config file  
and to do this he needs to break it for about an hour, you do not  
want this breakage to occur in the central working copy on the  
development server because nobody else could work on the project  
during that time. You want this to occur in a private environment set  
up just for Bob; that would be Bob's own working copy. When Bob is  
done fixing whatever problem it was and has repaired the intermediate  
breakage, then he can commit his changes, which can trigger a post- 
commit script which updates the central development working copy.

If every developer has their own working copy, you don't even  
particularly need a central development working copy, because nobody  
will be developing there, and anybody who wants to see what the  
system looks like can just look in their own working copy after  
updating it. The reason we still kept a central development working  
copy was for the nontechnical people who wanted to look at the  
current development state of the site -- the managers. Managers (at  
least ours) don't want to learn about "svn checkout" and "svn update"  
and they certainly don't want to wait seconds or minutes for those  
commands to complete. They want a simple well-defined URL where they  
can go to always see the current state of the project, so that's what  
we gave them.


> - Release use case: merge all changes from trunk into the release
>   branch, then do an 'svn update' on the production server.

You can do this if you want to always only have a single release  
branch, or you can do as we did and make a new release branch each  
time a major release is cut. This also helps if you say things like  
"feature foo is introduced in version 1.1" and you later want to see  
what the system looked like without feature foo. It's easy -- just  
look at the 1.0 branch. (Not that we wanted to do this often, but if  
we did, it would be easier to look in the 1.0 directory than to look  
through the log of trunk to try to figure out when that feature was  
created -- particularly with some of the nonspectacular commit  
messages some of our developers have written. :))


> Possible problems:
>
> - The various scripts and libraries under version control live in
> various locations on the production/test server. For instance, some
> files live in /usr/local/, some go into /www, some in /home et cetera.
> It seems I should make subdirectories in the repository, checkout the
> relevant parts of the repository at all the root points, and make a
> script to walk through all these directories on the production/test
> server and do a 'svn update' there. But can I also do the same for a
> commit, keeping all the created changes in one singular commit  
> (instead
> of executing a 'svn ci' for every dir, creating several commits)?

If you create separate working copies for the separate parts of this  
system, then you will need to commit changes to them separately as  
separate revisions. This doesn't sound optimal.

You should design your system so that it can be checked out as a  
single cohesive working copy in a single location. If for some reason  
you absolutely cannot change the fact that different parts of it must  
appear in different places, then perhaps you can at least still check  
out everything in one location, and then install symlinks at the  
various necessary paths that point to where the working copy really is.


> - I would like to have developers use their own login/password when
> committing. But at the same time, they may be working from the
> test/production servers which always have an existing local copy (not
> checked out by the developer account itself). Someone gave me the
> suggestion to have the test/production servers checked out  
> anonymously,
> so users would then be asked for their login while committing. Will  
> this
> work? Is it also possible with a non-anonymously created local copy  
> (for
> instance with a 'testserver' account)?

If each developer has their own working copy as they should, then  
each working copy will be owned by the respective developer. The  
developer will either ssh to the development server with their own  
login or access the files via Samba from their own desktop, and in  
the Subversion directory in their home directory, their username will  
be cached so that when they commit changes, their username will be used.


> - Merging changes to a different branch (releasing or backporting a
> hotfix) seems to require knowing the revision number of the first  
> change
> after the previous merge. I think it's probably easiest to create a
> merge-script which merges in the specified direction and then  
> stores the
> resulting revision number in a text file, so that the next merge in  
> this
> direction will be from this point. So I'd have to keep track of two
> revision numbers for the latest trunk->release merge and the latest
> release->trunk merge. Is my thinking on this correct?

The script is called svnmerge.py; you can Google for that to learn more.



[1] We could also "svn rm" the 1.0 branch but then it's more  
difficult to find it in the repository later should we need to do  
that. Since "svn rm" doesn't reclaim any disk space its only purpose  
would be to make the directory listing of the branches directory more  
manageable, so moving it to an old_branches directory solves this  
problem just as well and still lets us find it easily later.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org