You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Tony Butt <To...@cea.com.au> on 2011/07/06 07:20:27 UTC
Trials with memcached
We are running subversion 1.6.17 on a vmware hosted server. We recently
reconfigured the server to give 4 virtual CPUs (up from 1), and a
significant amount of memory.
In order to spruce up our performance a little, I looked into the use of
memcached with subversion again, found the correct config parameter, and
set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
mechanism is http (of course). The client used is running Ubuntu 11.04,
and svn commandline (1.6.17 also)
The results were interesting, to say the least.
Checkout of a tree, about 250M in size:
Without memcached, 1 1/2 to 2 minutes, varies with server load
With memcached, 12 minutes (!)
Update of the same tree,
Without memcached, 9 seconds
With memcached, 14 seconds - repeated several times, similar results.
I am not sure what anyone else's experience is, but we will not be
enabling memcached for subversion any time soon.
--
Tony Butt <tj...@cea.com.au>
CEA Technologies
Re: Trials with memcached
Posted by Tony Butt <To...@cea.com.au>.
On Fri, 2011-07-08 at 02:59 +0300, Daniel Shahaf wrote:
> This doesn't address memcached directly, but there has been a /lot/ of
> work on server-side optimization and caching in 1.7 (also for
> non-memcached-backed caches).
>
> http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
>
> You might want to take 1.7.0-alpha3 for a spin...
>
Probably don't want to do that.
We are in a commercial environment, with some 20 developers relying on
subversion - not the time for an alpha release.
We are actually happy with the current performance, particularly since
the load of other tasks on the server (backups, opengrok indexing of the
repo for instance) is now shared by 4 processors instead of 1. I was
reviewing the overall configuration, and we already use memcached to
support ReviewBoard for code reviews.
We will be particularly interested in server side performance
improvements when 1.7.0 is released - we have home grown build
dependency tools that are sometimes heavy on the repository usage -
these will be difficult to upgrade to 1.7.0, but if there are uncoupled
server performance improvements, the server upgrade is trivial by
comparison.
Thanks,
Tony Butt
> Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> > We are running subversion 1.6.17 on a vmware hosted server. We recently
> > reconfigured the server to give 4 virtual CPUs (up from 1), and a
> > significant amount of memory.
> >
> >
> > In order to spruce up our performance a little, I looked into the use of
> > memcached with subversion again, found the correct config parameter, and
> > set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> > mechanism is http (of course). The client used is running Ubuntu 11.04,
> > and svn commandline (1.6.17 also)
> >
> > The results were interesting, to say the least.
> >
> > Checkout of a tree, about 250M in size:
> >
> > Without memcached, 1 1/2 to 2 minutes, varies with server load
> > With memcached, 12 minutes (!)
> >
> > Update of the same tree,
> > Without memcached, 9 seconds
> > With memcached, 14 seconds - repeated several times, similar results.
> >
> > I am not sure what anyone else's experience is, but we will not be
> > enabling memcached for subversion any time soon.
> > --
> > Tony Butt <tj...@cea.com.au>
> > CEA Technologies
> >
--
Tony Butt <tj...@cea.com.au>
CEA Technologies
Re: Trials with memcached
Posted by Tony Butt <To...@cea.com.au>.
On Thu, 2011-07-07 at 17:31 -0700, Geoff Hoffman wrote:
Tony -
Strange results to be sure. You probably thought of all this, but...
Did you check Memcached is working correctly without Subversion?
Did you check the results of checking out or updating the 2nd or 3rd time?
In other words, it may take longer the first time because every object in the repo has to be checked for existence/expiry in the cache.
Did you check that you gave Memcached enough memory to fit the entire 250mb repo (comfortably) in RAM?
If not then memcached itself tries to use swap space?
32-bit or 64-bit VM?
As before, PAE kernel. Processor claims to be a Xeon. Not sure how to tell what the underlying VM software is doing.
I ran my test with memcached -m 512
That is, 512M memory. Checkout completed in 11 minutes - better than before, but still slower than without memcached.
memcached-tool localhost stats reports
get_hits 481964
get_misses 71527
among many other things, so the cache is certainly working (numbers were much smaller before the checkout).
Running again completed in 8m 34s, still too slow...
Tony
Did you try it on physical hardware?
On Thu, Jul 7, 2011 at 4:59 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
This doesn't address memcached directly, but there has been a /lot/ of
work on server-side optimization and caching in 1.7 (also for
non-memcached-backed caches).
http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
You might want to take 1.7.0-alpha3 for a spin...
Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> We are running subversion 1.6.17 on a vmware hosted server. We recently
> reconfigured the server to give 4 virtual CPUs (up from 1), and a
> significant amount of memory.
>
>
> In order to spruce up our performance a little, I looked into the use of
> memcached with subversion again, found the correct config parameter, and
> set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> mechanism is http (of course). The client used is running Ubuntu 11.04,
> and svn commandline (1.6.17 also)
>
> The results were interesting, to say the least.
>
> Checkout of a tree, about 250M in size:
>
> Without memcached, 1 1/2 to 2 minutes, varies with server load
> With memcached, 12 minutes (!)
>
> Update of the same tree,
> Without memcached, 9 seconds
> With memcached, 14 seconds - repeated several times, similar results.
>
> I am not sure what anyone else's experience is, but we will not be
> enabling memcached for subversion any time soon.
> --
> Tony Butt <tj...@cea.com.au>
> CEA Technologies
>
--
Tony Butt <tj...@cea.com.au>
CEA Technologies
Re: Trials with memcached
Posted by Tony Butt <To...@cea.com.au>.
On Thu, 2011-07-07 at 17:31 -0700, Geoff Hoffman wrote:
Tony -
Strange results to be sure. You probably thought of all this, but...
Did you check Memcached is working correctly without Subversion?
It seems to be - reviewboard works OK, and running memcached-tool shows cache hits, etc, as expected.
Did you check the results of checking out or updating the 2nd or 3rd time?
I ran the update several times, it was consistent.
I did not do the original checkout more than once.
In other words, it may take longer the first time because every object in the repo has to be checked for existence/expiry in the cache.
Yes - memcached-util showed plenty of cache hits after the initial checkout.
Did you check that you gave Memcached enough memory to fit the entire 250mb repo (comfortably) in RAM?
Configured memcached with 256M memory. Might not be enough. I will try with 512M, and watch the performance on the server.
Our entire repository is several gigabytes in size, checkout sizes might be around 500M or so. If populating the cache causes a 6x slowdown, it is probably not worth it for 20 developers anyway.
The issue there is that it blows out our 'worst case' timing.
If not then memcached itself tries to use swap space?
32-bit or 64-bit VM?
PAE kernel,
Did you try it on physical hardware?
No - we have a test machine that runs on physical hardware, I will try it there when I have time.
On Thu, Jul 7, 2011 at 4:59 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
This doesn't address memcached directly, but there has been a /lot/ of
work on server-side optimization and caching in 1.7 (also for
non-memcached-backed caches).
http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
You might want to take 1.7.0-alpha3 for a spin...
Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> We are running subversion 1.6.17 on a vmware hosted server. We recently
> reconfigured the server to give 4 virtual CPUs (up from 1), and a
> significant amount of memory.
>
>
> In order to spruce up our performance a little, I looked into the use of
> memcached with subversion again, found the correct config parameter, and
> set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> mechanism is http (of course). The client used is running Ubuntu 11.04,
> and svn commandline (1.6.17 also)
>
> The results were interesting, to say the least.
>
> Checkout of a tree, about 250M in size:
>
> Without memcached, 1 1/2 to 2 minutes, varies with server load
> With memcached, 12 minutes (!)
>
> Update of the same tree,
> Without memcached, 9 seconds
> With memcached, 14 seconds - repeated several times, similar results.
>
> I am not sure what anyone else's experience is, but we will not be
> enabling memcached for subversion any time soon.
> --
> Tony Butt <tj...@cea.com.au>
> CEA Technologies
>
--
Tony Butt <tj...@cea.com.au>
CEA Technologies
Re: Trials with memcached
Posted by Geoff Hoffman <gh...@cardinalpath.com>.
Tony -
Strange results to be sure. You probably thought of all this, but...
Did you check Memcached is working correctly without Subversion?
Did you check the results of checking out or updating the 2nd or 3rd time?
In other words, it may take longer the first time because every object in
the repo has to be checked for existence/expiry in the cache.
Did you check that you gave Memcached enough memory to fit the entire 250mb
repo (comfortably) in RAM?
If not then memcached itself tries to use swap space?
32-bit or 64-bit VM?
Did you try it on physical hardware?
On Thu, Jul 7, 2011 at 4:59 PM, Daniel Shahaf <d....@daniel.shahaf.name>wrote:
> This doesn't address memcached directly, but there has been a /lot/ of
> work on server-side optimization and caching in 1.7 (also for
> non-memcached-backed caches).
>
>
> http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
>
> You might want to take 1.7.0-alpha3 for a spin...
>
> Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> > We are running subversion 1.6.17 on a vmware hosted server. We recently
> > reconfigured the server to give 4 virtual CPUs (up from 1), and a
> > significant amount of memory.
> >
> >
> > In order to spruce up our performance a little, I looked into the use of
> > memcached with subversion again, found the correct config parameter, and
> > set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> > mechanism is http (of course). The client used is running Ubuntu 11.04,
> > and svn commandline (1.6.17 also)
> >
> > The results were interesting, to say the least.
> >
> > Checkout of a tree, about 250M in size:
> >
> > Without memcached, 1 1/2 to 2 minutes, varies with server load
> > With memcached, 12 minutes (!)
> >
> > Update of the same tree,
> > Without memcached, 9 seconds
> > With memcached, 14 seconds - repeated several times, similar results.
> >
> > I am not sure what anyone else's experience is, but we will not be
> > enabling memcached for subversion any time soon.
> > --
> > Tony Butt <tj...@cea.com.au>
> > CEA Technologies
> >
>
Testing 1.7 pre-releases Re: Trials with memcached
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Stefan Sperling wrote on Fri, Jul 08, 2011 at 11:13:05 +0200:
> Doing pre-release testing is a great service to the community.
> For us, it is a lot easier to handle problems before the release,
> and we can respond to problem reports a lot quicker.
>
> After release we are bound by compatibility guarantees that
> sometimes make bug fixing a little more painful. And you might
> have to wait for weeks for enough fixes to accumulate before
> a new patch release is issued that addresses your particular problem.
>
> You will likely upgrade your setup to Subversion 1.7 eventually.
> Any problems that show up which are specific to your setup will
> have to be dealt with at that point. It's easier to spot and fix
> problems now, while you're not running 1.7 in production and while
> we're still moving towards the 1.7.0 release.
Re: Trials with memcached
Posted by Stefan Sperling <st...@elego.de>.
On Fri, Jul 08, 2011 at 01:56:07PM +1000, Tony Butt wrote:
> On Fri, 2011-07-08 at 03:58 +0300, Daniel Shahaf wrote:
> > Tony Butt wrote on Fri, Jul 08, 2011 at 10:41:43 +1000:
> > > Probably don't want to do that.
> > > We are in a commercial environment, with some 20 developers relying on
> > > subversion - not the time for an alpha release.
> > >
> >
> > I wasn't suggesting that you upgrade your production server!
> I didn't really think you would be :-)
> >
> > Just that you install the alpha in a test environment to see if it
> > improves the situation for you. (or if there is anything you see that
> > requires modification /before/ the release --- before compatibility
> > promises apply --- as in eg issue #3952)
> >
> My available test server also syncs the production repository to itself
> as a hot spare. I am probably brave enough to install 1.7.0-alpha3 on
> that, so long as there are no issues syncing from 1.6.17 to 1.7.0
Hi Tony,
Doing pre-release testing is a great service to the community.
For us, it is a lot easier to handle problems before the release,
and we can respond to problem reports a lot quicker.
After release we are bound by compatibility guarantees that
sometimes make bug fixing a little more painful. And you might
have to wait for weeks for enough fixes to accumulate before
a new patch release is issued that addresses your particular problem.
You will likely upgrade your setup to Subversion 1.7 eventually.
Any problems that show up which are specific to your setup will
have to be dealt with at that point. It's easier to spot and fix
problems now, while you're not running 1.7 in production and while
we're still moving towards the 1.7.0 release.
One good way of testing is to make a somewhat recent (say, one week
old) repository snapshot available though a 1.7 server reachable at
a temporary URL. Then ask a few developers to put aside half an hour
to try to use this temporary server (with fresh working copies) and perform
whatever steps they performed on the real server again on this new server
(checkout, update, commit, merge, ...). The developers can use either
1.6 or 1.7 clients to do this. Both should "just work".
You don't need a separate server machine if you can install
Subversion 1.6 and Subversion 1.7 in parallel (this may not be easy
depending on what kind of package management you are using).
You can start the 1.7 server on a different port (e.g. 8080),
and point it at the repository snapshot which can be located anywhere
in the machine's filesystem. This way your production setup is not
affected at all, apart from sharing CPU and memory resources with the
test setup.
But if you cannot install 1.6 and 1.7 at the same time, you should
use a separate machine for testing. Please do not install and use 1.7
on a Subversion server which you rely on, for whatever purpose (even
for backup), just yet.
You can get binaries of Subversion 1.7 alpha releases for some platforms
here: http://subversion.apache.org/packages.html#pre-release
Thanks! (And if you cannot test now, that's fine -- it just means you
will get to test later, when it will be less convenient ;)
Re: Trials with memcached
Posted by Tony Butt <To...@cea.com.au>.
On Fri, 2011-07-08 at 03:58 +0300, Daniel Shahaf wrote:
> Tony Butt wrote on Fri, Jul 08, 2011 at 10:41:43 +1000:
> > On Fri, 2011-07-08 at 02:59 +0300, Daniel Shahaf wrote:
> > > This doesn't address memcached directly, but there has been a /lot/ of
> > > work on server-side optimization and caching in 1.7 (also for
> > > non-memcached-backed caches).
> > >
> > > http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
> > >
> > > You might want to take 1.7.0-alpha3 for a spin...
> > >
> > Probably don't want to do that.
> > We are in a commercial environment, with some 20 developers relying on
> > subversion - not the time for an alpha release.
> >
>
> I wasn't suggesting that you upgrade your production server!
I didn't really think you would be :-)
>
> Just that you install the alpha in a test environment to see if it
> improves the situation for you. (or if there is anything you see that
> requires modification /before/ the release --- before compatibility
> promises apply --- as in eg issue #3952)
>
My available test server also syncs the production repository to itself
as a hot spare. I am probably brave enough to install 1.7.0-alpha3 on
that, so long as there are no issues syncing from 1.6.17 to 1.7.0
I was starting to think that apache 2.2 + subversion 1.6.17 probably did
as much caching as was necessary for good performance, and that adding
in memcache to serve what is almost static data was really adding an
overhead.
Anyway...
Not really complaing about the performance of apache + subversion, just
surprised at the effect of apache + subversion + memcached, and
wondering if anyone else had any positive or negative experiences.
Thanks,
Tony
> > We are actually happy with the current performance, particularly since
> > the load of other tasks on the server (backups, opengrok indexing of the
> > repo for instance) is now shared by 4 processors instead of 1. I was
> > reviewing the overall configuration, and we already use memcached to
> > support ReviewBoard for code reviews.
> >
> > We will be particularly interested in server side performance
> > improvements when 1.7.0 is released - we have home grown build
> > dependency tools that are sometimes heavy on the repository usage -
> > these will be difficult to upgrade to 1.7.0, but if there are uncoupled
> > server performance improvements, the server upgrade is trivial by
> > comparison.
> >
> > Thanks,
> > Tony Butt
> > > Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> > > > We are running subversion 1.6.17 on a vmware hosted server. We recently
> > > > reconfigured the server to give 4 virtual CPUs (up from 1), and a
> > > > significant amount of memory.
> > > >
> > > >
> > > > In order to spruce up our performance a little, I looked into the use of
> > > > memcached with subversion again, found the correct config parameter, and
> > > > set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> > > > mechanism is http (of course). The client used is running Ubuntu 11.04,
> > > > and svn commandline (1.6.17 also)
> > > >
> > > > The results were interesting, to say the least.
> > > >
> > > > Checkout of a tree, about 250M in size:
> > > >
> > > > Without memcached, 1 1/2 to 2 minutes, varies with server load
> > > > With memcached, 12 minutes (!)
> > > >
> > > > Update of the same tree,
> > > > Without memcached, 9 seconds
> > > > With memcached, 14 seconds - repeated several times, similar results.
> > > >
> > > > I am not sure what anyone else's experience is, but we will not be
> > > > enabling memcached for subversion any time soon.
> > > > --
> > > > Tony Butt <tj...@cea.com.au>
> > > > CEA Technologies
> > > >
> >
> > --
> > Tony Butt <tj...@cea.com.au>
> > CEA Technologies
--
Tony Butt <tj...@cea.com.au>
CEA Technologies
Re: Trials with memcached
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Tony Butt wrote on Fri, Jul 08, 2011 at 13:56:07 +1000:
> On Fri, 2011-07-08 at 03:58 +0300, Daniel Shahaf wrote:
> > Tony Butt wrote on Fri, Jul 08, 2011 at 10:41:43 +1000:
> > > On Fri, 2011-07-08 at 02:59 +0300, Daniel Shahaf wrote:
> > > > This doesn't address memcached directly, but there has been a /lot/ of
> > > > work on server-side optimization and caching in 1.7 (also for
> > > > non-memcached-backed caches).
> > > >
> > > > http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
> > > >
> > > > You might want to take 1.7.0-alpha3 for a spin...
> > > >
> > > Probably don't want to do that.
> > > We are in a commercial environment, with some 20 developers relying on
> > > subversion - not the time for an alpha release.
> > >
> >
> > I wasn't suggesting that you upgrade your production server!
> I didn't really think you would be :-)
> >
> > Just that you install the alpha in a test environment to see if it
> > improves the situation for you. (or if there is anything you see that
> > requires modification /before/ the release --- before compatibility
> > promises apply --- as in eg issue #3952)
> >
> My available test server also syncs the production repository to itself
> as a hot spare. I am probably brave enough to install 1.7.0-alpha3 on
> that, so long as there are no issues syncing from 1.6.17 to 1.7.0
>
In theory: should work.
In practice: we may have introduced bugs on trunk that no one has
unearthed yet.
Re: Trials with memcached
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Tony Butt wrote on Fri, Jul 08, 2011 at 10:41:43 +1000:
> On Fri, 2011-07-08 at 02:59 +0300, Daniel Shahaf wrote:
> > This doesn't address memcached directly, but there has been a /lot/ of
> > work on server-side optimization and caching in 1.7 (also for
> > non-memcached-backed caches).
> >
> > http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
> >
> > You might want to take 1.7.0-alpha3 for a spin...
> >
> Probably don't want to do that.
> We are in a commercial environment, with some 20 developers relying on
> subversion - not the time for an alpha release.
>
I wasn't suggesting that you upgrade your production server!
Just that you install the alpha in a test environment to see if it
improves the situation for you. (or if there is anything you see that
requires modification /before/ the release --- before compatibility
promises apply --- as in eg issue #3952)
> We are actually happy with the current performance, particularly since
> the load of other tasks on the server (backups, opengrok indexing of the
> repo for instance) is now shared by 4 processors instead of 1. I was
> reviewing the overall configuration, and we already use memcached to
> support ReviewBoard for code reviews.
>
> We will be particularly interested in server side performance
> improvements when 1.7.0 is released - we have home grown build
> dependency tools that are sometimes heavy on the repository usage -
> these will be difficult to upgrade to 1.7.0, but if there are uncoupled
> server performance improvements, the server upgrade is trivial by
> comparison.
>
> Thanks,
> Tony Butt
> > Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> > > We are running subversion 1.6.17 on a vmware hosted server. We recently
> > > reconfigured the server to give 4 virtual CPUs (up from 1), and a
> > > significant amount of memory.
> > >
> > >
> > > In order to spruce up our performance a little, I looked into the use of
> > > memcached with subversion again, found the correct config parameter, and
> > > set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> > > mechanism is http (of course). The client used is running Ubuntu 11.04,
> > > and svn commandline (1.6.17 also)
> > >
> > > The results were interesting, to say the least.
> > >
> > > Checkout of a tree, about 250M in size:
> > >
> > > Without memcached, 1 1/2 to 2 minutes, varies with server load
> > > With memcached, 12 minutes (!)
> > >
> > > Update of the same tree,
> > > Without memcached, 9 seconds
> > > With memcached, 14 seconds - repeated several times, similar results.
> > >
> > > I am not sure what anyone else's experience is, but we will not be
> > > enabling memcached for subversion any time soon.
> > > --
> > > Tony Butt <tj...@cea.com.au>
> > > CEA Technologies
> > >
>
> --
> Tony Butt <tj...@cea.com.au>
> CEA Technologies
Re: Trials with memcached
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
This doesn't address memcached directly, but there has been a /lot/ of
work on server-side optimization and caching in 1.7 (also for
non-memcached-backed caches).
http://subversion.apache.org/docs/release-notes/1.7#server-performance-tuning
You might want to take 1.7.0-alpha3 for a spin...
Tony Butt wrote on Wed, Jul 06, 2011 at 15:20:27 +1000:
> We are running subversion 1.6.17 on a vmware hosted server. We recently
> reconfigured the server to give 4 virtual CPUs (up from 1), and a
> significant amount of memory.
>
>
> In order to spruce up our performance a little, I looked into the use of
> memcached with subversion again, found the correct config parameter, and
> set it up. Our server is running Ubuntu 10.04, Apache 2.2. Access
> mechanism is http (of course). The client used is running Ubuntu 11.04,
> and svn commandline (1.6.17 also)
>
> The results were interesting, to say the least.
>
> Checkout of a tree, about 250M in size:
>
> Without memcached, 1 1/2 to 2 minutes, varies with server load
> With memcached, 12 minutes (!)
>
> Update of the same tree,
> Without memcached, 9 seconds
> With memcached, 14 seconds - repeated several times, similar results.
>
> I am not sure what anyone else's experience is, but we will not be
> enabling memcached for subversion any time soon.
> --
> Tony Butt <tj...@cea.com.au>
> CEA Technologies
>