You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Charles Butterfield <ch...@nextcentury.com> on 2008/02/18 00:21:53 UTC
Candidate bug in build process -- cannot pass tests building/testing as root
As suggested on the Issue Tracker page, I'm sending this to "users" for
review.
While rebuilding the subversion code (to experiment with
SVN_DELTA_WINDOW_SIZE), I noticed that a few of the tests ("make check")
fail if running as root. I don't know if the problem is building as
root, or running the tests as root. My solution was to build as
non-root, but then that is annoying because I have to run in another
window and do the root only things (make install, and creating a few of
the necessary output directories).
I spent quite a while puzzling over this before eventually noticing that
others had run into this issue. (Thank you maillists and Google!).
Suggestions
* If the build is likely to fail while running as root, it would
be very helpful to stop immediately with an error message to that
effect. Much time would be saved for all of the souls that try the easy
way of building.
* And naturally, if fixing things so you can build as root is not
hard, that would be a good thing too.
Is there any objection to adding this as an open issue? I searched the
Issue Tracker for "root" and found nothing related.
Regards
-- Charlie
Re: Impact of SVN_DELTA_WINDOW_SIZE on large files with small differences
Posted by eg <eg...@gmail.com>.
Charles Butterfield wrote:
...
> Conclusions
> -----------
> For really big files, the large number of windows causes small deltas to
> be magnified linearly in the number of windows. Allowing a large window
> size
> seems to have positive effects on both the encoding of deltas as well as
> compression of the "base" revision.
>
> With my admin hat on, I would sure like the ability to specify a larger
> window size (within limits) when processing repos with large files,
> perhaps only when processing those large files.
>
I thought there was something in the code which made changing window
sizes backward incompatible.
See:
http://svn.haxx.se/dev/archive-2006-07/0911.shtml
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Impact of SVN_DELTA_WINDOW_SIZE on large files with small differences
Posted by Charles Butterfield <ch...@nextcentury.com>.
One of my users commits a large text file (50 MB) containing minor
changes every couple of days. I was surprised at the size of the
deltas and did some crude testing (which was flawed by my lack of
understanding of the issues). I reported this in the thread titled
"Binary differencing performs poorly (erractically) on very large text
file", and got some excellent guidance.
It turns out the erratic aspect was largely due to my ignorance of the
skip-deltas algorithm which I exacerbated by checking in a zero length
initial version for my "controlled tests". Oh well.
However, I was also introduced to the SVN_DELTA_WINDOW_SIZE (currently a
#define) and wondered what the effect of varying it would be. Here are
my results from revised testing with a specific (real) set of sample
files.
Results:
DELTA Local Remote Average
WIN Commit Commit Rev 0 Rev 1-9
Size Time Time Size Size
---- ---- ---- ------ ----
100KB 73 71 11.5MB 2,138,000
1MB 66 61 8.3MB 174,600
10MB 103 84 6.8MB 20,535
100MB 248 156 5.9MB 4,800
Observations
------------
1) The penalty for the number of windows is roughly linear until the
Window is the size of the file. In this case the file is 50 MB
with an average delta of 4,800 bytes. At 10MB window size uses
5 windows and bumps the average delta to 20 KB. A 1 MB window
bumps the average delta by another factor of 10 and so on.
2) A big window compresses really nicely too.
3) Running the client remotely is faster on a 1 GB Ethernet.
4) The 1 MB window was actually faster than the 100 KB window. That
Was surprising. Then there is a performance hit for larger
Windows.
5) "make check" started to fail some tests at WIN=100MB, but
The server and client (local and remote) seemed to work fine.
Perhaps SEEMED is the operative word?
Conclusions
-----------
For really big files, the large number of windows causes small deltas to
be magnified linearly in the number of windows. Allowing a large window
size
seems to have positive effects on both the encoding of deltas as well as
compression of the "base" revision.
With my admin hat on, I would sure like the ability to specify a larger
window size (within limits) when processing repos with large files,
perhaps only when processing those large files.
So here is some food for thought if a developer gets bored.
Regards
-- Charlie
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Candidate bug in build process -- cannot pass tests building/testing as root
Posted by Karl Fogel <kf...@red-bean.com>.
"Charles Butterfield" <ch...@nextcentury.com> writes:
> As suggested on the Issue Tracker page, I?m sending this to ?users?
> for review.
>
> While rebuilding the subversion code (to experiment with
> SVN_DELTA_WINDOW_SIZE), I noticed that a few of the tests (?make
> check?) fail if running as root. I don?t know if the problem is
> building as root, or running the tests as root. My solution was to
> build as non-root, but then that is annoying because I have to run in
> another window and do the root only things (make install, and creating
> a few of the necessary output directories).
>
> I spent quite a while puzzling over this before eventually noticing
> that others had run into this issue. (Thank you maillists and
> Google!).
>
> Suggestions
>
> * If the build is likely to fail while running as root, it would be
> very helpful to stop immediately with an error message to that effect.
> Much time would be saved for all of the souls that try the easy way of
> building.
>
> * And naturally, if fixing things so you can build as root
> is not hard, that would be a good thing too.
>
> Is there any objection to adding this as an open issue? I searched
> the Issue Tracker for ?root? and found nothing related.
Before we add an issue, we should get some more details on why it
failed, what error messages you saw, what tests.log said, what RA
layer you were running the test suit over (i.e., 'make check' versus
'make svncheck' versus 'make davcheck'), etc.
In other words, if this is a bug report, let's make it into a real bug
report, with transcript and everything. See
http://subversion.tigris.org/bugs.html
Thanks,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org