You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Robert Pluim <rp...@bigfoot.com> on 2003/05/14 12:24:16 UTC
[PATCH] make cvs2svn check the exit status of commands
Query to black-belt pythoners (pythonistas?): is there a more
pythonesque way of doing this? According to my docs, os.system
doesn't throw exceptions, so checking the return code is the only
way to do this.
* define run_external as an error checking wrapper around os.system,
and use it in pass3, pass4 & pass5.
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Greg Stein <gs...@lyra.org>.
On Wed, May 14, 2003 at 05:06:37PM +0200, Tobias Ringstrom wrote:
>...
> > > If not, the following fairly obvious code might work:
> > >
> > > def sort_file(dst, src):
> > > lines = open(src).readlines()
> > > lines.sort()
> > > open(dst, 'w').writelines(lines)
> >
> >I assume that would work, but it might consume a large chunk of
> >memory. People who know more about python than I will have to
> >comment.
That is definitely an in-memory operation.
> True. It will consume about as much memory as the file is large. The
> sort happens in place, so there will not be a factor of two.
True.
> One possibility is to try to run sort first, and fall back to a python
> sort if it fails. On the other hand, RAM is really cheap these days! ;-)
Not *that* cheap. The sort was performed externally because that file could
be gigabytes in size. cvs2svn is intended to convert repositories from small
to huge.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Tobias Ringstrom <to...@ringstrom.mine.nu>.
Robert Pluim wrote:
> Well, if you install cygwin on your windows box, yes. You mean you
> don't? ;-)
Of course I do. I'm talking about all the other guys! :-)
> > If
> > not, the following fairly obvious code might work:
> >
> > def sort_file(dst, src):
> > lines = open(src).readlines()
> > lines.sort()
> > open(dst, 'w').writelines(lines)
>
> I assume that would work, but it might consume a large chunk of
> memory. People who know more about python than I will have to
> comment.
True. It will consume about as much memory as the file is large. The
sort happens in place, so there will not be a factor of two.
One possibility is to try to run sort first, and fall back to a python
sort if it fails. On the other hand, RAM is really cheap these days! ;-)
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Tobias Ringstrom <to...@ringstrom.mine.nu>.
Michael Wood wrote:
> This is what the cvs2svn.py README says about it:
>
> - Make sure you have GNU 'sort' installed, or some other sort
> program that handles large files other than by reading them
> entirely into memory.
Duh! Yes of course. When will I learn to RTFM... :-)
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Michael Wood <mw...@its.uct.ac.za>.
On Wed, May 14, 2003 at 04:15:06PM +0200, Robert Pluim wrote:
> Tobias Ringstrom writes:
[snip]
> > > def pass3(ctx):
> > > # sort the log files
> > >- os.system('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
> > >- ctx.log_fname_base + SORTED_REVS_SUFFIX))
> > >+ run_external('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
> > >+ ctx.log_fname_base + SORTED_REVS_SUFFIX))
> > Unrelated to your patch, but does sort exist for non-unix systems?
>
> Well, if you install cygwin on your windows box, yes. You mean you
> don't? ;-)
This is what the cvs2svn.py README says about it:
- Make sure you have GNU 'sort' installed, or some other sort
program that handles large files other than by reading them
entirely into memory.
> > If
> > not, the following fairly obvious code might work:
> >
> > def sort_file(dst, src):
> > lines = open(src).readlines()
> > lines.sort()
> > open(dst, 'w').writelines(lines)
>
> I assume that would work, but it might consume a large chunk of
> memory. People who know more about python than I will have to
> comment.
[snip]
I think the whole point of using GNU sort or an equivalent is that you
avoid using "a large chunk of memory."
--
Michael Wood <mw...@its.uct.ac.za>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Robert Pluim <rp...@bigfoot.com>.
Tobias Ringstrom writes:
> Robert Pluim wrote:
> >+def run_external(command):
> >+ if os.system(command):
> >+ print 'Error running "%s"' % command
> >+ sys.exit(1)
> >
> Micro-optimization: sys.exit('Error running "%s"' % command)
>
Ahh, the joys of a proper object-oriented language! Patch mark II
attached.
> > def pass3(ctx):
> > # sort the log files
> >- os.system('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
> >- ctx.log_fname_base + SORTED_REVS_SUFFIX))
> >+ run_external('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
> >+ ctx.log_fname_base + SORTED_REVS_SUFFIX))
> >
> >
> Unrelated to your patch, but does sort exist for non-unix systems?
Well, if you install cygwin on your windows box, yes. You mean you
don't? ;-)
> If
> not, the following fairly obvious code might work:
>
> def sort_file(dst, src):
> lines = open(src).readlines()
> lines.sort()
> open(dst, 'w').writelines(lines)
I assume that would work, but it might consume a large chunk of
memory. People who know more about python than I will have to
comment.
Robert
Re: [PATCH] make cvs2svn check the exit status of commands
Posted by Tobias Ringstrom <to...@ringstrom.mine.nu>.
Robert Pluim wrote:
>Query to black-belt pythoners (pythonistas?): is there a more
>pythonesque way of doing this? According to my docs, os.system
>doesn't throw exceptions, so checking the return code is the only
>way to do this.
>
>* define run_external as an error checking wrapper around os.system,
> and use it in pass3, pass4 & pass5.
>
>
>
>------------------------------------------------------------------------
>
>Index: tools/cvs2svn/cvs2svn.py
>===================================================================
>--- tools/cvs2svn/cvs2svn.py (revision 5932)
>+++ tools/cvs2svn/cvs2svn.py (working copy)
>@@ -272,6 +272,10 @@
> gen_key_base = gen_key_base + 1
> return key
>
>+def run_external(command):
>+ if os.system(command):
>+ print 'Error running "%s"' % command
>+ sys.exit(1)
>
Micro-optimization: sys.exit('Error running "%s"' % command)
> def pass3(ctx):
> # sort the log files
>- os.system('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
>- ctx.log_fname_base + SORTED_REVS_SUFFIX))
>+ run_external('sort %s > %s' % (ctx.log_fname_base + CLEAN_REVS_SUFFIX,
>+ ctx.log_fname_base + SORTED_REVS_SUFFIX))
>
>
Unrelated to your patch, but does sort exist for non-unix systems? If
not, the following fairly obvious code might work:
def sort_file(dst, src):
lines = open(src).readlines()
lines.sort()
open(dst, 'w').writelines(lines)
/Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org