You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Nikolaus Demmel <ni...@nikolaus-demmel.de> on 2012/02/21 22:26:34 UTC
svn:externals parser API
Hi,
this is my first post to this list and I'm not subscribed, so please CC
me when answering.
I'm investigating how to improve using git-svn with svn-externals,
either with an additional script or directly in within git. In any case
I would need git-svn to display the externals. There is "git svn
propget", but there is no "recursive" parameter yet. A long time ago
"git svn show-externals" was written, which gets the svn:externals
property recursively and does some very crude processing. Unfortunetly
it can only deal with the pre 1.5 format for svn:externals.
Since external definitions are quite complex by now, with the old and
new format, with comments and empty lines, (peg) revisions, relative and
absolute URLs, it would be nice to get a parsed form of these externals
with the URLs already resolved and the order always the same
irrespective of whether the actual definitions are in the old or the new
format.
The obvious way to get those parse external definitions would be through
the language bindings. I looked at the python bindings for example, but
didn't find anything of the likes. In the case of git-svn, they use the
perl bindings, but I guess it is a similar story there.
I have the following questions:
1) Is there a way to leverage an existing svn:externals parser through
the language bindings (and I have been unable to find it)?
2) If not, would it be easy to add this to the language bindings?
A quick look at hgsubversion indicates that their handling of
svn:externals is far more sophisticated that that of git-svn at the
moment, but even in their code they mention that they don't support
escaping of special characters in paths, like spaces for example (this
was introduced in svn 1.6, I think).
So in any case it would be hard and error prone, if every tool that
wants to deal with externals has to write a parser itsself. Also, the
documentation on the special cases and all the details is not so easy to
find. The svn book explains it from a user perspective, but doesn't give
a full specification of the exact format. The most accurate and concise
description is maybe "svn help propget", but even there some things are
missing, e.g. that empty lines and trailing/perceding white space seems
to be allowed etc.
I hope someone can give me some insight in the matter. Currently the
simplest way to get my svn repository with many externals working nicely
in git, is to first use hgsubversion to create a mercurial repo and then
export that to git using hggit. I hope thats not the end of the story...
Best,
Nikolaus Demmel
Re: svn:externals parser API
Posted by Daniel Shahaf <da...@elego.de>.
Nikolaus Demmel wrote on Wed, Feb 22, 2012 at 14:22:26 +0100:
>
> Am 22.02.2012 um 11:16 schrieb Daniel Shahaf:
>
> > Daniel Shahaf wrote on Wed, Feb 22, 2012 at 11:31:00 +0200:
> >> Stefan Sperling wrote on Wed, Feb 22, 2012 at 00:08:35 +0100:
> >>> On Tue, Feb 21, 2012 at 11:50:54PM +0100, Stefan Sperling wrote:
> >>>> I think svn_wc_parse_externals_description3() should do what you need.
> >>>> Does this help you?
> >>>>
> >>>> The function is mapped in the python bindings:
> >>>>
> >>>>>>> import svn.wc
> >>>>>>> svn.wc.svn_wc_parse_externals_description3
> >>>> <function svn_wc_parse_externals_description3 at 0x20779a758>
> >>>
> >>>
> >>
> >> Based the difference between the signatures of the
> >> svn_wc_parse_externals_description() and
> >> svn_wc_parse_externals_description2(), I would suspect that someone
> >> needs to provide a Python implementation of the following typemap:
> >>
> >> #ifdef SWIGRUBY
> >> %typemap(argout) apr_array_header_t **externals_p {
> >> %append_output(svn_swig_rb_apr_array_to_array_external_item2(*$1));
> >> }
> >> #endif
> >>
> >
> > Done in r1292223:
> >
> > % PYTHONPATH=$prefix/svn-t1/lib/svn-python/ python -c \
> > 'import svn,svn.wc; print svn.wc.svn_wc_parse_externals_description3("","",1)'
> > []
> > %
> >
> > It will be available in 1.8.0.
> >
> > (perhaps we should relax our compat guidelines to allow adding exposed
> > APIs to the bindings in patch releases?)
> >
>
> Hi,
>
> this sound pretty good, thanks for the quick replies. If I understand this correctly (I havent used SWIG before), we could get this in the Perl bindings by adding a similar typemap for Perl?
>
Sounds right. (I haven't checked whether something beyond a typemap
would be needed for Perl.)
> I won't attempt it myself, since I know neither Perl, nor swig, but I will head over to the git folks and tell them that this should be easy to add.
>
Sure.
http://subversion.apache.org/patches
http://github.com/apache/subversion
:-)
> Cheers,
> Nikolaus
> (please CC on answers, I'm not subscribed to the list - thanks :-) )
Re: svn:externals parser API
Posted by Nikolaus Demmel <ni...@nikolaus-demmel.de>.
Am 22.02.2012 um 11:16 schrieb Daniel Shahaf:
> Daniel Shahaf wrote on Wed, Feb 22, 2012 at 11:31:00 +0200:
>> Stefan Sperling wrote on Wed, Feb 22, 2012 at 00:08:35 +0100:
>>> On Tue, Feb 21, 2012 at 11:50:54PM +0100, Stefan Sperling wrote:
>>>> I think svn_wc_parse_externals_description3() should do what you need.
>>>> Does this help you?
>>>>
>>>> The function is mapped in the python bindings:
>>>>
>>>>>>> import svn.wc
>>>>>>> svn.wc.svn_wc_parse_externals_description3
>>>> <function svn_wc_parse_externals_description3 at 0x20779a758>
>>>
>>>
>>
>> Based the difference between the signatures of the
>> svn_wc_parse_externals_description() and
>> svn_wc_parse_externals_description2(), I would suspect that someone
>> needs to provide a Python implementation of the following typemap:
>>
>> #ifdef SWIGRUBY
>> %typemap(argout) apr_array_header_t **externals_p {
>> %append_output(svn_swig_rb_apr_array_to_array_external_item2(*$1));
>> }
>> #endif
>>
>
> Done in r1292223:
>
> % PYTHONPATH=$prefix/svn-t1/lib/svn-python/ python -c \
> 'import svn,svn.wc; print svn.wc.svn_wc_parse_externals_description3("","",1)'
> []
> %
>
> It will be available in 1.8.0.
>
> (perhaps we should relax our compat guidelines to allow adding exposed
> APIs to the bindings in patch releases?)
>
Hi,
this sound pretty good, thanks for the quick replies. If I understand this correctly (I havent used SWIG before), we could get this in the Perl bindings by adding a similar typemap for Perl?
I won't attempt it myself, since I know neither Perl, nor swig, but I will head over to the git folks and tell them that this should be easy to add.
Cheers,
Nikolaus
(please CC on answers, I'm not subscribed to the list - thanks :-) )
Re: svn:externals parser API
Posted by Daniel Shahaf <da...@elego.de>.
Daniel Shahaf wrote on Wed, Feb 22, 2012 at 11:31:00 +0200:
> Stefan Sperling wrote on Wed, Feb 22, 2012 at 00:08:35 +0100:
> > On Tue, Feb 21, 2012 at 11:50:54PM +0100, Stefan Sperling wrote:
> > > I think svn_wc_parse_externals_description3() should do what you need.
> > > Does this help you?
> > >
> > > The function is mapped in the python bindings:
> > >
> > > >>> import svn.wc
> > > >>> svn.wc.svn_wc_parse_externals_description3
> > > <function svn_wc_parse_externals_description3 at 0x20779a758>
> >
> > Hmmm... my suggestion probably won't work.
> > The function doesn't seem to be usable.
> >
> > >>> svn.wc.svn_wc_parse_externals_description3("/tmp", "^/branch/gamma foo", 0)
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > File "/usr/local/lib/python2.7/site-packages/libsvn/wc.py", line 398, in svn_w
> > c_parse_externals_description3
> > return apply(_wc.svn_wc_parse_externals_description3, args)
> > ValueError: svn_wc_parse_externals_description3 is not implemented yet
> >
> >
> > But the 1.4 version of this function is working:
> >
> > >>> e = svn.wc.svn_wc_parse_externals_description("/tmp", "^/branch/gamma foo")
> > >>> e
> > {'foo': <libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc
> > _external_item_t *' at 0x2083e50c0> >}
> > >>> e['foo']
> > <libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc_externa
> > l_item_t *' at 0x2083e50c0> >
> > >>> e['foo'].target_dir
> > 'foo'
> > >>> e['foo'].url
> > '^/branch/gamma'
> > >>> e['foo'].revision
> > <libsvn.core.svn_opt_revision_t; proxy of <Swig Object of type 'svn_opt_revision_t *' at 0x2043d90d0> >
> >
> >
> > But that doesn't provide you with the enhancements of the 1.5 syntax,
> > e.g. peg-revision support.
> >
> > I am not a bindings expert so I don't know what we'd need to do to make
> > the newer variants of this function work from the bindings.
>
> Based the difference between the signatures of the
> svn_wc_parse_externals_description() and
> svn_wc_parse_externals_description2(), I would suspect that someone
> needs to provide a Python implementation of the following typemap:
>
> #ifdef SWIGRUBY
> %typemap(argout) apr_array_header_t **externals_p {
> %append_output(svn_swig_rb_apr_array_to_array_external_item2(*$1));
> }
> #endif
>
Done in r1292223:
% PYTHONPATH=$prefix/svn-t1/lib/svn-python/ python -c \
'import svn,svn.wc; print svn.wc.svn_wc_parse_externals_description3("","",1)'
[]
%
It will be available in 1.8.0.
(perhaps we should relax our compat guidelines to allow adding exposed
APIs to the bindings in patch releases?)
> Daniel
> (not a swig expert, either)
Re: svn:externals parser API
Posted by Daniel Shahaf <da...@elego.de>.
Stefan Sperling wrote on Wed, Feb 22, 2012 at 00:08:35 +0100:
> On Tue, Feb 21, 2012 at 11:50:54PM +0100, Stefan Sperling wrote:
> > I think svn_wc_parse_externals_description3() should do what you need.
> > Does this help you?
> >
> > The function is mapped in the python bindings:
> >
> > >>> import svn.wc
> > >>> svn.wc.svn_wc_parse_externals_description3
> > <function svn_wc_parse_externals_description3 at 0x20779a758>
>
> Hmmm... my suggestion probably won't work.
> The function doesn't seem to be usable.
>
> >>> svn.wc.svn_wc_parse_externals_description3("/tmp", "^/branch/gamma foo", 0)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/local/lib/python2.7/site-packages/libsvn/wc.py", line 398, in svn_w
> c_parse_externals_description3
> return apply(_wc.svn_wc_parse_externals_description3, args)
> ValueError: svn_wc_parse_externals_description3 is not implemented yet
>
>
> But the 1.4 version of this function is working:
>
> >>> e = svn.wc.svn_wc_parse_externals_description("/tmp", "^/branch/gamma foo")
> >>> e
> {'foo': <libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc
> _external_item_t *' at 0x2083e50c0> >}
> >>> e['foo']
> <libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc_externa
> l_item_t *' at 0x2083e50c0> >
> >>> e['foo'].target_dir
> 'foo'
> >>> e['foo'].url
> '^/branch/gamma'
> >>> e['foo'].revision
> <libsvn.core.svn_opt_revision_t; proxy of <Swig Object of type 'svn_opt_revision_t *' at 0x2043d90d0> >
>
>
> But that doesn't provide you with the enhancements of the 1.5 syntax,
> e.g. peg-revision support.
>
> I am not a bindings expert so I don't know what we'd need to do to make
> the newer variants of this function work from the bindings.
Based the difference between the signatures of the
svn_wc_parse_externals_description() and
svn_wc_parse_externals_description2(), I would suspect that someone
needs to provide a Python implementation of the following typemap:
#ifdef SWIGRUBY
%typemap(argout) apr_array_header_t **externals_p {
%append_output(svn_swig_rb_apr_array_to_array_external_item2(*$1));
}
#endif
Daniel
(not a swig expert, either)
Re: svn:externals parser API
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Feb 21, 2012 at 11:50:54PM +0100, Stefan Sperling wrote:
> I think svn_wc_parse_externals_description3() should do what you need.
> Does this help you?
>
> The function is mapped in the python bindings:
>
> >>> import svn.wc
> >>> svn.wc.svn_wc_parse_externals_description3
> <function svn_wc_parse_externals_description3 at 0x20779a758>
Hmmm... my suggestion probably won't work.
The function doesn't seem to be usable.
>>> svn.wc.svn_wc_parse_externals_description3("/tmp", "^/branch/gamma foo", 0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/libsvn/wc.py", line 398, in svn_w
c_parse_externals_description3
return apply(_wc.svn_wc_parse_externals_description3, args)
ValueError: svn_wc_parse_externals_description3 is not implemented yet
But the 1.4 version of this function is working:
>>> e = svn.wc.svn_wc_parse_externals_description("/tmp", "^/branch/gamma foo")
>>> e
{'foo': <libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc
_external_item_t *' at 0x2083e50c0> >}
>>> e['foo']
<libsvn.wc.svn_wc_external_item_t; proxy of <Swig Object of type 'svn_wc_externa
l_item_t *' at 0x2083e50c0> >
>>> e['foo'].target_dir
'foo'
>>> e['foo'].url
'^/branch/gamma'
>>> e['foo'].revision
<libsvn.core.svn_opt_revision_t; proxy of <Swig Object of type 'svn_opt_revision_t *' at 0x2043d90d0> >
But that doesn't provide you with the enhancements of the 1.5 syntax,
e.g. peg-revision support.
I am not a bindings expert so I don't know what we'd need to do to make
the newer variants of this function work from the bindings.
Re: svn:externals parser API
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Feb 21, 2012 at 10:26:34PM +0100, Nikolaus Demmel wrote:
> Hi,
>
> this is my first post to this list and I'm not subscribed, so please CC
> me when answering.
>
> I'm investigating how to improve using git-svn with svn-externals,
> either with an additional script or directly in within git. In any case
> I would need git-svn to display the externals. There is "git svn
> propget", but there is no "recursive" parameter yet. A long time ago
> "git svn show-externals" was written, which gets the svn:externals
> property recursively and does some very crude processing. Unfortunetly
> it can only deal with the pre 1.5 format for svn:externals.
>
> Since external definitions are quite complex by now, with the old and
> new format, with comments and empty lines, (peg) revisions, relative and
> absolute URLs, it would be nice to get a parsed form of these externals
> with the URLs already resolved and the order always the same
> irrespective of whether the actual definitions are in the old or the new
> format.
>
> The obvious way to get those parse external definitions would be through
> the language bindings. I looked at the python bindings for example, but
> didn't find anything of the likes. In the case of git-svn, they use the
> perl bindings, but I guess it is a similar story there.
>
> I have the following questions:
>
> 1) Is there a way to leverage an existing svn:externals parser through
> the language bindings (and I have been unable to find it)?
>
> 2) If not, would it be easy to add this to the language bindings?
>
> A quick look at hgsubversion indicates that their handling of
> svn:externals is far more sophisticated that that of git-svn at the
> moment, but even in their code they mention that they don't support
> escaping of special characters in paths, like spaces for example (this
> was introduced in svn 1.6, I think).
>
> So in any case it would be hard and error prone, if every tool that
> wants to deal with externals has to write a parser itsself. Also, the
> documentation on the special cases and all the details is not so easy to
> find. The svn book explains it from a user perspective, but doesn't give
> a full specification of the exact format. The most accurate and concise
> description is maybe "svn help propget", but even there some things are
> missing, e.g. that empty lines and trailing/perceding white space seems
> to be allowed etc.
>
> I hope someone can give me some insight in the matter. Currently the
> simplest way to get my svn repository with many externals working nicely
> in git, is to first use hgsubversion to create a mercurial repo and then
> export that to git using hggit. I hope thats not the end of the story...
I think svn_wc_parse_externals_description3() should do what you need.
Does this help you?
The function is mapped in the python bindings:
>>> import svn.wc
>>> svn.wc.svn_wc_parse_externals_description3
<function svn_wc_parse_externals_description3 at 0x20779a758>
Below is the C API documentation:
/**
* If @a externals_p is non-NULL, set @a *externals_p to an array of
* #svn_wc_external_item2_t * objects based on @a desc.
*
* If the format of @a desc is invalid, don't touch @a *externals_p and
* return #SVN_ERR_CLIENT_INVALID_EXTERNALS_DESCRIPTION. Thus, if
* you just want to check the validity of an externals description,
* and don't care about the parsed result, pass NULL for @a externals_p.
*
* The format of @a desc is the same as for values of the directory
* property #SVN_PROP_EXTERNALS. Look there for more details.
*
* If @a canonicalize_url is @c TRUE, canonicalize the @a url member
* of those objects. If the @a url member refers to an absolute URL,
* it will be canonicalized as URL consistent with the way URLs are
* canonicalized throughout the Subversion API. If, however, the
* @a url member makes use of the recognized (and proprietary)
* relative URL syntax, "canonicalization" is a less easily-defined
* concept which may even result in munging the relative URL syntax
* beyond recognition. You've been warned.
*
* Allocate the table, keys, and values in @a pool.
*
* Use @a parent_directory only in constructing error strings.
*
* @since New in 1.5.
*/
svn_error_t *
svn_wc_parse_externals_description3(apr_array_header_t **externals_p,
const char *parent_directory,
const char *desc,
svn_boolean_t canonicalize_url,
apr_pool_t *pool);