A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

Hey, all!

I've been watching the Python 3.x support shaping up over the course of the past couple of release cycles, and would like to begin converting ViewVC to Python3 now that doing so won't immediately have the effect of reverting that tool to ViewCVSOnly.  :-)  But I'm having to lay aside my pride and admit that I haven't built Subversion from a working copy in a few years and lack the confidence to figure it all out in a timely manner.  Is there a developer who has a viable release packaging environment who would be willing to package up a release (call it 1.14.0-pre-general-alpha-unfinished?) of Subversion for me that I can drop into a VM for a more user-level build experience so I can focus on integrating that with ViewVC?  In return I'd be happy to report back any Python-API-level problems that crop up.

Your always and forever,

-- Mike

Re: A little pre-packaging help, please?

[Nathan Hartman <ha...@gmail.com>]

On Wed, Mar 11, 2020 at 10:25 AM Michael Pilato <cm...@collab.net> wrote:

> > While you're waiting for someone to manually roll a nightly tarball, can
> we
> > circle back to the original problem for a minute: What stops you from
> > running autogen.sh?  Is it just that you'd like someone to figure out
> > the list of build dependency package names and configure arguments for
> > a working copy build for your platform?  (Note this could be a concern
> > even for tarball builds, if both py2 and py3 are installed.)
>
> So, yes, I "just" need a functional 1.14-to-be Subversion with Python 3
> bindings for the purposes of testing my attempts to convert ViewVC itself
> to Python 3.  But I'd also like to get a head start on getting some of this
> working in CollabNet's public and private ecosystems, which depend on
> having a Subversion release tarball as their starting block.  My request
> was aimed at hopefully meeting all of those needs simultaneously.


There's also the makefile in tools/dev/unix-build. It sets up a development
working copy, downloads and builds all the dependencies, and can run all
tests with all back ends and repo access layers. It is used on the OpenBSD
buildbot and I successfully used it on the latest Debian (though I've had
to install some -dev versions of some Debian packages, and change a few
versions of downloaded dependencies -- if interested, look for a dev@
thread titled "Makefile.svn: a dumb question" in December 2019 where among
other things I sent some patches for this makefile). I'm away from my
computer right now but if you go this route I'll be happy to help as much
as possible.

Cheers,
Nathan

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

Just to bring a bit of closure here, so far my testing with the Python 3 bindings has been really encouraging.  Once I wrapped my brain around the whole string/bytestring thing, upgrading existing consumers of the bindings has thus far been relatively straightforward.

As for the Docker-based release process stuff I attached in my previous message, I've since thrown that onto Github so anyone else who cares can benefit from/improve it.  See https://github.com/cmpilato/subversion-dist .

Thanks again for the tarball assembly help!

-- Mike

-- Mike
________________________________
From: Michael Pilato <cm...@collab.net>
Sent: Friday, March 13, 2020 3:03 PM
To: Yasuhito FUTATSUKI <fu...@poem.co.jp>; dev@subversion.apache.org <de...@subversion.apache.org>
Subject: Re: A little pre-packaging help, please?


[EXTERNAL EMAIL]

I'm attaching what I used to make this work, which is a Docker-based process. Just expand the tarball, enter the resulting directory, and run ./docker-build.sh.
________________________________
From: Michael Pilato <cm...@collab.net>
Sent: Friday, March 13, 2020 2:49 PM
To: Yasuhito FUTATSUKI <fu...@poem.co.jp>; dev@subversion.apache.org <de...@subversion.apache.org>
Subject: Re: A little pre-packaging help, please?

[EXTERNAL EMAIL]

> However as I wrote before[1] it is need to specify SWIG_PY_OPTS
> shell/environment variable to generate SWIG Python bindings C source
> and support codes. The value of SWIG_PY_OPTS depends on which is its
> target py2 or py3, on which SWIG version is prior 4.0 or not.

This was the key thing lacking!  When I added the SWIG_PY_OPTS stuff you recommended to the release.py command-line, those files generated as expected.  (And I wound up with what appears to be working build, including Python3 bindings.)

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

I'm attaching what I used to make this work, which is a Docker-based process. Just expand the tarball, enter the resulting directory, and run ./docker-build.sh.
________________________________
From: Michael Pilato <cm...@collab.net>
Sent: Friday, March 13, 2020 2:49 PM
To: Yasuhito FUTATSUKI <fu...@poem.co.jp>; dev@subversion.apache.org <de...@subversion.apache.org>
Subject: Re: A little pre-packaging help, please?

[EXTERNAL EMAIL]

> However as I wrote before[1] it is need to specify SWIG_PY_OPTS
> shell/environment variable to generate SWIG Python bindings C source
> and support codes. The value of SWIG_PY_OPTS depends on which is its
> target py2 or py3, on which SWIG version is prior 4.0 or not.

This was the key thing lacking!  When I added the SWIG_PY_OPTS stuff you recommended to the release.py command-line, those files generated as expected.  (And I wound up with what appears to be working build, including Python3 bindings.)

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

> However as I wrote before[1] it is need to specify SWIG_PY_OPTS
> shell/environment variable to generate SWIG Python bindings C source
> and support codes. The value of SWIG_PY_OPTS depends on which is its
> target py2 or py3, on which SWIG version is prior 4.0 or not.

This was the key thing lacking!  When I added the SWIG_PY_OPTS stuff you recommended to the release.py command-line, those files generated as expected.  (And I wound up with what appears to be working build, including Python3 bindings.)

Re: A little pre-packaging help, please?

[Yasuhito FUTATSUKI <fu...@poem.co.jp>]

On 2020/03/13 21:53, Michael Pilato wrote:

> One thing that appears to be different about this tarball than a typical release tarball is the absence of the swig-generated wrapper files (...python/svn_client.c, etc.).  I'm going to take a crack at replicating your release environment (the Debian VM, etc.), but is there a flag or package or something that was omitted that would obviously have resulted in the missing SWIG-generated files?
> 

As far as I read tools/dist/release.py, it runs "autogen.sh --release".

However as I wrote before[1] it is need to specify SWIG_PY_OPTS
shell/environment variable to generate SWIG Python bindings C source
and support codes. The value of SWIG_PY_OPTS depends on which is its
target py2 or py3, on which SWIG version is prior 4.0 or not.

Anyway, release tarball cannot include py2 and py3 bindings source
at the same time in current build system. So it is need to determine
which is included in release tarball, before fix release.py
(and/or autogen.sh). And if we want bundle both, we should modify
build system to handle both of source files. One of propose to do
so is a patch I posted before[2] (but as it require to rename
support library for Python 2 bindings from libsvn_swig_py to
libsvn_swig_py2, I don't commit it).


[1] https://mail-archives.apache.org/mod_mbox/subversion-dev/201912.mbox/%3c5b691321-209a-fd38-cb0d-85e2f5baeabf@poem.co.jp%3e
[2] https://mail-archives.apache.org/mod_mbox/subversion-dev/201912.mbox/%3cac0c9f98-df80-cdf6-e9c3-234248359482@poem.co.jp%3e

Cheers,
-- 
Yasuhito FUTATSUKI <fu...@poem.co.jp>

Re: A little pre-packaging help, please?

[Daniel Shahaf <d....@daniel.shahaf.name>]

Michael Pilato wrote on Fri, 13 Mar 2020 12:53 +00:00:
> > https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz
> > https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz.asc
> > 
> > For future reference, that's the output of «release.py build-env trunk-nightly &&
> > release.py roll trunk-nightly $(svn info --show-item=revision -- $REPOS_URL)»
> > in a plain Debian stable VM with the following packages installed:
> > subversion python-yaml pax gettext m4
> > 
> > You may need to pass PYTHON=python3 to configure.
> 
> Daniel, thanks for this -- the filedrop *and* the education.  I've 
> gotten farther along in my auxiliary work with this than I was getting 
> without it.  (I couldn't managed to get "release.py build-env" to 
> succeed.  Shoulda tried a clean VM.)
> 
> One thing that appears to be different about this tarball than a 
> typical release tarball is the absence of the swig-generated wrapper 
> files (...python/svn_client.c, etc.).  I'm going to take a crack at 
> replicating your release environment (the Debian VM, etc.), but is 
> there a flag or package or something that was omitted that would 
> obviously have resulted in the missing SWIG-generated files?
> 

In addition to futatuki's point about SWIG_PY_OPTS, I wonder if the fact
there wasn't a swig installed in the default $PATH made autogen.sh think
swig isn't installed and silently skip generating the swig files.
(release.py downloads and builds its own swig — see SwigDep — and ought
to point autogen.sh to it, but perhaps it neglects to do so.)  This
could be tested by installing the swig (and, for good measure,
py3c-dev) packages prior to rolling and seeing if that changes the outcome.

> Thanks, again.

You're welcome.

Daniel

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

> https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz
> https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz.asc
> 
> For future reference, that's the output of «release.py build-env trunk-nightly &&
> release.py roll trunk-nightly $(svn info --show-item=revision -- $REPOS_URL)»
> in a plain Debian stable VM with the following packages installed:
> subversion python-yaml pax gettext m4
> 
> You may need to pass PYTHON=python3 to configure.

Daniel, thanks for this -- the filedrop *and* the education.  I've gotten farther along in my auxiliary work with this than I was getting without it.  (I couldn't managed to get "release.py build-env" to succeed.  Shoulda tried a clean VM.)

One thing that appears to be different about this tarball than a typical release tarball is the absence of the swig-generated wrapper files (...python/svn_client.c, etc.).  I'm going to take a crack at replicating your release environment (the Debian VM, etc.), but is there a flag or package or something that was omitted that would obviously have resulted in the missing SWIG-generated files?

Thanks, again.
-- Mike

Re: A little pre-packaging help, please?

[Daniel Shahaf <d....@daniel.shahaf.name>]

Michael Pilato wrote on Wed, 11 Mar 2020 14:25 +00:00:
> > While you're waiting for someone to manually roll a nightly tarball, can we
> > circle back to the original problem for a minute: What stops you from
> > running autogen.sh?  Is it just that you'd like someone to figure out
> > the list of build dependency package names and configure arguments for
> > a working copy build for your platform?  (Note this could be a concern
> > even for tarball builds, if both py2 and py3 are installed.)
> 
> So, yes, I "just" need a functional 1.14-to-be Subversion with Python 3 
> bindings for the purposes of testing my attempts to convert ViewVC 
> itself to Python 3.  But I'd also like to get a head start on getting 
> some of this working in CollabNet's public and private ecosystems, 
> which depend on having a Subversion release tarball as their starting 
> block.  My request was aimed at hopefully meeting all of those needs 
> simultaneously.

https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz
https://home.apache.org/~danielsh/subversion-20200311_001/subversion-nightly-r1875088.tar.gz.asc

For future reference, that's the output of «release.py build-env trunk-nightly &&
release.py roll trunk-nightly $(svn info --show-item=revision -- $REPOS_URL)»
in a plain Debian stable VM with the following packages installed:
subversion python-yaml pax gettext m4

You may need to pass PYTHON=python3 to configure.

Cheers,

Daniel

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

> While you're waiting for someone to manually roll a nightly tarball, can we
> circle back to the original problem for a minute: What stops you from
> running autogen.sh?  Is it just that you'd like someone to figure out
> the list of build dependency package names and configure arguments for
> a working copy build for your platform?  (Note this could be a concern
> even for tarball builds, if both py2 and py3 are installed.)

So, yes, I "just" need a functional 1.14-to-be Subversion with Python 3 bindings for the purposes of testing my attempts to convert ViewVC itself to Python 3.  But I'd also like to get a head start on getting some of this working in CollabNet's public and private ecosystems, which depend on having a Subversion release tarball as their starting block.  My request was aimed at hopefully meeting all of those needs simultaneously.

Re: A little pre-packaging help, please?

[Daniel Shahaf <d....@daniel.shahaf.name>]

Michael Pilato wrote on Wed, 11 Mar 2020 13:35 +00:00:
> > Unfortunately, it seems that nightly release hasn't updated since
> > Feb 28, 2017, as it was reported on users@.
> 
> Looks to be the case.  svn_version.h carries "1.10.0 (r1784492)".
> 
> So ... about that original request, then?

While you're waiting for someone to manually roll a nightly tarball, can we
circle back to the original problem for a minute: What stops you from
running autogen.sh?  Is it just that you'd like someone to figure out
the list of build dependency package names and configure arguments for
a working copy build for your platform?  (Note this could be a concern
even for tarball builds, if both py2 and py3 are installed.)

Cheers,

Daniel

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

> Unfortunately, it seems that nightly release hasn't updated since
> Feb 28, 2017, as it was reported on users@.

Looks to be the case.  svn_version.h carries "1.10.0 (r1784492)".

So ... about that original request, then?

Re: A little pre-packaging help, please?

[Yasuhito FUTATSUKI <fu...@poem.co.jp>]

On 2020/03/11 0:26, Stefan Sperling wrote:
> On Tue, Mar 10, 2020 at 01:51:26PM +0000, Michael Pilato wrote:
>> Hey, all!
>>
>> I've been watching the Python 3.x support shaping up over the course of the past couple of release cycles, and would like to begin converting ViewVC to Python3 now that doing so won't immediately have the effect of reverting that tool to ViewCVSOnly.  :-)  But I'm having to lay aside my pride and admit that I haven't built Subversion from a working copy in a few years and lack the confidence to figure it all out in a timely manner.  Is there a developer who has a viable release packaging environment who would be willing to package up a release (call it 1.14.0-pre-general-alpha-unfinished?) of Subversion for me that I can drop into a VM for a more user-level build experience so I can focus on integrating that with ViewVC?  In return I'd be happy to report back any Python-API-level problems that crop up.
>>
>> Your always and forever,
>>
>> -- Mike
>>
> 
> Hey Mike!
> 
> Perhaps a nightly release would work?
> https://ci.apache.org/projects/subversion/nightlies/index.html

Unfortunately, it seems that nightly release hasn't updated since
Feb 28, 2017, as it was reported on users@.

https://mail-archives.apache.org/mod_mbox/subversion-users/201904.mbox/%3cAB14373A-8C76-4039-B9C7-BE1D0076399C@mails.ucas.ac.cn%3e
https://mail-archives.apache.org/mod_mbox/subversion-users/201905.mbox/%3cCAN0Gg1c_bim9zWucTsz1H8CMFt7-3ZQaxiOXCUtDQ0TTYW6PBw@mail.gmail.com%3e

> 
> Cheers,
> Stefan
> 

-- 
Yasuhito FUTATSUKI <fu...@poem.co.jp>

Re: A little pre-packaging help, please?

[Michael Pilato <cm...@collab.net>]

> Hey Mike!
> 
> Perhaps a nightly release would work?
> 
> https://ci.apache.org/projects/subversion/nightlies/index.html

Yes!  Awesome!  Thanks so much.

-- Mike

Re: A little pre-packaging help, please?

[Daniel Shahaf <d....@daniel.shahaf.name>]

Stefan Sperling wrote on Tue, 10 Mar 2020 15:26 +00:00:
> On Tue, Mar 10, 2020 at 01:51:26PM +0000, Michael Pilato wrote:
> > Hey, all!
> > 
> > I've been watching the Python 3.x support shaping up over the course of the past couple of release cycles, and would like to begin converting ViewVC to Python3 now that doing so won't immediately have the effect of reverting that tool to ViewCVSOnly.  :-)  But I'm having to lay aside my pride and admit that I haven't built Subversion from a working copy in a few years and lack the confidence to figure it all out in a timely manner.

What distro (name and version) are you targeting?  Are you looking for a
Subversion release (in which case stsp's suggestion ought to work) or
for a binary package (.deb/.rpm/etc)?

> > Is there a developer who has a viable release packaging environment who would be willing to package up a release (call it 1.14.0-pre-general-alpha-unfinished?) of Subversion for me that I can drop into a VM for a more user-level build experience so I can focus on integrating that with ViewVC?  In return I'd be happy to report back any Python-API-level problems that crop up.
> > 
> > Your always and forever,
> > 
> > -- Mike
> > 
> 
> Hey Mike!
> 
> Perhaps a nightly release would work?
> https://ci.apache.org/projects/subversion/nightlies/index.html

Re: A little pre-packaging help, please?

[Stefan Sperling <st...@elego.de>]

On Tue, Mar 10, 2020 at 01:51:26PM +0000, Michael Pilato wrote:
> Hey, all!
> 
> I've been watching the Python 3.x support shaping up over the course of the past couple of release cycles, and would like to begin converting ViewVC to Python3 now that doing so won't immediately have the effect of reverting that tool to ViewCVSOnly.  :-)  But I'm having to lay aside my pride and admit that I haven't built Subversion from a working copy in a few years and lack the confidence to figure it all out in a timely manner.  Is there a developer who has a viable release packaging environment who would be willing to package up a release (call it 1.14.0-pre-general-alpha-unfinished?) of Subversion for me that I can drop into a VM for a more user-level build experience so I can focus on integrating that with ViewVC?  In return I'd be happy to report back any Python-API-level problems that crop up.
> 
> Your always and forever,
> 
> -- Mike
> 

Hey Mike!

Perhaps a nightly release would work?
https://ci.apache.org/projects/subversion/nightlies/index.html

Cheers,
Stefan