You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Yasuhito FUTATSUKI <fu...@poem.co.jp> on 2019/10/19 09:45:28 UTC
SWIG Python bindings build with SWIG 4.0 (was: Re: Issue tracker
housecleaning: SVN-1722)
On 2019/10/18 8:39, Branko Čibej wrote:
> On 17.10.2019 23:46, Branko Čibej wrote:
>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the build and
>>>>>>> test process under Python 3. Any committer can edit the buildbot
>>>>>>> scripts[1], but the question is which of the buildbot slaves has Python 3
>>>>>>> installed?
>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>
>>>>>> Been there, done that, bought the DVD.
>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing on that
>>>>> bot? They (currently) fail under python3.
>>>> I don't know. It's possible, I suppose, that the activation of the
>>>> python3 virtual environment has no effect on the test driver. It might
>>>> not be a bad idea to have the tests print the Python version in the
>>>> summary and possibly in the log of every test case.
>>> What's the value of $(PYTHON) in Makefile? That's what the «check» target uses.
>>
>> Yep, apparently that's the bug ... I'm testing script changes now, along
>> with r1868566 for good measure.
>
> I've set up this:
> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>
> It will build and test the core libraries and swig-py bindings with
> Python 3 on the swig-py3 branch.
>
> I hope ... build #0 running as we speak.
Unfortunately build #2, which ran after upgrading SWIG and Python, failed
to build SWIG Python bindings because of SWIG 4.0, as reported on SVN-4818.
This also affects on trunk with SWIG 4.0.
(e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be produced,
but the modules don't work correctly.
It seems they were caused by incompatibility of Python code for proxy
object generated by SWIG, and it can not be resolved so simple....
(importlib module vs simply use 'import', absense of _swig_setattr()
and _swig_getattr(), etc.)
Cheers,
--
Yasuhito FUTATSUKI <fu...@poem.co.jp>
Re: SWIG Python bindings build with SWIG 4.0
Posted by Branko Čibej <br...@apache.org>.
On 21.10.2019 04:11, Branko Čibej wrote:
> You're right. Fixed in r1868674, this makes the Python bindings tests
> work on my laptop, with both the system Python 2 and the Homebrewed
> Python 3. I've forced the tests to run on the buildslave.
I merged trunk to swig-py3 to get the fixes from trunk for the regular
test suite.
-- Brane
Re: SWIG Python bindings build with SWIG 4.0
Posted by Branko Čibej <br...@apache.org>.
On 21.10.2019 01:15, Yasuhito FUTATSUKI wrote:
> On 2019/10/20 21:58, Branko Čibej wrote:
>> On 20.10.2019 06:35, Yasuhito FUTATSUKI wrote:
>>> On 2019/10/20 8:37, Branko Čibej wrote:
>>>> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote:
>>>>> On 2019/10/20 6:59, Branko Čibej wrote:
>>>>>> On 19.10.2019 23:06, Branko Čibej wrote:
>>>>>>> On 19.10.2019 19:55, Branko Čibej wrote:
>>>>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>>>>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to
>>>>>>>>>>>>>>>> run the
>>>>>>>>>>>>>>>> build and
>>>>>>>>>>>>>>>> test process under Python 3. Any committer can edit the
>>>>>>>>>>>>>>>> buildbot
>>>>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot
>>>>>>>>>>>>>>>> slaves
>>>>>>>>>>>>>>>> has Python 3
>>>>>>>>>>>>>>>> installed?
>>>>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing
>>>>>>>>>>>>>> on that
>>>>>>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>>>>>>> I don't know. It's possible, I suppose, that the
>>>>>>>>>>>>> activation of
>>>>>>>>>>>>> the
>>>>>>>>>>>>> python3 virtual environment has no effect on the test
>>>>>>>>>>>>> driver. It
>>>>>>>>>>>>> might
>>>>>>>>>>>>> not be a bad idea to have the tests print the Python version
>>>>>>>>>>>>> in the
>>>>>>>>>>>>> summary and possibly in the log of every test case.
>>>>>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the
>>>>>>>>>>>> «check»
>>>>>>>>>>>> target uses.
>>>>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes
>>>>>>>>>>> now,
>>>>>>>>>>> along
>>>>>>>>>>> with r1868566 for good measure.
>>>>>>>>>> I've set up this:
>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>>>>>>
>>>>>>>>>> It will build and test the core libraries and swig-py bindings
>>>>>>>>>> with
>>>>>>>>>> Python 3 on the swig-py3 branch.
>>>>>>>>>>
>>>>>>>>>> I hope ... build #0 running as we speak.
>>>>>>>>> Unfortunately build #2, which ran after upgrading SWIG and
>>>>>>>>> Python,
>>>>>>>>> failed
>>>>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>>>>>>> SVN-4818.
>>>>>>>>>
>>>>>>>>> This also affects on trunk with SWIG 4.0.
>>>>>>>>> (e.g.
>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>>>>>>
>>>>>>>>> With attached patch on trunk
>>>>>>>>> (trunk_build_with_swig4_patch.txt) and
>>>>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be
>>>>>>>>> produced,
>>>>>>>>> but the modules don't work correctly.
>>>>>>>>>
>>>>>>>>> It seems they were caused by incompatibility of Python code for
>>>>>>>>> proxy
>>>>>>>>> object generated by SWIG, and it can not be resolved so
>>>>>>>>> simple....
>>>>>>>>> (importlib module vs simply use 'import', absense of
>>>>>>>>> _swig_setattr()
>>>>>>>>> and _swig_getattr(), etc.)
>>>>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed
>>>>>>>> builds.
>>>>>>>
>>>>>>> Hm, that did not really help. On the swig-py3 branch:
>>>>>>>
>>>>>>> /bin/sh: line 1: 62481 Segmentation fault: 11
>>>>>>> /srv/virtualenv-3/bin/python
>>>>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>>>>>>>
>>>>>>>
>>>>>>> make: *** [check-swig-py] Error 139
>>>>>>>
>>>>>>>
>>>>>>> So, the Python bindings seem to have crashed the Python 3
>>>>>>> interpreter on
>>>>>>> macOS?
>>>>>>
>>>>>> Yes, they did:
>>>>>>
>>>>>> Process 28278 stopped
>>>>>> * thread #2, queue = 'com.apple.main-thread', stop reason =
>>>>>> EXC_BAD_ACCESS (code=1, address=0x48)
>>>>>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
>>>>>> Python`PyErr_Restore:
>>>>>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
>>>>>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
>>>>>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
>>>>>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
>>>>>> Target 0: (Python) stopped.
>>>>>>
>>>>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at
>>>>>> core.c:40158:12
>>>>>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
>>>>>> 40156 #endif
>>>>>> 40157
>>>>>> -> 40158 md = d = PyModule_GetDict(m);
>>>>>> 40159 (void)md;
>>>>>> 40160
>>>>>> 40161 SWIG_InitializeModule(0);
>>>>>> (lldb) print m
>>>>>> (PyObject *) $0 = 0x000000010279ad70
>>>>>> (lldb) print *m
>>>>>> (PyObject) $1 = {
>>>>>> ob_refcnt = 785
>>>>>> ob_type = 0x000000010026ea30
>>>>>> }
>>>>>
>>>>> I guess the _core module linked incompatible Python library to
>>>>> execute.
>>>>>
>>>>> on
>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio
>>>>>
>>>>>
>>>>> :
>>>>> [[[
>>>>> checking for compiling Python extensions... clang
>>>>> checking for linking Python extensions... clang -bundle -undefined
>>>>> dynamic_lookup -isysroot
>>>>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
>>>>>
>>>>>
>>>>> -F/usr/local/opt/python/Frameworks -framework Python
>>>>> checking for linking Python libraries... -bundle -undefined
>>>>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
>>>>> ]]]
>>>>
>>>>
>>>> Good catch. This certainly looks correct, it's where Homebrew installs
>>>> python. However ...
>>>>
>>>> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so
>>>> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so:
>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
>>>> (compatibility version 2.7.0, current version 2.7.10)
>>>> ...
>>>>
>>>>
>>>> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so
>>>> that it
>>>> finds the Python 3 installation.
>>
>> (Actually that would be DYLD_FRAMEWORK_PATH.)
>>
>>> ... or have to use Run-Path Dependent Libraries feature on link time,
>>> if DYLD_LIBRARY_PATH cannot work well due to SIP.
>>>
>>> https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
>>>
>>>
>>>
>>> The link option in log above comes from result of
>>> 'build/get-py-info.py --link', so should we tweak build/get-py-info.py,
>>> or add option to configure to pass Python dynamic library path?
>>
>>
>> Using RPATH for the Python framework on macOS would be the right
>> solution, but it doesn't solve the problem. It looks like we need some
>> magic incantation on macOS to explicitly pick the Python 3 framework;
>> using just the -F option doesn't work as expected, the path to the
>> system default Python 2.7 framework is hard-coded in to the extension
>> libraries.
>
> Umm.... it seems extension libraries actually don't need to link
> "Python" dynamic library.
>
> e.g.
> $ otool -L `python -c 'import _socket;print(_socket.__file__)'`
> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so:
>
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
> current version 1252.250.1)
You're right. Fixed in r1868674, this makes the Python bindings tests
work on my laptop, with both the system Python 2 and the Homebrewed
Python 3. I've forced the tests to run on the buildslave.
-- Brane
Re: SWIG Python bindings build with SWIG 4.0
Posted by Yasuhito FUTATSUKI <fu...@poem.co.jp>.
On 2019/10/20 21:58, Branko Čibej wrote:
> On 20.10.2019 06:35, Yasuhito FUTATSUKI wrote:
>> On 2019/10/20 8:37, Branko Čibej wrote:
>>> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote:
>>>> On 2019/10/20 6:59, Branko Čibej wrote:
>>>>> On 19.10.2019 23:06, Branko Čibej wrote:
>>>>>> On 19.10.2019 19:55, Branko Čibej wrote:
>>>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>>>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to
>>>>>>>>>>>>>>> run the
>>>>>>>>>>>>>>> build and
>>>>>>>>>>>>>>> test process under Python 3. Any committer can edit the
>>>>>>>>>>>>>>> buildbot
>>>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>>>>>>> has Python 3
>>>>>>>>>>>>>>> installed?
>>>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing
>>>>>>>>>>>>> on that
>>>>>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>>>>>> I don't know. It's possible, I suppose, that the activation of
>>>>>>>>>>>> the
>>>>>>>>>>>> python3 virtual environment has no effect on the test
>>>>>>>>>>>> driver. It
>>>>>>>>>>>> might
>>>>>>>>>>>> not be a bad idea to have the tests print the Python version
>>>>>>>>>>>> in the
>>>>>>>>>>>> summary and possibly in the log of every test case.
>>>>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the
>>>>>>>>>>> «check»
>>>>>>>>>>> target uses.
>>>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes
>>>>>>>>>> now,
>>>>>>>>>> along
>>>>>>>>>> with r1868566 for good measure.
>>>>>>>>> I've set up this:
>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>>>>>
>>>>>>>>> It will build and test the core libraries and swig-py bindings
>>>>>>>>> with
>>>>>>>>> Python 3 on the swig-py3 branch.
>>>>>>>>>
>>>>>>>>> I hope ... build #0 running as we speak.
>>>>>>>> Unfortunately build #2, which ran after upgrading SWIG and Python,
>>>>>>>> failed
>>>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>>>>>> SVN-4818.
>>>>>>>>
>>>>>>>> This also affects on trunk with SWIG 4.0.
>>>>>>>> (e.g.
>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>>>>>
>>>>>>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be
>>>>>>>> produced,
>>>>>>>> but the modules don't work correctly.
>>>>>>>>
>>>>>>>> It seems they were caused by incompatibility of Python code for
>>>>>>>> proxy
>>>>>>>> object generated by SWIG, and it can not be resolved so simple....
>>>>>>>> (importlib module vs simply use 'import', absense of
>>>>>>>> _swig_setattr()
>>>>>>>> and _swig_getattr(), etc.)
>>>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>>>>>>
>>>>>> Hm, that did not really help. On the swig-py3 branch:
>>>>>>
>>>>>> /bin/sh: line 1: 62481 Segmentation fault: 11
>>>>>> /srv/virtualenv-3/bin/python
>>>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>>>>>>
>>>>>> make: *** [check-swig-py] Error 139
>>>>>>
>>>>>>
>>>>>> So, the Python bindings seem to have crashed the Python 3
>>>>>> interpreter on
>>>>>> macOS?
>>>>>
>>>>> Yes, they did:
>>>>>
>>>>> Process 28278 stopped
>>>>> * thread #2, queue = 'com.apple.main-thread', stop reason =
>>>>> EXC_BAD_ACCESS (code=1, address=0x48)
>>>>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
>>>>> Python`PyErr_Restore:
>>>>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
>>>>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
>>>>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
>>>>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
>>>>> Target 0: (Python) stopped.
>>>>>
>>>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
>>>>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
>>>>> 40156 #endif
>>>>> 40157
>>>>> -> 40158 md = d = PyModule_GetDict(m);
>>>>> 40159 (void)md;
>>>>> 40160
>>>>> 40161 SWIG_InitializeModule(0);
>>>>> (lldb) print m
>>>>> (PyObject *) $0 = 0x000000010279ad70
>>>>> (lldb) print *m
>>>>> (PyObject) $1 = {
>>>>> ob_refcnt = 785
>>>>> ob_type = 0x000000010026ea30
>>>>> }
>>>>
>>>> I guess the _core module linked incompatible Python library to execute.
>>>>
>>>> on
>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio
>>>>
>>>> :
>>>> [[[
>>>> checking for compiling Python extensions... clang
>>>> checking for linking Python extensions... clang -bundle -undefined
>>>> dynamic_lookup -isysroot
>>>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
>>>>
>>>> -F/usr/local/opt/python/Frameworks -framework Python
>>>> checking for linking Python libraries... -bundle -undefined
>>>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
>>>> ]]]
>>>
>>>
>>> Good catch. This certainly looks correct, it's where Homebrew installs
>>> python. However ...
>>>
>>> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so
>>> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so:
>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
>>> (compatibility version 2.7.0, current version 2.7.10)
>>> ...
>>>
>>>
>>> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so that it
>>> finds the Python 3 installation.
>
> (Actually that would be DYLD_FRAMEWORK_PATH.)
>
>> ... or have to use Run-Path Dependent Libraries feature on link time,
>> if DYLD_LIBRARY_PATH cannot work well due to SIP.
>>
>> https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
>>
>>
>> The link option in log above comes from result of
>> 'build/get-py-info.py --link', so should we tweak build/get-py-info.py,
>> or add option to configure to pass Python dynamic library path?
>
>
> Using RPATH for the Python framework on macOS would be the right
> solution, but it doesn't solve the problem. It looks like we need some
> magic incantation on macOS to explicitly pick the Python 3 framework;
> using just the -F option doesn't work as expected, the path to the
> system default Python 2.7 framework is hard-coded in to the extension
> libraries.
Umm.... it seems extension libraries actually don't need to link
"Python" dynamic library.
e.g.
$ otool -L `python -c 'import _socket;print(_socket.__file__)'`
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
_core.so built on FreeBSD:
$ ldd subversion/bindings/swig/python/.libs/_core.so
subversion/bindings/swig/python/.libs/_core.so:
libsvn_swig_py-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py/.libs/libsvn_swig_py-1.so.0 (0x801298000)
libsvn_client-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_client/.libs/libsvn_client-1.so.0 (0x8014ac000)
libsvn_wc-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_wc/.libs/libsvn_wc-1.so.0 (0x801730000)
libsvn_diff-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_diff/.libs/libsvn_diff-1.so.0 (0x8019d5000)
libsvn_ra-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_ra/.libs/libsvn_ra-1.so.0 (0x801bec000)
libsvn_ra_local-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_ra_local/.libs/libsvn_ra_local-1.so.0 (0x801df8000)
libsvn_repos-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_repos/.libs/libsvn_repos-1.so.0 (0x802001000)
libsvn_fs-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_fs/.libs/libsvn_fs-1.so.0 (0x80223c000)
libsvn_fs_fs-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_fs_fs/.libs/libsvn_fs_fs-1.so.0 (0x802449000)
libsvn_fs_x-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_fs_x/.libs/libsvn_fs_x-1.so.0 (0x802698000)
libsvn_fs_util-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_fs_util/.libs/libsvn_fs_util-1.so.0 (0x8028e7000)
libsvn_ra_svn-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_ra_svn/.libs/libsvn_ra_svn-1.so.0 (0x802ae9000)
libsvn_ra_serf-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_ra_serf/.libs/libsvn_ra_serf-1.so.0 (0x802d0d000)
libserf-1.so.1 => /usr/local/lib/libserf-1.so.1 (0x802f3e000)
libsvn_delta-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_delta/.libs/libsvn_delta-1.so.0 (0x803159000)
libsvn_subr-1.so.0 => /home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/libsvn_subr/.libs/libsvn_subr-1.so.0 (0x803377000)
libaprutil-1.so.0 => /usr/local/lib/libaprutil-1.so.0 (0x803608000)
libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x803835000)
libz.so.6 => /lib/libz.so.6 (0x803a5f000)
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x803c77000)
libmagic.so.4 => /usr/lib/libmagic.so.4 (0x803fd9000)
liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x8041fc000)
libutf8proc.so.2 => /usr/local/lib/libutf8proc.so.2 (0x80442b000)
libapr-1.so.0 => /usr/local/lib/libapr-1.so.0 (0x804675000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x8048af000)
libthr.so.3 => /lib/libthr.so.3 (0x804aba000)
libc.so.7 => /lib/libc.so.7 (0x800823000)
libssl.so.8 => /usr/lib/libssl.so.8 (0x804ce2000)
libcrypto.so.8 => /lib/libcrypto.so.8 (0x805000000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x80546f000)
libdb-5.3.so.0 => /usr/local/lib/libdb-5.3.so.0 (0x80568e000)
libgdbm.so.6 => /usr/local/lib/libgdbm.so.6 (0x805a33000)
--
Yasuhito FUTATSUKI <fu...@poem.co.jp>
Re: SWIG Python bindings build with SWIG 4.0
Posted by Branko Čibej <br...@apache.org>.
On 20.10.2019 06:35, Yasuhito FUTATSUKI wrote:
> On 2019/10/20 8:37, Branko Čibej wrote:
>> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote:
>>> On 2019/10/20 6:59, Branko Čibej wrote:
>>>> On 19.10.2019 23:06, Branko Čibej wrote:
>>>>> On 19.10.2019 19:55, Branko Čibej wrote:
>>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to
>>>>>>>>>>>>>> run the
>>>>>>>>>>>>>> build and
>>>>>>>>>>>>>> test process under Python 3. Any committer can edit the
>>>>>>>>>>>>>> buildbot
>>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>>>>>> has Python 3
>>>>>>>>>>>>>> installed?
>>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>>>>
>>>>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing
>>>>>>>>>>>> on that
>>>>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>>>>> I don't know. It's possible, I suppose, that the activation of
>>>>>>>>>>> the
>>>>>>>>>>> python3 virtual environment has no effect on the test
>>>>>>>>>>> driver. It
>>>>>>>>>>> might
>>>>>>>>>>> not be a bad idea to have the tests print the Python version
>>>>>>>>>>> in the
>>>>>>>>>>> summary and possibly in the log of every test case.
>>>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the
>>>>>>>>>> «check»
>>>>>>>>>> target uses.
>>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes
>>>>>>>>> now,
>>>>>>>>> along
>>>>>>>>> with r1868566 for good measure.
>>>>>>>> I've set up this:
>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>>>>
>>>>>>>> It will build and test the core libraries and swig-py bindings
>>>>>>>> with
>>>>>>>> Python 3 on the swig-py3 branch.
>>>>>>>>
>>>>>>>> I hope ... build #0 running as we speak.
>>>>>>> Unfortunately build #2, which ran after upgrading SWIG and Python,
>>>>>>> failed
>>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>>>>> SVN-4818.
>>>>>>>
>>>>>>> This also affects on trunk with SWIG 4.0.
>>>>>>> (e.g.
>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>>>>
>>>>>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be
>>>>>>> produced,
>>>>>>> but the modules don't work correctly.
>>>>>>>
>>>>>>> It seems they were caused by incompatibility of Python code for
>>>>>>> proxy
>>>>>>> object generated by SWIG, and it can not be resolved so simple....
>>>>>>> (importlib module vs simply use 'import', absense of
>>>>>>> _swig_setattr()
>>>>>>> and _swig_getattr(), etc.)
>>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>>>>>
>>>>> Hm, that did not really help. On the swig-py3 branch:
>>>>>
>>>>> /bin/sh: line 1: 62481 Segmentation fault: 11
>>>>> /srv/virtualenv-3/bin/python
>>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>>>>>
>>>>> make: *** [check-swig-py] Error 139
>>>>>
>>>>>
>>>>> So, the Python bindings seem to have crashed the Python 3
>>>>> interpreter on
>>>>> macOS?
>>>>
>>>> Yes, they did:
>>>>
>>>> Process 28278 stopped
>>>> * thread #2, queue = 'com.apple.main-thread', stop reason =
>>>> EXC_BAD_ACCESS (code=1, address=0x48)
>>>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
>>>> Python`PyErr_Restore:
>>>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
>>>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
>>>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
>>>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
>>>> Target 0: (Python) stopped.
>>>>
>>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
>>>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
>>>> 40156 #endif
>>>> 40157
>>>> -> 40158 md = d = PyModule_GetDict(m);
>>>> 40159 (void)md;
>>>> 40160
>>>> 40161 SWIG_InitializeModule(0);
>>>> (lldb) print m
>>>> (PyObject *) $0 = 0x000000010279ad70
>>>> (lldb) print *m
>>>> (PyObject) $1 = {
>>>> ob_refcnt = 785
>>>> ob_type = 0x000000010026ea30
>>>> }
>>>
>>> I guess the _core module linked incompatible Python library to execute.
>>>
>>> on
>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio
>>>
>>> :
>>> [[[
>>> checking for compiling Python extensions... clang
>>> checking for linking Python extensions... clang -bundle -undefined
>>> dynamic_lookup -isysroot
>>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
>>>
>>> -F/usr/local/opt/python/Frameworks -framework Python
>>> checking for linking Python libraries... -bundle -undefined
>>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
>>> ]]]
>>
>>
>> Good catch. This certainly looks correct, it's where Homebrew installs
>> python. However ...
>>
>> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so
>> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so:
>> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
>> (compatibility version 2.7.0, current version 2.7.10)
>> ...
>>
>>
>> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so that it
>> finds the Python 3 installation.
(Actually that would be DYLD_FRAMEWORK_PATH.)
> ... or have to use Run-Path Dependent Libraries feature on link time,
> if DYLD_LIBRARY_PATH cannot work well due to SIP.
>
> https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
>
>
> The link option in log above comes from result of
> 'build/get-py-info.py --link', so should we tweak build/get-py-info.py,
> or add option to configure to pass Python dynamic library path?
Using RPATH for the Python framework on macOS would be the right
solution, but it doesn't solve the problem. It looks like we need some
magic incantation on macOS to explicitly pick the Python 3 framework;
using just the -F option doesn't work as expected, the path to the
system default Python 2.7 framework is hard-coded in to the extension
libraries.
-- Brane
Re: SWIG Python bindings build with SWIG 4.0
Posted by Yasuhito FUTATSUKI <fu...@poem.co.jp>.
On 2019/10/20 8:37, Branko Čibej wrote:
> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote:
>> On 2019/10/20 6:59, Branko Čibej wrote:
>>> On 19.10.2019 23:06, Branko Čibej wrote:
>>>> On 19.10.2019 19:55, Branko Čibej wrote:
>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>>>>>>> build and
>>>>>>>>>>>>> test process under Python 3. Any committer can edit the
>>>>>>>>>>>>> buildbot
>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>>>>> has Python 3
>>>>>>>>>>>>> installed?
>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>>>
>>>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing
>>>>>>>>>>> on that
>>>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>>>> I don't know. It's possible, I suppose, that the activation of
>>>>>>>>>> the
>>>>>>>>>> python3 virtual environment has no effect on the test driver. It
>>>>>>>>>> might
>>>>>>>>>> not be a bad idea to have the tests print the Python version
>>>>>>>>>> in the
>>>>>>>>>> summary and possibly in the log of every test case.
>>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the
>>>>>>>>> «check»
>>>>>>>>> target uses.
>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>>>>>>> along
>>>>>>>> with r1868566 for good measure.
>>>>>>> I've set up this:
>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>>>
>>>>>>> It will build and test the core libraries and swig-py bindings with
>>>>>>> Python 3 on the swig-py3 branch.
>>>>>>>
>>>>>>> I hope ... build #0 running as we speak.
>>>>>> Unfortunately build #2, which ran after upgrading SWIG and Python,
>>>>>> failed
>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>>>> SVN-4818.
>>>>>>
>>>>>> This also affects on trunk with SWIG 4.0.
>>>>>> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>>>
>>>>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be
>>>>>> produced,
>>>>>> but the modules don't work correctly.
>>>>>>
>>>>>> It seems they were caused by incompatibility of Python code for proxy
>>>>>> object generated by SWIG, and it can not be resolved so simple....
>>>>>> (importlib module vs simply use 'import', absense of _swig_setattr()
>>>>>> and _swig_getattr(), etc.)
>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>>>>
>>>> Hm, that did not really help. On the swig-py3 branch:
>>>>
>>>> /bin/sh: line 1: 62481 Segmentation fault: 11
>>>> /srv/virtualenv-3/bin/python
>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>>>> make: *** [check-swig-py] Error 139
>>>>
>>>>
>>>> So, the Python bindings seem to have crashed the Python 3
>>>> interpreter on
>>>> macOS?
>>>
>>> Yes, they did:
>>>
>>> Process 28278 stopped
>>> * thread #2, queue = 'com.apple.main-thread', stop reason =
>>> EXC_BAD_ACCESS (code=1, address=0x48)
>>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
>>> Python`PyErr_Restore:
>>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
>>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
>>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
>>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
>>> Target 0: (Python) stopped.
>>>
>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
>>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
>>> 40156 #endif
>>> 40157
>>> -> 40158 md = d = PyModule_GetDict(m);
>>> 40159 (void)md;
>>> 40160
>>> 40161 SWIG_InitializeModule(0);
>>> (lldb) print m
>>> (PyObject *) $0 = 0x000000010279ad70
>>> (lldb) print *m
>>> (PyObject) $1 = {
>>> ob_refcnt = 785
>>> ob_type = 0x000000010026ea30
>>> }
>>
>> I guess the _core module linked incompatible Python library to execute.
>>
>> on
>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio
>> :
>> [[[
>> checking for compiling Python extensions... clang
>> checking for linking Python extensions... clang -bundle -undefined
>> dynamic_lookup -isysroot
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
>> -F/usr/local/opt/python/Frameworks -framework Python
>> checking for linking Python libraries... -bundle -undefined
>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
>> ]]]
>
>
> Good catch. This certainly looks correct, it's where Homebrew installs
> python. However ...
>
> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so
> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so:
> /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.10)
> ...
>
>
> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so that it
> finds the Python 3 installation.
... or have to use Run-Path Dependent Libraries feature on link time,
if DYLD_LIBRARY_PATH cannot work well due to SIP.
https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
The link option in log above comes from result of
'build/get-py-info.py --link', so should we tweak build/get-py-info.py,
or add option to configure to pass Python dynamic library path?
Cheers,
--
Yasuhito FUTATSUKI <fu...@poem.co.jp>
Re: SWIG Python bindings build with SWIG 4.0
Posted by Branko Čibej <br...@apache.org>.
On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote:
> On 2019/10/20 6:59, Branko Čibej wrote:
>> On 19.10.2019 23:06, Branko Čibej wrote:
>>> On 19.10.2019 19:55, Branko Čibej wrote:
>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>>>>>> build and
>>>>>>>>>>>> test process under Python 3. Any committer can edit the
>>>>>>>>>>>> buildbot
>>>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>>>> has Python 3
>>>>>>>>>>>> installed?
>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>>
>>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing
>>>>>>>>>> on that
>>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>>> I don't know. It's possible, I suppose, that the activation of
>>>>>>>>> the
>>>>>>>>> python3 virtual environment has no effect on the test driver. It
>>>>>>>>> might
>>>>>>>>> not be a bad idea to have the tests print the Python version
>>>>>>>>> in the
>>>>>>>>> summary and possibly in the log of every test case.
>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the
>>>>>>>> «check»
>>>>>>>> target uses.
>>>>>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>>>>>> along
>>>>>>> with r1868566 for good measure.
>>>>>> I've set up this:
>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>>
>>>>>> It will build and test the core libraries and swig-py bindings with
>>>>>> Python 3 on the swig-py3 branch.
>>>>>>
>>>>>> I hope ... build #0 running as we speak.
>>>>> Unfortunately build #2, which ran after upgrading SWIG and Python,
>>>>> failed
>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>>> SVN-4818.
>>>>>
>>>>> This also affects on trunk with SWIG 4.0.
>>>>> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>>
>>>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be
>>>>> produced,
>>>>> but the modules don't work correctly.
>>>>>
>>>>> It seems they were caused by incompatibility of Python code for proxy
>>>>> object generated by SWIG, and it can not be resolved so simple....
>>>>> (importlib module vs simply use 'import', absense of _swig_setattr()
>>>>> and _swig_getattr(), etc.)
>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>>>
>>> Hm, that did not really help. On the swig-py3 branch:
>>>
>>> /bin/sh: line 1: 62481 Segmentation fault: 11
>>> /srv/virtualenv-3/bin/python
>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>>> make: *** [check-swig-py] Error 139
>>>
>>>
>>> So, the Python bindings seem to have crashed the Python 3
>>> interpreter on
>>> macOS?
>>
>> Yes, they did:
>>
>> Process 28278 stopped
>> * thread #2, queue = 'com.apple.main-thread', stop reason =
>> EXC_BAD_ACCESS (code=1, address=0x48)
>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
>> Python`PyErr_Restore:
>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
>> Target 0: (Python) stopped.
>>
>> frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
>> 40156 #endif
>> 40157
>> -> 40158 md = d = PyModule_GetDict(m);
>> 40159 (void)md;
>> 40160
>> 40161 SWIG_InitializeModule(0);
>> (lldb) print m
>> (PyObject *) $0 = 0x000000010279ad70
>> (lldb) print *m
>> (PyObject) $1 = {
>> ob_refcnt = 785
>> ob_type = 0x000000010026ea30
>> }
>
> I guess the _core module linked incompatible Python library to execute.
>
> on
> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio
> :
> [[[
> checking for compiling Python extensions... clang
> checking for linking Python extensions... clang -bundle -undefined
> dynamic_lookup -isysroot
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
> -F/usr/local/opt/python/Frameworks -framework Python
> checking for linking Python libraries... -bundle -undefined
> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
> ]]]
Good catch. This certainly looks correct, it's where Homebrew installs
python. However ...
$ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so
/opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so:
/System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.10)
...
That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so that it
finds the Python 3 installation.
-- Brane
Re: SWIG Python bindings build with SWIG 4.0
Posted by Yasuhito FUTATSUKI <fu...@poem.co.jp>.
On 2019/10/20 6:59, Branko Čibej wrote:
> On 19.10.2019 23:06, Branko Čibej wrote:
>> On 19.10.2019 19:55, Branko Čibej wrote:
>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>>>>> build and
>>>>>>>>>>> test process under Python 3. Any committer can edit the buildbot
>>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>>> has Python 3
>>>>>>>>>>> installed?
>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>>
>>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing on that
>>>>>>>>> bot? They (currently) fail under python3.
>>>>>>>> I don't know. It's possible, I suppose, that the activation of the
>>>>>>>> python3 virtual environment has no effect on the test driver. It
>>>>>>>> might
>>>>>>>> not be a bad idea to have the tests print the Python version in the
>>>>>>>> summary and possibly in the log of every test case.
>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the «check»
>>>>>>> target uses.
>>>>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>>>>> along
>>>>>> with r1868566 for good measure.
>>>>> I've set up this:
>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>>
>>>>> It will build and test the core libraries and swig-py bindings with
>>>>> Python 3 on the swig-py3 branch.
>>>>>
>>>>> I hope ... build #0 running as we speak.
>>>> Unfortunately build #2, which ran after upgrading SWIG and Python, failed
>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>>> SVN-4818.
>>>>
>>>> This also affects on trunk with SWIG 4.0.
>>>> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>>
>>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be produced,
>>>> but the modules don't work correctly.
>>>>
>>>> It seems they were caused by incompatibility of Python code for proxy
>>>> object generated by SWIG, and it can not be resolved so simple....
>>>> (importlib module vs simply use 'import', absense of _swig_setattr()
>>>> and _swig_getattr(), etc.)
>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>>
>> Hm, that did not really help. On the swig-py3 branch:
>>
>> /bin/sh: line 1: 62481 Segmentation fault: 11 /srv/virtualenv-3/bin/python /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
>> make: *** [check-swig-py] Error 139
>>
>>
>> So, the Python bindings seem to have crashed the Python 3 interpreter on
>> macOS?
>
> Yes, they did:
>
> Process 28278 stopped
> * thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48)
> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
> Python`PyErr_Restore:
> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
> Target 0: (Python) stopped.
>
> frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
> 40156 #endif
> 40157
> -> 40158 md = d = PyModule_GetDict(m);
> 40159 (void)md;
> 40160
> 40161 SWIG_InitializeModule(0);
> (lldb) print m
> (PyObject *) $0 = 0x000000010279ad70
> (lldb) print *m
> (PyObject) $1 = {
> ob_refcnt = 785
> ob_type = 0x000000010026ea30
> }
I guess the _core module linked incompatible Python library to execute.
on https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio :
[[[
checking for compiling Python extensions... clang
checking for linking Python extensions... clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -F/usr/local/opt/python/Frameworks -framework Python
checking for linking Python libraries... -bundle -undefined dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python
]]]
Is this correct?
--
Yasuhito FUTATSUKI <fu...@poem.co.jp>
Re: SWIG Python bindings build with SWIG 4.0 (was: Re: Issue tracker
housecleaning: SVN-1722)
Posted by Branko Čibej <br...@apache.org>.
On 19.10.2019 23:06, Branko Čibej wrote:
> On 19.10.2019 19:55, Branko Čibej wrote:
>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>>> On 2019/10/18 8:39, Branko Čibej wrote:
>>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>>>> build and
>>>>>>>>>> test process under Python 3. Any committer can edit the buildbot
>>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>>> has Python 3
>>>>>>>>>> installed?
>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>>
>>>>>>>>> Been there, done that, bought the DVD.
>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing on that
>>>>>>>> bot? They (currently) fail under python3.
>>>>>>> I don't know. It's possible, I suppose, that the activation of the
>>>>>>> python3 virtual environment has no effect on the test driver. It
>>>>>>> might
>>>>>>> not be a bad idea to have the tests print the Python version in the
>>>>>>> summary and possibly in the log of every test case.
>>>>>> What's the value of $(PYTHON) in Makefile? That's what the «check»
>>>>>> target uses.
>>>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>>>> along
>>>>> with r1868566 for good measure.
>>>> I've set up this:
>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>>
>>>> It will build and test the core libraries and swig-py bindings with
>>>> Python 3 on the swig-py3 branch.
>>>>
>>>> I hope ... build #0 running as we speak.
>>> Unfortunately build #2, which ran after upgrading SWIG and Python, failed
>>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>>> SVN-4818.
>>>
>>> This also affects on trunk with SWIG 4.0.
>>> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>>
>>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be produced,
>>> but the modules don't work correctly.
>>>
>>> It seems they were caused by incompatibility of Python code for proxy
>>> object generated by SWIG, and it can not be resolved so simple....
>>> (importlib module vs simply use 'import', absense of _swig_setattr()
>>> and _swig_getattr(), etc.)
>> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
>
> Hm, that did not really help. On the swig-py3 branch:
>
> /bin/sh: line 1: 62481 Segmentation fault: 11 /srv/virtualenv-3/bin/python /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
> make: *** [check-swig-py] Error 139
>
>
> So, the Python bindings seem to have crashed the Python 3 interpreter on
> macOS?
Yes, they did:
Process 28278 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48)
frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61
Python`PyErr_Restore:
-> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi
0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13
0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12
0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx)
Target 0: (Python) stopped.
frame #3: 0x0000000103800fea _core.so`PyInit__core at core.c:40158:12
40155 m = Py_InitModule((char *) SWIG_name, SwigMethods);
40156 #endif
40157
-> 40158 md = d = PyModule_GetDict(m);
40159 (void)md;
40160
40161 SWIG_InitializeModule(0);
(lldb) print m
(PyObject *) $0 = 0x000000010279ad70
(lldb) print *m
(PyObject) $1 = {
ob_refcnt = 785
ob_type = 0x000000010026ea30
}
Full traceback attached, if that helps.
-- Brane
Re: SWIG Python bindings build with SWIG 4.0 (was: Re: Issue tracker
housecleaning: SVN-1722)
Posted by Branko Čibej <br...@apache.org>.
On 19.10.2019 19:55, Branko Čibej wrote:
> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
>> On 2019/10/18 8:39, Branko Čibej wrote:
>>> On 17.10.2019 23:46, Branko Čibej wrote:
>>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>>> build and
>>>>>>>>> test process under Python 3. Any committer can edit the buildbot
>>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>>> has Python 3
>>>>>>>>> installed?
>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>>
>>>>>>>> Been there, done that, bought the DVD.
>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing on that
>>>>>>> bot? They (currently) fail under python3.
>>>>>> I don't know. It's possible, I suppose, that the activation of the
>>>>>> python3 virtual environment has no effect on the test driver. It
>>>>>> might
>>>>>> not be a bad idea to have the tests print the Python version in the
>>>>>> summary and possibly in the log of every test case.
>>>>> What's the value of $(PYTHON) in Makefile? That's what the «check»
>>>>> target uses.
>>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>>> along
>>>> with r1868566 for good measure.
>>> I've set up this:
>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>>
>>> It will build and test the core libraries and swig-py bindings with
>>> Python 3 on the swig-py3 branch.
>>>
>>> I hope ... build #0 running as we speak.
>> Unfortunately build #2, which ran after upgrading SWIG and Python, failed
>> to build SWIG Python bindings because of SWIG 4.0, as reported on
>> SVN-4818.
>>
>> This also affects on trunk with SWIG 4.0.
>> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>>
>> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be produced,
>> but the modules don't work correctly.
>>
>> It seems they were caused by incompatibility of Python code for proxy
>> object generated by SWIG, and it can not be resolved so simple....
>> (importlib module vs simply use 'import', absense of _swig_setattr()
>> and _swig_getattr(), etc.)
> My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
Hm, that did not really help. On the swig-py3 branch:
/bin/sh: line 1: 62481 Segmentation fault: 11 /srv/virtualenv-3/bin/python /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py
make: *** [check-swig-py] Error 139
So, the Python bindings seem to have crashed the Python 3 interpreter on
macOS?
(I installed the python3 package from Homebrew; it works perfectly on my
laptop.)
-- Brane
Re: SWIG Python bindings build with SWIG 4.0 (was: Re: Issue tracker
housecleaning: SVN-1722)
Posted by Branko Čibej <br...@apache.org>.
On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote:
> On 2019/10/18 8:39, Branko Čibej wrote:
>> On 17.10.2019 23:46, Branko Čibej wrote:
>>> On 17.10.2019 23:01, Daniel Shahaf wrote:
>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200:
>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote:
>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00:
>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote:
>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to run the
>>>>>>>> build and
>>>>>>>> test process under Python 3. Any committer can edit the buildbot
>>>>>>>> scripts[1], but the question is which of the buildbot slaves
>>>>>>>> has Python 3
>>>>>>>> installed?
>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3
>>>>>>>
>>>>>>> Been there, done that, bought the DVD.
>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing on that
>>>>>> bot? They (currently) fail under python3.
>>>>> I don't know. It's possible, I suppose, that the activation of the
>>>>> python3 virtual environment has no effect on the test driver. It
>>>>> might
>>>>> not be a bad idea to have the tests print the Python version in the
>>>>> summary and possibly in the log of every test case.
>>>> What's the value of $(PYTHON) in Makefile? That's what the «check»
>>>> target uses.
>>>
>>> Yep, apparently that's the bug ... I'm testing script changes now,
>>> along
>>> with r1868566 for good measure.
>>
>> I've set up this:
>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig
>>
>> It will build and test the core libraries and swig-py bindings with
>> Python 3 on the swig-py3 branch.
>>
>> I hope ... build #0 running as we speak.
>
> Unfortunately build #2, which ran after upgrading SWIG and Python, failed
> to build SWIG Python bindings because of SWIG 4.0, as reported on
> SVN-4818.
>
> This also affects on trunk with SWIG 4.0.
> (e.g. https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418)
>
> With attached patch on trunk (trunk_build_with_swig4_patch.txt) and
> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be produced,
> but the modules don't work correctly.
>
> It seems they were caused by incompatibility of Python code for proxy
> object generated by SWIG, and it can not be resolved so simple....
> (importlib module vs simply use 'import', absense of _swig_setattr()
> and _swig_getattr(), etc.)
My bad. I downgraded to Swig 3.0.12 and restarted the failed builds.
-- Brane