You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Jiri Daněk (Jira)" <ji...@apache.org> on 2021/04/28 10:17:00 UTC
[jira] [Comment Edited] (PROTON-2095) Move away from SWIG to CFFI
[ https://issues.apache.org/jira/browse/PROTON-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325666#comment-17325666 ]
Jiri Daněk edited comment on PROTON-2095 at 4/28/21, 10:16 AM:
---------------------------------------------------------------
h3. My CFFI references
[https://medium.com/paypal-tech/python-by-the-c-side-1f82fc94c4ad]
Some hints about Setuptools, [https://carsonip.me/posts/writing-a-python-wrapper-for-html2text-using-cffi/]
We want to use “out-of-line” + “API mode” CFFI with setuptools, [https://cffi.readthedocs.io/en/release-1.9/cdef.html]
Worked example for a relatively small C API [https://aldnav.com/blog/writing-a-geohash-wrapper-using-cffi/]
Python 3 strings [https://cffi.readthedocs.io/en/release-1.9/using.html?highlight=re#python-3-support]
Automatic generation of .pyi typing stub files [https://groups.google.com/g/python-cffi/c/6V8PBjA6ZJ4]
Slowest part of cffi seems to be processing the data in interpreted code, in and out (will be better in PyPy?) [https://blog.ian.stapletoncordas.co/2018/01/making-python-faster-with-rust-and-cffi-or-not.html]
More perf, Cython vs cffi [https://zpz.github.io/blog/speeding-up-python-with-c-and-cffi/]
h4. Build
Proton now has two modes. First, swig generates c code, then the cmake swig module compiles that; used in sysinstall binding and in running tests. Second, setuptools either locates installed Proton in the system or builds it from sources and creates a wheel.
I dislike the mode of working of pip install picking up systemwide library and building the binding for it. But cffi seems to be fine with it, it even has pkgconfig integration on Linux. I'd prefer using CMake even in building the wheel, but I am not sure how practical that would be.
Python has multiple toolsets to build packages: distutils (1st party), setuptools (2nd party, sort of) (https://setuptools.readthedocs.io/en/stable), and 3rd party scikit-build (https://scikit-build.readthedocs.io/en/latest), flit, hatch, poetry, and other
Building a minimal custom tooling is possible too, e.g. like the PyZMQ code currently used in Proton. https://martinopilia.com/posts/2018/09/15/building-python-extension.html https://gist.github.com/hovren/5b62175731433c741d07ee6f482e2936 https://stackoverflow.com/questions/42585210/extending-setuptools-extension-to-use-cmake-in-setup-py https://pypi.org/project/cmaketools/
Packaging tutorials: https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/guides/distributing-packages-using-setuptools/ https://cffi.readthedocs.io/en/latest/cdef.html#preparing-and-distributing-modules https://www.benjack.io/2018/02/02/python-cpp-revisited.html
[~astitcher] btw, https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ ;P
keep using tox to run python tests
setup.py can take arguments? from https://stackoverflow.com/questions/16787649/how-to-configure-python-cffi-library-to-use-mingw
{code}
cd cffi-src-dir
python setup.py config --compiler=mingw32 build --compiler=mingw32 install
cd ..
{code}
was (Author: jdanek):
h3. My CFFI references
[https://medium.com/paypal-tech/python-by-the-c-side-1f82fc94c4ad]
Some hints about Setuptools, [https://carsonip.me/posts/writing-a-python-wrapper-for-html2text-using-cffi/]
We want to use “out-of-line” + “API mode” CFFI with setuptools, [https://cffi.readthedocs.io/en/release-1.9/cdef.html]
Worked example for a relatively small C API [https://aldnav.com/blog/writing-a-geohash-wrapper-using-cffi/]
Python 3 strings [https://cffi.readthedocs.io/en/release-1.9/using.html?highlight=re#python-3-support]
Automatic generation of .pyi typing stub files [https://groups.google.com/g/python-cffi/c/6V8PBjA6ZJ4]
Slowest part of cffi seems to be processing the data in interpreted code, in and out (will be better in PyPy?) [https://blog.ian.stapletoncordas.co/2018/01/making-python-faster-with-rust-and-cffi-or-not.html]
More perf, Cython vs cffi [https://zpz.github.io/blog/speeding-up-python-with-c-and-cffi/]
> Move away from SWIG to CFFI
> ---------------------------
>
> Key: PROTON-2095
> URL: https://issues.apache.org/jira/browse/PROTON-2095
> Project: Qpid Proton
> Issue Type: Wish
> Components: python-binding
> Affects Versions: proton-c-future, proton-c-0.29.0
> Reporter: Omer Katz
> Priority: Major
> Fix For: proton-c-future
>
>
> SWIG is fine but we're not using it for anything other than exporting all of proton-c's API as is.
> Unfortunately SWIG only generates CPython extension bindings. This may be a problem on PyPy where CPython extensions are either slow or simply won't compile.
> Unlike SWIG, CFFI is portable both on CPython and PyPy.
> It also satisfies the same requirements as SWIG currently does.
> In addition, calls to CFFI simply release the GIL which will help parallelizing Python applications using threads.
> By using CFFI we can also get rid of all of our setup.py code and simply use it to build the extension. We will also no longer have problems building wheels.
> The newest version of CFFI supports pkg-config so we can use that to find proton-c easily.
> I'm willing to help with the refactor but I'll need a mentor since I'm not familiar with the code base.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org