You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "DylanLukes (via GitHub)" <gi...@apache.org> on 2023/10/22 20:52:41 UTC

[I] [CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS. [arrow]

DylanLukes opened a new issue, #38391:
URL: https://github.com/apache/arrow/issues/38391

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ### Minimal Reproduction
   
   A minimal reproduction and run is available here: https://github.com/DylanLukes/pyarrow-ci-macos13-minimal-repro/actions/runs/6606055123/job/17941866516
   
   ### Issue Description
   
   As PyArrow 13.0.0 has no wheels, I've been having to compile the Cython extensions for CI. The Arrow 13.0.0 header files make use of C++ features which are only available on macOS from 13 onwards, for example:
   
   ```c++
   /usr/local/include/arrow/type.h:1844:29: error: 'get<std::string, arrow::FieldPath, std::string, std::vector<arrow::FieldRef>>' is unavailable: introduced in macOS 10.13
             return IsName() ? &std::get<std::string>(impl_) : NULLPTR;
   ```
   
   It turns out that the `macos-latest` runner is, in fact, not the latest, that would be `macos-13`. Switching the runner to `macos-13` however doesn't fix the issue! What's going on? Inspecting the logs for my own project's failed CI:
   
   ```
   Installing project in development mode
     error: subprocess-exited-with-error
     
     × Building wheel for pyarrow (pyproject.toml) did not run successfully.
     │ exit code: 1
     ╰─> [468 lines of output]
         <string>:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
         WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
         running bdist_wheel
         running build
         running build_py
         creating build
         creating build/lib.macosx-10.9-universal2-cpython-[31](https://github.com/DylanLukes/renkon/actions/runs/6600268842/job/17930093683#step:9:32)2
         creating build/lib.macosx-10.9-universal2-cpython-312/pyarrow
         copying pyarrow/orc.py -> build/lib.macosx-10.9-universal2-cpython-312/pyarrow
         copying pyarrow/conftest.py -> build/lib.macosx-10.9-universal2-cpython-312/pyarrow
   ```
   
   Mac OS X... **_10.9?_**
   
   **It's not clear why, but `setuptools` seems to be inferring the incorrect system version, which in turn propagates down to CMake, which then results in the aforementioned compilation error.** Checking the system version in a few ways I get:
   
   ```
   > uname -a
   Darwin Mac-1697921125444.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:39:52 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_X86_64 x86_64
   
   > gcc -dumpmachine
   x86_64-apple-darwin22.6.0
   
   > python -c "import platform; print(platform.platform())"
   macOS-13.6-x86_64-i386-64bit
   ```
   
   ### Workaround
   
   While it doesn't fix `setuptools` thinking it's building a wheel for Mac OS X 10.9 rather than macOS 13, we _can_ get CMake back on the right page with the following extra step:
   
   ```yml
   ...
         - if: runner.os == 'macOS'
           run: echo "MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion)" >> $GITHUB_ENV
   ...
   ```
   
   ### Diagnostic Progress So Far 
   
   I was pretty baffled by this. I'm not sure where `setuptools` is getting 10.9 from. I've not had any issue like this on a local Mac running macOS 14, it _seems_ specific to the GitHub Actions runner environment? 
   
   I then went poking around the details of `pip inspect` and the contents of the packaged Python the setup step installs. AT first glance it appears to have been built against yet another different version (macOS 11), as it's marked as `python-3.12.0-macos11.pkg` in the accompanying `setup.sh`. 
   
   But, if we actually run the package (or expand it with `pkgutil --expand` and take a look inside...), we find that in the accompanying license, readme, and welcome messages it refers to 10.9!
   
   > ```
   > This package will install Python 3.12.0 for macOS 10.9 or later.
   > ```
   
   Aha! That explains it.
   
   ### Thoughts
   
   It seems to me that despite this being the case, building pyarrow _should work_. Yes, 10.9 is not supported... but, we're not on 10.9. And this build _can_ work perfectly well, so long as CMake is not misled about which `MACOSX_DEPLOYMENT_TARGET` to use. 
   
   So, if Arrow 13.0.0 _requires_ macOS 13 in its header files (it won't work on older systems anyways), it seems reasonable that `arrow/setup.py` should ensure that rather than deferring to a Python installation which may be misleading it.
   
   ### Component(s)
   
   C++, Continuous Integration, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS. [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche closed issue #38391: [CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS.
URL: https://github.com/apache/arrow/issues/38391


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS. [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #38391:
URL: https://github.com/apache/arrow/issues/38391#issuecomment-1774935541

   Hi Dylan,
   Thanks for the great diagnostic. Pyarrow 13.0.0 is not supposed to be Python 3.12 compatible. We are releasing Arrow 14.0.0 and we are providing wheels for Python 3.12, there which I do think should work, see:
   * https://github.com/apache/arrow/issues/37880
   
   If you want to give those wheels a try you can install them from our nightly builds, take into account that those are not official releases and we are in the process of releasing 14.0.0 which (unless something wrong happens on verification) could be available in the next days.
   See for the nightlies wheel installation: https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS. [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #38391:
URL: https://github.com/apache/arrow/issues/38391#issuecomment-2263485538

   Going to close this issue, as I don't think there is anything actionable left.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org