You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2024/05/02 09:49:21 UTC

[PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

jorisvandenbossche opened a new pull request, #41494:
URL: https://github.com/apache/arrow/pull/41494

   ### Rationale for this change
   
   Currently, when building pyarrow from source, one needs to manually enable the optional components through setting `PYARROW_WITH_...` environment variables. However, we could also make a default choice of components based on which ones where enabled in the Arrow C++ build.
   
   ### What changes are included in this PR?
   
   Set defaults for the various `PYARROW_BUILD_<component>` based on the `ARROW_<component>` setting. Keep the current `PYARROW_WITH_<component>` environment variables working to allow to override this default.
   
   One minor unfortunate consequence is that if you change your Arrow C++ build (eg enable a new component), you need to do a clean python build, otherwise CMake is still getting the old options (those option defaults are cached? Not sure if there is a way to solve this)
   
   ### Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ### Are there any user-facing changes?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2109641798

   @kou any other comments? 
   (I still need to remove some of the STATUS messages, but only want to do that when it is ready for merging, as that has been helpful while fixing/refactoring things)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2090311783

   @github-actions submit -g python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092829903

   One "feature" of our setup.py that I forgot about is that you actually can pass command line flags to setup.py's build_ext like `python setup.py build_ext --with-parquet`, and that also gets passed through to cmake right now. 
   
   Is that something we think we should keep working? It's nowhere used in our own codebase, and it is also not documented, except if you run `python setup.py build_ext --help`) 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600752816


##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   It does have `get_outputs`:
   
   ```
   In [9]: 'get_outputs' in dir(setuptools.command.build_ext.build_ext)
   Out[9]: True
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600368568


##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   It's used in `get_names` -> `get_outputs` methods. We don't use those ourselves, but I was assuming this might be part of the setuptools build_ext class interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587611052


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,44 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
+  option(PYARROW_BUILD_ACERO "Build the PyArrow Acero integration" ${ARROW_ACERO})
+  option(PYARROW_BUILD_CUDA "Build the PyArrow CUDA support" ${ARROW_CUDA})
+  option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" ${ARROW_DATASET})
+  option(PYARROW_BUILD_FLIGHT "Build the PyArrow Flight integration" ${ARROW_FLIGHT})
+  option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" ${ARROW_GANDIVA})
+  option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" ${ARROW_ORC})
+  option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" ${ARROW_PARQUET})
+  option(PYARROW_BUILD_PARQUET_ENCRYPTION
+         "Build the PyArrow Parquet encryption integration" ${PARQUET_REQUIRE_ENCRYPTION})
+  option(PYARROW_BUILD_SUBSTRAIT "Build the PyArrow Substrait integration"
+         ${ARROW_SUBSTRAIT})
+  option(PYARROW_BUILD_AZURE "Build the PyArrow Azure integration" ${ARROW_AZURE})
+  option(PYARROW_BUILD_GCS "Build the PyArrow GCS integration" ${ARROW_GCS})
+  option(PYARROW_BUILD_S3 "Build the PyArrow S3 integration" ${ARROW_S3})
+  option(PYARROW_BUILD_HDFS "Build the PyArrow HDFS integration" ${ARROW_HDFS})
+  option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
+  option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
+  option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
+  set(PYARROW_CXXFLAGS
+      ""
+      CACHE STRING "Compiler flags to append when compiling Arrow")

Review Comment:
   ```suggestion
         CACHE STRING "Compiler flags to append when compiling PyArrow C++")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092832830

   > It's nowhere used in our own codebase, and it is also not documented, except if you run `python setup.py build_ext --help`)
   
   We did document it in the past, though (https://github.com/apache/arrow/blob/8ca41384b5324bfd0ef3d3ed3f728e1d10ed73f0/docs/source/python/development.rst?plain=1#L208-L215), and so this probably is still used quite a bit in the wild (one of the first google searches eg gives this gist that uses it: https://gist.github.com/cpcloud/9702e1de6e234989d215ff00f3ab728b
   
   I will add it back here for now, and then we can still see if we want to deprecate this separately or not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092753114

   @github-actions crossbow submit -g python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587672427


##########
ci/scripts/python_build.sh:
##########
@@ -58,7 +58,7 @@ fi
 export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
 export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
 
-export PYARROW_WITH_ACERO=${ARROW_ACERO:-OFF}
+export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}

Review Comment:
   If possible, I would rather we make them both off by default, especially as they are non-trivial and take time to build. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600366741


##########
python/setup.py:
##########
@@ -152,48 +152,27 @@ def initialize_options(self):
             if not hasattr(sys, 'gettotalrefcount'):
                 self.build_type = 'release'
 
-        self.with_azure = strtobool(
-            os.environ.get('PYARROW_WITH_AZURE', '0'))
-        self.with_gcs = strtobool(
-            os.environ.get('PYARROW_WITH_GCS', '0'))
-        self.with_s3 = strtobool(
-            os.environ.get('PYARROW_WITH_S3', '0'))
-        self.with_hdfs = strtobool(
-            os.environ.get('PYARROW_WITH_HDFS', '0'))
-        self.with_cuda = strtobool(
-            os.environ.get('PYARROW_WITH_CUDA', '0'))
-        self.with_substrait = strtobool(
-            os.environ.get('PYARROW_WITH_SUBSTRAIT', '0'))
-        self.with_flight = strtobool(
-            os.environ.get('PYARROW_WITH_FLIGHT', '0'))
-        self.with_acero = strtobool(
-            os.environ.get('PYARROW_WITH_ACERO', '0'))
-        self.with_dataset = strtobool(
-            os.environ.get('PYARROW_WITH_DATASET', '0'))
-        self.with_parquet = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET', '0'))
-        self.with_parquet_encryption = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
-        self.with_orc = strtobool(
-            os.environ.get('PYARROW_WITH_ORC', '0'))
-        self.with_gandiva = strtobool(
-            os.environ.get('PYARROW_WITH_GANDIVA', '0'))
+        self.with_azure = None

Review Comment:
   Yes, they are still used in case someone is passing that to setup.py. See https://github.com/apache/arrow/pull/41494#issuecomment-2092832830
   
   I think that's something we should deprecate, though. But planning to do a separate follow-up PR to that, since it's not actually related to this PR (it's just another way that someone can right now override the default, just as setting an environment variable)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1588416238


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,44 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")

Review Comment:
   It's for avoiding defining our options when this is used via `add_subdirectory()` (or [`FetchContent`](https://cmake.org/cmake/help/latest/module/FetchContent.html)).
   
   We don't need this because:
   
   * This is never used via `add_subdirectory()`
   * Our option definitions are harmless with `add_subdirectory()` use



##########
python/setup.py:
##########
@@ -270,23 +235,28 @@ def append_cmake_bool(value, varname):
                 cmake_options.append('-D{0}={1}'.format(
                     varname, 'on' if value else 'off'))
 
+            def maybe_append_cmake_bool(varname):
+                env_var = varname.replace("PYARROW_BUILD", "PYARROW_WITH")
+                if env_var in os.environ:
+                    value = strtobool(os.environ.get(env_var))
+                    append_cmake_bool(value, varname)

Review Comment:
   How about doing this in `CMakeLists.txt` too?
   We can refer an environment variable by `"$ENV{PYARROW_WITH_CUDA}` in `CMakeLists.txt`.
   See also: https://cmake.org/cmake/help/latest/variable/ENV.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2090046765

   :warning: GitHub issue #41480 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587672961


##########
ci/scripts/python_build.sh:
##########
@@ -58,7 +58,7 @@ fi
 export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
 export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
 
-export PYARROW_WITH_ACERO=${ARROW_ACERO:-OFF}
+export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}

Review Comment:
   (this change should also not break anything for users as these are CI scripts)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2099829072

   Revision: f9b2b3c207d7b21b3c4b1a361aa85d149537a6cb
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-92fbc43a42](https://github.com/ursacomputing/crossbow/branches/all?query=actions-92fbc43a42)
   
   |Task|Status|
   |----|------|
   |example-python-minimal-build-fedora-conda|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-example-python-minimal-build-fedora-conda)](https://github.com/ursacomputing/crossbow/actions/runs/8997260775/job/24715081437)|
   |example-python-minimal-build-ubuntu-venv|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-example-python-minimal-build-ubuntu-venv)](https://github.com/ursacomputing/crossbow/actions/runs/8997261078/job/24715082995)|
   |python-sdist|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-python-sdist)](https://github.com/ursacomputing/crossbow/actions/runs/8997260660/job/24715080879)|
   |test-conda-python-3.10|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10)](https://github.com/ursacomputing/crossbow/actions/runs/8997260571/job/24715080282)|
   |test-conda-python-3.10-cython2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-cython2)](https://github.com/ursacomputing/crossbow/actions/runs/8997261054/job/24715082765)|
   |test-conda-python-3.10-hdfs-2.9.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-hdfs-2.9.2)](https://github.com/ursacomputing/crossbow/actions/runs/8997259989/job/24715078108)|
   |test-conda-python-3.10-hdfs-3.2.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-hdfs-3.2.1)](https://github.com/ursacomputing/crossbow/actions/runs/8997261709/job/24715112693)|
   |test-conda-python-3.10-pandas-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-pandas-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8997261719/job/24715112680)|
   |test-conda-python-3.10-pandas-nightly|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-pandas-nightly)](https://github.com/ursacomputing/crossbow/actions/runs/8997261222/job/24715083820)|
   |test-conda-python-3.10-spark-v3.5.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-spark-v3.5.0)](https://github.com/ursacomputing/crossbow/actions/runs/8997260424/job/24715079979)|
   |test-conda-python-3.10-substrait|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.10-substrait)](https://github.com/ursacomputing/crossbow/actions/runs/8997260453/job/24715080299)|
   |test-conda-python-3.11|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11)](https://github.com/ursacomputing/crossbow/actions/runs/8997260989/job/24715082948)|
   |test-conda-python-3.11-dask-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11-dask-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8997260853/job/24715081432)|
   |test-conda-python-3.11-dask-upstream_devel|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11-dask-upstream_devel)](https://github.com/ursacomputing/crossbow/actions/runs/8997261226/job/24715083387)|
   |test-conda-python-3.11-hypothesis|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11-hypothesis)](https://github.com/ursacomputing/crossbow/actions/runs/8997261152/job/24715083297)|
   |test-conda-python-3.11-pandas-upstream_devel|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11-pandas-upstream_devel)](https://github.com/ursacomputing/crossbow/actions/runs/8997261703/job/24715112562)|
   |test-conda-python-3.11-spark-master|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.11-spark-master)](https://github.com/ursacomputing/crossbow/actions/runs/8997260578/job/24715080287)|
   |test-conda-python-3.12|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.12)](https://github.com/ursacomputing/crossbow/actions/runs/8997261193/job/24715083151)|
   |test-conda-python-3.8|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.8)](https://github.com/ursacomputing/crossbow/actions/runs/8997260782/job/24715082144)|
   |test-conda-python-3.8-pandas-1.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.8-pandas-1.0)](https://github.com/ursacomputing/crossbow/actions/runs/8997260871/job/24715081357)|
   |test-conda-python-3.8-spark-v3.5.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.8-spark-v3.5.0)](https://github.com/ursacomputing/crossbow/actions/runs/8997261117/job/24715083642)|
   |test-conda-python-3.9|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.9)](https://github.com/ursacomputing/crossbow/actions/runs/8997261725/job/24715112790)|
   |test-conda-python-3.9-pandas-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-conda-python-3.9-pandas-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8997261543/job/24715112344)|
   |test-debian-12-python-3-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-debian-12-python-3-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260844/job/24715081318)|
   |test-debian-12-python-3-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-debian-12-python-3-i386)](https://github.com/ursacomputing/crossbow/actions/runs/8997261216/job/24715083358)|
   |test-fedora-39-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-fedora-39-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8997261286/job/24715083641)|
   |test-ubuntu-20.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-ubuntu-20.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8997260361/job/24715079343)|
   |test-ubuntu-22.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-test-ubuntu-22.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8997260835/job/24715081794)|
   |verify-rc-source-python-linux-almalinux-8-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-linux-almalinux-8-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997259959/job/24715078084)|
   |verify-rc-source-python-linux-conda-latest-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-linux-conda-latest-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997259960/job/24715078102)|
   |verify-rc-source-python-linux-ubuntu-20.04-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-linux-ubuntu-20.04-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997259958/job/24715078112)|
   |verify-rc-source-python-linux-ubuntu-22.04-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-linux-ubuntu-22.04-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260975/job/24715082628)|
   |verify-rc-source-python-macos-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-macos-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260066/job/24715078587)|
   |verify-rc-source-python-macos-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-macos-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260419/job/24715079800)|
   |verify-rc-source-python-macos-conda-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-verify-rc-source-python-macos-conda-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260757/job/24715081773)|
   |wheel-macos-big-sur-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-big-sur-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260903/job/24715081689)|
   |wheel-macos-big-sur-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-big-sur-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260131/job/24715078901)|
   |wheel-macos-big-sur-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-big-sur-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260285/job/24715078964)|
   |wheel-macos-big-sur-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-big-sur-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260612/job/24715080487)|
   |wheel-macos-big-sur-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-big-sur-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261133/job/24715083075)|
   |wheel-macos-catalina-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-catalina-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261453/job/24715112355)|
   |wheel-macos-catalina-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-catalina-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261077/job/24715082626)|
   |wheel-macos-catalina-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-catalina-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260753/job/24715082109)|
   |wheel-macos-catalina-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-catalina-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260769/job/24715081656)|
   |wheel-macos-catalina-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-macos-catalina-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261756/job/24715112719)|
   |wheel-manylinux-2-28-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260311/job/24715079349)|
   |wheel-manylinux-2-28-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261279/job/24715083513)|
   |wheel-manylinux-2-28-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260744/job/24715081971)|
   |wheel-manylinux-2-28-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260228/job/24715078970)|
   |wheel-manylinux-2-28-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997259986/job/24715078308)|
   |wheel-manylinux-2-28-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261810/job/24715112885)|
   |wheel-manylinux-2-28-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260736/job/24715081979)|
   |wheel-manylinux-2-28-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261579/job/24715112486)|
   |wheel-manylinux-2-28-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261507/job/24715112373)|
   |wheel-manylinux-2-28-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2-28-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260594/job/24715080457)|
   |wheel-manylinux-2014-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260367/job/24715079793)|
   |wheel-manylinux-2014-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261272/job/24715083551)|
   |wheel-manylinux-2014-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261572/job/24715112382)|
   |wheel-manylinux-2014-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261707/job/24715112553)|
   |wheel-manylinux-2014-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260928/job/24715081859)|
   |wheel-manylinux-2014-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260458/job/24715080304)|
   |wheel-manylinux-2014-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260771/job/24715082003)|
   |wheel-manylinux-2014-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260420/job/24715079799)|
   |wheel-manylinux-2014-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260340/job/24715079356)|
   |wheel-manylinux-2014-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-manylinux-2014-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260971/job/24715082692)|
   |wheel-windows-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-windows-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260074/job/24715078672)|
   |wheel-windows-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-windows-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260081/job/24715078675)|
   |wheel-windows-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-windows-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260402/job/24715079798)|
   |wheel-windows-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-windows-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997261890/job/24715112934)|
   |wheel-windows-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-92fbc43a42-github-wheel-windows-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8997260339/job/24715079557)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092872006

   Revision: 14101851cb9bf02c5d6372b3daea962c10ea99ca
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-e6753e2814](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e6753e2814)
   
   |Task|Status|
   |----|------|
   |python-sdist|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-python-sdist)](https://github.com/ursacomputing/crossbow/actions/runs/8938598433/job/24553056531)|
   |wheel-macos-big-sur-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-big-sur-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598252/job/24553055850)|
   |wheel-macos-big-sur-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-big-sur-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598339/job/24553056079)|
   |wheel-macos-big-sur-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-big-sur-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598486/job/24553056732)|
   |wheel-macos-big-sur-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-big-sur-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598272/job/24553056062)|
   |wheel-macos-big-sur-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-big-sur-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598693/job/24553057787)|
   |wheel-macos-catalina-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-catalina-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598349/job/24553055945)|
   |wheel-macos-catalina-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-catalina-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598040/job/24553055000)|
   |wheel-macos-catalina-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-catalina-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598320/job/24553055827)|
   |wheel-macos-catalina-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-catalina-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598001/job/24553054674)|
   |wheel-macos-catalina-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-macos-catalina-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598403/job/24553056530)|
   |wheel-manylinux-2-28-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598719/job/24553057780)|
   |wheel-manylinux-2-28-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598636/job/24553057218)|
   |wheel-manylinux-2-28-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598911/job/24553058725)|
   |wheel-manylinux-2-28-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598496/job/24553056731)|
   |wheel-manylinux-2-28-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598790/job/24553058437)|
   |wheel-manylinux-2-28-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598213/job/24553055484)|
   |wheel-manylinux-2-28-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597957/job/24553054354)|
   |wheel-manylinux-2-28-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597945/job/24553054375)|
   |wheel-manylinux-2-28-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598785/job/24553058460)|
   |wheel-manylinux-2-28-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2-28-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598897/job/24553058657)|
   |wheel-manylinux-2014-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597961/job/24553054528)|
   |wheel-manylinux-2014-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598199/job/24553055472)|
   |wheel-manylinux-2014-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598470/job/24553056549)|
   |wheel-manylinux-2014-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598468/job/24553056524)|
   |wheel-manylinux-2014-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598263/job/24553055783)|
   |wheel-manylinux-2014-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598828/job/24553058507)|
   |wheel-manylinux-2014-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598725/job/24553058004)|
   |wheel-manylinux-2014-cp38-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp38-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597938/job/24553054390)|
   |wheel-manylinux-2014-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598509/job/24553057206)|
   |wheel-manylinux-2014-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-manylinux-2014-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598807/job/24553058466)|
   |wheel-windows-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-windows-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598111/job/24553055194)|
   |wheel-windows-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-windows-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597989/job/24553054570)|
   |wheel-windows-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-windows-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598183/job/24553055463)|
   |wheel-windows-cp38-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-windows-cp38-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938597963/job/24553054337)|
   |wheel-windows-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-e6753e2814-github-wheel-windows-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8938598697/job/24553057783)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587608731


##########
ci/scripts/python_build.sh:
##########
@@ -58,7 +58,7 @@ fi
 export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
 export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
 
-export PYARROW_WITH_ACERO=${ARROW_ACERO:-OFF}
+export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}

Review Comment:
   I'm not sure why this would be on by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1593138237


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,77 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+macro(define_option name description arrow_option)
+  set("PYARROW_${name}"
+      "AUTO"
+      CACHE STRING ${description})
+
+  if("${PYARROW_${name}}" STREQUAL "AUTO")
+    # by default, first check if env variable exists, otherwise use Arrow C++ config
+    set(env_variable "PYARROW_WITH_${name}")
+    if(DEFINED ENV{${env_variable}})
+      message(STATUS "Env variable is defined: ${env_variable}=$ENV{${env_variable}}")
+      if($ENV{${env_variable}})
+        set("PYARROW_BUILD_${name}" ON)
+        message(STATUS "Setting ${name} to ON through env variable")
+      else()
+        set("PYARROW_BUILD_${name}" OFF)
+        message(STATUS "Setting ${name} to OFF through env variable")
+      endif()
+    else()
+      if(${arrow_option})
+        set("PYARROW_BUILD_${name}" ON)
+        message(STATUS "Setting ${name} to ON through Arrow C++ config")
+      else()
+        set("PYARROW_BUILD_${name}" OFF)
+        message(STATUS "Setting ${name} to OFF through Arrow C++ config")
+      endif()
+    endif()
+  else()
+    if("${PYARROW_${name}}")
+      set("PYARROW_BUILD_${name}" ON)
+      message(STATUS "Setting ${name} to ON through CMake option")
+    else()
+      set("PYARROW_BUILD_${name}" OFF)
+      message(STATUS "Setting ${name} to OFF through CMake option")
+    endif()
+  endif()
+endmacro()
+
+define_option(ACERO "Build the PyArrow Acero integration" ARROW_ACERO)
+define_option(CUDA "Build the PyArrow CUDA support" ARROW_CUDA)
+define_option(DATASET "Build the PyArrow Dataset integration"{ARROW_DATASET)

Review Comment:
   ```suggestion
   define_option(DATASET "Build the PyArrow Dataset integration" ARROW_DATASET)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1603181162


##########
.pre-commit-config.yaml:
##########
@@ -123,6 +123,7 @@ repos:
           ?^go/.*/CMakeLists\.txt$|
           ?^java/.*/CMakeLists\.txt$|
           ?^matlab/.*/CMakeLists\.txt$|
+          ?^python/CMakeLists\.txt$|

Review Comment:
   Opened https://github.com/apache/arrow/pull/41689 specifically for that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587650798


##########
ci/scripts/python_build.sh:
##########
@@ -58,7 +58,7 @@ fi
 export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
 export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
 
-export PYARROW_WITH_ACERO=${ARROW_ACERO:-OFF}
+export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}

Review Comment:
   Dataset is on by default (a few lines below), and so that in practice means Acero is on by default as well. Now, I changed this in the CMakeLists file to do that automatically, so I could also leave it here as is, but this seems more correct



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2090470240

   > One minor unfortunate consequence is that if you change your Arrow C++ build (eg enable a new component), you need to do a clean python build, otherwise CMake is still getting the old options (those option defaults are cached? Not sure if there is a way to solve this)
   
   It would be nice if this could be avoided. @kou does that sound possible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587735136


##########
python/CMakeLists.txt:
##########
@@ -305,6 +319,7 @@ set(PYARROW_CPP_LINK_LIBS "")
 
 # Check all the options from Arrow and PyArrow C++ to be in line
 if(PYARROW_BUILD_DATASET)
+  message(STATUS "Building PyArrow Dataset")

Review Comment:
   ```suggestion
     message(STATUS "Building PyArrow with dataset")
   ```
   
   (I'm not sure anybody would actually be confused by this, but the actual building doesn't happen here)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2090063646

   I have not yet cleaned up all our CI scripts that contain things like `export PYARROW_WITH_ACERO=${ARROW_ACERO:-OFF}`, which is now in theory no longer necessary. Although we might also want to keep some of those ones that are explicitly set to 1 to ensure we don't accidentally turn it of because of an Arrow C++ config change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092863651

   @github-actions crossbow submit wheel-* python-sdist


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600289742


##########
.pre-commit-config.yaml:
##########
@@ -123,6 +123,7 @@ repos:
           ?^go/.*/CMakeLists\.txt$|
           ?^java/.*/CMakeLists\.txt$|
           ?^matlab/.*/CMakeLists\.txt$|
+          ?^python/CMakeLists\.txt$|

Review Comment:
   Oh... These patterns are wrong...
   
   Could you use `^XXX/.*CMakeLists\.txt$|` for all existing patterns too?



##########
python/setup.py:
##########
@@ -152,48 +152,27 @@ def initialize_options(self):
             if not hasattr(sys, 'gettotalrefcount'):
                 self.build_type = 'release'
 
-        self.with_azure = strtobool(
-            os.environ.get('PYARROW_WITH_AZURE', '0'))
-        self.with_gcs = strtobool(
-            os.environ.get('PYARROW_WITH_GCS', '0'))
-        self.with_s3 = strtobool(
-            os.environ.get('PYARROW_WITH_S3', '0'))
-        self.with_hdfs = strtobool(
-            os.environ.get('PYARROW_WITH_HDFS', '0'))
-        self.with_cuda = strtobool(
-            os.environ.get('PYARROW_WITH_CUDA', '0'))
-        self.with_substrait = strtobool(
-            os.environ.get('PYARROW_WITH_SUBSTRAIT', '0'))
-        self.with_flight = strtobool(
-            os.environ.get('PYARROW_WITH_FLIGHT', '0'))
-        self.with_acero = strtobool(
-            os.environ.get('PYARROW_WITH_ACERO', '0'))
-        self.with_dataset = strtobool(
-            os.environ.get('PYARROW_WITH_DATASET', '0'))
-        self.with_parquet = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET', '0'))
-        self.with_parquet_encryption = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
-        self.with_orc = strtobool(
-            os.environ.get('PYARROW_WITH_ORC', '0'))
-        self.with_gandiva = strtobool(
-            os.environ.get('PYARROW_WITH_GANDIVA', '0'))
+        self.with_azure = None

Review Comment:
   Do we still need `self.with_XXX`? It seems that nobody sets them because we remove `os.environ.get(...)`.



##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   Do we need to prepare `self._found_names`?
   It seems that nobody uses it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600718225


##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   Hmm. It seems that the setuptools `build_ext` doesn't have `get_names` interface:
   
   ```
   >>> import setuptools.command.build_ext
   >>> setuptools.command.build_ext.build_ext
   <class 'setuptools.command.build_ext.build_ext'>
   >>> 'get_names' in dir(setuptools.command.build_ext.build_ext)
   False
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600719194


##########
python/setup.py:
##########
@@ -152,48 +152,27 @@ def initialize_options(self):
             if not hasattr(sys, 'gettotalrefcount'):
                 self.build_type = 'release'
 
-        self.with_azure = strtobool(
-            os.environ.get('PYARROW_WITH_AZURE', '0'))
-        self.with_gcs = strtobool(
-            os.environ.get('PYARROW_WITH_GCS', '0'))
-        self.with_s3 = strtobool(
-            os.environ.get('PYARROW_WITH_S3', '0'))
-        self.with_hdfs = strtobool(
-            os.environ.get('PYARROW_WITH_HDFS', '0'))
-        self.with_cuda = strtobool(
-            os.environ.get('PYARROW_WITH_CUDA', '0'))
-        self.with_substrait = strtobool(
-            os.environ.get('PYARROW_WITH_SUBSTRAIT', '0'))
-        self.with_flight = strtobool(
-            os.environ.get('PYARROW_WITH_FLIGHT', '0'))
-        self.with_acero = strtobool(
-            os.environ.get('PYARROW_WITH_ACERO', '0'))
-        self.with_dataset = strtobool(
-            os.environ.get('PYARROW_WITH_DATASET', '0'))
-        self.with_parquet = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET', '0'))
-        self.with_parquet_encryption = strtobool(
-            os.environ.get('PYARROW_WITH_PARQUET_ENCRYPTION', '0'))
-        self.with_orc = strtobool(
-            os.environ.get('PYARROW_WITH_ORC', '0'))
-        self.with_gandiva = strtobool(
-            os.environ.get('PYARROW_WITH_GANDIVA', '0'))
+        self.with_azure = None

Review Comment:
   Ah, we need them for `--with-XXX` options!
   I see.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #41494:
URL: https://github.com/apache/arrow/pull/41494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2099356475

   Could you rebase on main to use #41567?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587437033


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,44 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")

Review Comment:
   @kou for my education, could you explain what this `if` check is exactly doing or why we need it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600368568


##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   It's used in `get_names` -> `get_outputs` methods. We don't use those ourselves, but I am assuming this might be part of the setuptools build_ext class interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1600801281


##########
python/setup.py:
##########
@@ -329,54 +315,8 @@ def append_cmake_bool(value, varname):
             self._found_names = []
             for name in self.CYTHON_MODULE_NAMES:
                 built_path = pjoin(install_prefix, name + ext_suffix)
-                if not os.path.exists(built_path):
-                    print(f'Did not find {built_path}')
-                    if self._failure_permitted(name):
-                        print(f'Cython module {name} failure permitted')
-                        continue
-                    raise RuntimeError('PyArrow C-extension failed to build:',
-                                       os.path.abspath(built_path))
-
-                self._found_names.append(name)
-
-    def _failure_permitted(self, name):
-        if name == '_parquet' and not self.with_parquet:
-            return True
-        if name == '_parquet_encryption' and not self.with_parquet_encryption:
-            return True
-        if name == '_orc' and not self.with_orc:
-            return True
-        if name == '_flight' and not self.with_flight:
-            return True
-        if name == '_substrait' and not self.with_substrait:
-            return True
-        if name == '_azurefs' and not self.with_azure:
-            return True
-        if name == '_gcsfs' and not self.with_gcs:
-            return True
-        if name == '_s3fs' and not self.with_s3:
-            return True
-        if name == '_hdfs' and not self.with_hdfs:
-            return True
-        if name == '_dataset' and not self.with_dataset:
-            return True
-        if name == '_acero' and not self.with_acero:
-            return True
-        if name == '_exec_plan' and not self.with_acero:
-            return True
-        if name == '_dataset_orc' and not (
-                self.with_orc and self.with_dataset
-        ):
-            return True
-        if name == '_dataset_parquet' and not (
-                self.with_parquet and self.with_dataset
-        ):
-            return True
-        if name == '_cuda' and not self.with_cuda:
-            return True
-        if name == 'gandiva' and not self.with_gandiva:
-            return True
-        return False
+                if os.path.exists(built_path):
+                    self._found_names.append(name)

Review Comment:
   Ah, sorry. I checked wrong name...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2092756729

   Revision: f09699924382bf40efc5df14b978242fa1277dc4
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-47aab0586a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-47aab0586a)
   
   |Task|Status|
   |----|------|
   |example-python-minimal-build-fedora-conda|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-example-python-minimal-build-fedora-conda)](https://github.com/ursacomputing/crossbow/actions/runs/8937816880/job/24550754396)|
   |example-python-minimal-build-ubuntu-venv|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-example-python-minimal-build-ubuntu-venv)](https://github.com/ursacomputing/crossbow/actions/runs/8937817017/job/24550754898)|
   |test-conda-python-3.10|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10)](https://github.com/ursacomputing/crossbow/actions/runs/8937816892/job/24550754424)|
   |test-conda-python-3.10-cython2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-cython2)](https://github.com/ursacomputing/crossbow/actions/runs/8937816763/job/24550753804)|
   |test-conda-python-3.10-hdfs-2.9.2|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-hdfs-2.9.2)](https://github.com/ursacomputing/crossbow/actions/runs/8937816821/job/24550754107)|
   |test-conda-python-3.10-hdfs-3.2.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-hdfs-3.2.1)](https://github.com/ursacomputing/crossbow/actions/runs/8937816933/job/24550754643)|
   |test-conda-python-3.10-pandas-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-pandas-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8937816782/job/24550753802)|
   |test-conda-python-3.10-pandas-nightly|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-pandas-nightly)](https://github.com/ursacomputing/crossbow/actions/runs/8937817022/job/24550755012)|
   |test-conda-python-3.10-spark-v3.5.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-spark-v3.5.0)](https://github.com/ursacomputing/crossbow/actions/runs/8937816805/job/24550753992)|
   |test-conda-python-3.10-substrait|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.10-substrait)](https://github.com/ursacomputing/crossbow/actions/runs/8937816775/job/24550753856)|
   |test-conda-python-3.11|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11)](https://github.com/ursacomputing/crossbow/actions/runs/8937816873/job/24550754410)|
   |test-conda-python-3.11-dask-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11-dask-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8937816602/job/24550753073)|
   |test-conda-python-3.11-dask-upstream_devel|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11-dask-upstream_devel)](https://github.com/ursacomputing/crossbow/actions/runs/8937816877/job/24550754355)|
   |test-conda-python-3.11-hypothesis|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11-hypothesis)](https://github.com/ursacomputing/crossbow/actions/runs/8937816901/job/24550754574)|
   |test-conda-python-3.11-pandas-upstream_devel|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11-pandas-upstream_devel)](https://github.com/ursacomputing/crossbow/actions/runs/8937816857/job/24550754263)|
   |test-conda-python-3.11-spark-master|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.11-spark-master)](https://github.com/ursacomputing/crossbow/actions/runs/8937816832/job/24550754087)|
   |test-conda-python-3.12|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.12)](https://github.com/ursacomputing/crossbow/actions/runs/8937816718/job/24550753633)|
   |test-conda-python-3.8|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.8)](https://github.com/ursacomputing/crossbow/actions/runs/8937816580/job/24550753066)|
   |test-conda-python-3.8-pandas-1.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.8-pandas-1.0)](https://github.com/ursacomputing/crossbow/actions/runs/8937816664/job/24550753426)|
   |test-conda-python-3.8-spark-v3.5.0|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.8-spark-v3.5.0)](https://github.com/ursacomputing/crossbow/actions/runs/8937816599/job/24550753100)|
   |test-conda-python-3.9|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.9)](https://github.com/ursacomputing/crossbow/actions/runs/8937816578/job/24550753117)|
   |test-conda-python-3.9-pandas-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-conda-python-3.9-pandas-latest)](https://github.com/ursacomputing/crossbow/actions/runs/8937816979/job/24550754861)|
   |test-cuda-python|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-cuda-python)](https://github.com/ursacomputing/crossbow/actions/runs/8937816851/job/24550754147)|
   |test-debian-12-python-3-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-debian-12-python-3-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/8937816688/job/24550753465)|
   |test-debian-12-python-3-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-debian-12-python-3-i386)](https://github.com/ursacomputing/crossbow/actions/runs/8937816957/job/24550754741)|
   |test-fedora-39-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-fedora-39-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8937816636/job/24550753346)|
   |test-ubuntu-20.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-ubuntu-20.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8937816950/job/24550754751)|
   |test-ubuntu-22.04-python-3|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-47aab0586a-github-test-ubuntu-22.04-python-3)](https://github.com/ursacomputing/crossbow/actions/runs/8937816739/job/24550753689)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1589115180


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,77 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+macro(define_option name description arrow_option)
+  set("PYARROW_${name}"
+      "AUTO"
+      CACHE STRING ${description})
+
+  if("${PYARROW_${name}}" STREQUAL "AUTO")
+    # by default, first check if env variable exists, otherwise use Arrow C++ config
+    set(env_variable "PYARROW_WITH_${name}")
+    if(DEFINED ENV{${env_variable}})
+      message(STATUS "Env variable is defined: ${env_variable}=$ENV{${env_variable}}")
+      if($ENV{${env_variable}})
+        set("PYARROW_BUILD_${name}" ON)
+        message(STATUS "Setting ${name} to ON through env variable")
+      else()
+        set("PYARROW_BUILD_${name}" OFF)
+        message(STATUS "Setting ${name} to OFF through env variable")

Review Comment:
   I made this quite verbose, mostly for testing / debugging purposes. But can trim most of the messages after everything is working if that is preferred.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1587440885


##########
python/CMakeLists.txt:
##########
@@ -265,11 +246,44 @@ message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
 
 include(UseCython)
 
-# PyArrow C++
+# Arrow C++ and set default PyArrow build options
 include(GNUInstallDirs)
-
 find_package(Arrow REQUIRED)
 
+# Top level cmake dir
+if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
+  option(PYARROW_BUILD_ACERO "Build the PyArrow Acero integration" ${ARROW_ACERO})
+  option(PYARROW_BUILD_CUDA "Build the PyArrow CUDA support" ${ARROW_CUDA})
+  option(PYARROW_BUILD_DATASET "Build the PyArrow Dataset integration" ${ARROW_DATASET})
+  option(PYARROW_BUILD_FLIGHT "Build the PyArrow Flight integration" ${ARROW_FLIGHT})
+  option(PYARROW_BUILD_GANDIVA "Build the PyArrow Gandiva integration" ${ARROW_GANDIVA})
+  option(PYARROW_BUILD_ORC "Build the PyArrow ORC integration" ${ARROW_ORC})
+  option(PYARROW_BUILD_PARQUET "Build the PyArrow Parquet integration" ${ARROW_PARQUET})
+  option(PYARROW_BUILD_PARQUET_ENCRYPTION
+         "Build the PyArrow Parquet encryption integration" ${PARQUET_REQUIRE_ENCRYPTION})
+  option(PYARROW_BUILD_SUBSTRAIT "Build the PyArrow Substrait integration"
+         ${ARROW_SUBSTRAIT})
+  option(PYARROW_BUILD_AZURE "Build the PyArrow Azure integration" ${ARROW_AZURE})
+  option(PYARROW_BUILD_GCS "Build the PyArrow GCS integration" ${ARROW_GCS})
+  option(PYARROW_BUILD_S3 "Build the PyArrow S3 integration" ${ARROW_S3})
+  option(PYARROW_BUILD_HDFS "Build the PyArrow HDFS integration" ${ARROW_HDFS})
+  option(PYARROW_BUNDLE_ARROW_CPP "Bundle the Arrow C++ libraries" OFF)
+  option(PYARROW_BUNDLE_CYTHON_CPP "Bundle the C++ files generated by Cython" OFF)
+  option(PYARROW_GENERATE_COVERAGE "Build with Cython code coverage enabled" OFF)
+  set(PYARROW_CXXFLAGS
+      ""
+      CACHE STRING "Compiler flags to append when compiling Arrow")
+endif()
+
+# enforce module dependencies
+if(PYARROW_BUILD_SUBSTRAIT)
+  set(PYARROW_BUILD_DATASET ON)
+endif()
+if(PYARROW_BUILD_DATASET)
+  set(PYARROW_BUILD_ACERO ON)
+endif()

Review Comment:
   This is to ensure we keep the same logic as we have now in setup.py, but we could in theory also leave this out (and ensure the scripts don't explicitly disable Acero when enabling Dataset)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41494:
URL: https://github.com/apache/arrow/pull/41494#discussion_r1588837384


##########
python/setup.py:
##########
@@ -270,23 +235,28 @@ def append_cmake_bool(value, varname):
                 cmake_options.append('-D{0}={1}'.format(
                     varname, 'on' if value else 'off'))
 
+            def maybe_append_cmake_bool(varname):
+                env_var = varname.replace("PYARROW_BUILD", "PYARROW_WITH")
+                if env_var in os.environ:
+                    value = strtobool(os.environ.get(env_var))
+                    append_cmake_bool(value, varname)

Review Comment:
   I did that initially in a first version locally, but then moved it again to setup.py to avoid the same problem with it being cached. But with you suggestion above to use an AUTO default, then I can indeed move it back to the cmake.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2097893582

   The failures for the conda builds are due to https://github.com/apache/arrow/issues/41563


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2099823872

   @github-actions crossbow submit wheel-* python-sdist *-python-*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41480: [Python] Building PyArrow: enable/disable python components by default based on availability in Arrow C++ [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #41494:
URL: https://github.com/apache/arrow/pull/41494#issuecomment-2115802600

   After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 1c546fb3c130fc6a4f3e06ad31dc49d923785104.
   
   There were 2 benchmark results with an error:
   
   - Commit Run on `ursa-i9-9960x` at [2024-05-16 15:15:25Z](https://conbench.ursa.dev/compare/runs/8fda2ebafab04bb8bf38f23cb1549aa0...e8395056b5ae411fabede97f542c334e/)
     - [`dataset-read` (Python) with async=True, dataset=nyctaxi_multi_parquet_s3, pre_buffer=false](https://conbench.ursa.dev/benchmark-results/06646012ec847ead800041053cd8c812)
     - [`dataset-read` (Python) with async=True, dataset=nyctaxi_multi_parquet_s3, pre_buffer=false](https://conbench.ursa.dev/benchmark-results/06646012edfe7cdc80006aa2901f4722)
   
   There was 1 benchmark result indicating a performance regression:
   
   - Commit Run on `ursa-i9-9960x` at [2024-05-16 15:15:25Z](https://conbench.ursa.dev/compare/runs/8fda2ebafab04bb8bf38f23cb1549aa0...e8395056b5ae411fabede97f542c334e/)
     - [`dataset-read` (Python) with async=True, dataset=nyctaxi_multi_parquet_s3, pre_buffer=false](https://conbench.ursa.dev/compare/benchmarks/06645543ed3a7c318000b88feae1ad82...0664609f566278468000db88fd1cd68d)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/25065117121) has more details. It also includes information about 13 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org