[GitHub] [arrow] raulcd commented on a diff in pull request #33714: GH-33697: [CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool

[GitBox <gi...@apache.org>]

raulcd commented on code in PR #33714:
URL: https://github.com/apache/arrow/pull/33714#discussion_r1072247000


##########
ci/docker/conda-python-spark.dockerfile:
##########
@@ -22,10 +22,15 @@ FROM ${repo}:${arch}-conda-python-${python}
 
 ARG jdk=8
 ARG maven=3.5
+# https://github.com/apache/arrow/issues/33697
+# numpy version pin should be removed with new apache spark release
+# that includes https://github.com/apache/spark/pull/37817
+ARG numpy=1.23

Review Comment:
   we already test spark master nightly, this is the current testing combination:
   ```
   {% for python_version, spark_version, test_pyarrow_only in [("3.7", "v3.1.2", "false"),
                                                               ("3.8", "v3.2.0", "false"),
                                                               ("3.9", "master", "false")] %}
   ```
   And the build for spark master is currently passing: https://github.com/ursacomputing/crossbow/actions/runs/3934958561/jobs/6730195747#step:5:10
   Maybe we can add the numpy version to the task definition only for 3.2.0 and if it is different than latest install the pinned version. I am thinking on something like:
   ```
   {% for python_version, spark_version, test_pyarrow_only, numpy_version in [("3.7", "v3.1.2", "false", "latest"),
                                                               ("3.8", "v3.2.0", "false", "1.23"),
                                                               ("3.9", "master", "false", "latest")] %}
   ```
   And the corresponding if to validate if we have to install numpy or not?
   @kiszk you are  spark committer. I suppose this fix won't get backported to spark 3.2.0 and we have to pin numpy always for it?  Should we update the tasks for our nightlies to test with spark 3.3.0 maybe remove 3.2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org