[GitHub] [arrow] raulcd commented on a diff in pull request #14715: ARROW-18390: [CI][Python] Update spark test modules to match spark master

[GitBox <gi...@apache.org>]

raulcd commented on code in PR #14715:
URL: https://github.com/apache/arrow/pull/14715#discussion_r1032402225


##########
ci/scripts/integration_spark.sh:
##########
@@ -73,14 +73,38 @@ pushd ${spark_dir}
 
   # Run pyarrow related Python tests only
   spark_python_tests=(
-    "pyspark.sql.tests.test_arrow"
-    "pyspark.sql.tests.test_pandas_map"
-    "pyspark.sql.tests.test_pandas_cogrouped_map"
-    "pyspark.sql.tests.test_pandas_grouped_map"
-    "pyspark.sql.tests.test_pandas_udf"
-    "pyspark.sql.tests.test_pandas_udf_scalar"
-    "pyspark.sql.tests.test_pandas_udf_grouped_agg"
-    "pyspark.sql.tests.test_pandas_udf_window")
+    "pyspark.sql.tests.test_arrow")
+
+  old_test_modules=false
+  # Spark <= 2.x or <= 3.3.x
+  if [ "${SPARK_VERSION:0:2}" == "2." ]; then
+    old_test_modules=true
+  elif [ "${SPARK_VERSION:0:2}" == "3." ]; then
+    if [ "${SPARK_VERSION:2:2}" == "1." ] || \
+       [ "${SPARK_VERSION:2:2}" == "2." ] || \
+       [ "${SPARK_VERSION:2:2}" == "3." ]; then
+        old_test_modules=true
+    fi
+  fi

Review Comment:
   Thanks @kou ! Yes, this is much better than what I was doing!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org