You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/06 09:25:49 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request, #38934: [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import WriteOperation only when pandas is available
dongjoon-hyun opened a new pull request, #38934:
URL: https://github.com/apache/spark/pull/38934
### What changes were proposed in this pull request?
This is the last piece to recover `pyspark-connect` tests on a system where pandas is unavailable.
### Why are the changes needed?
**BEFORE**
```
$ python/run-tests.py --modules pyspark-connect
...
ModuleNotFoundError: No module named 'pandas'
```
**AFTER**
```
$ python/run-tests.py --modules pyspark-connect
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3.9']
Will test the following Python modules: ['pyspark-connect']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.14
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_column_expressions (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/fc8f073e-edbe-470f-a3cc-f35b3f4b5262/python3.9__pyspark.sql.tests.connect.test_connect_column_expressions___lzz5fky.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_basic (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/c14dfac4-14df-45b4-b54e-6dbc3c58f128/python3.9__pyspark.sql.tests.connect.test_connect_basic__x8rz2dz1.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_column (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/5810e639-75c1-4e6c-91b3-2a1894dab319/python3.9__pyspark.sql.tests.connect.test_connect_column__3e_qmue7.log)
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_function (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/8bebb3be-c146-4030-96b3-82ff7a873d49/python3.9__pyspark.sql.tests.connect.test_connect_function__hwe9aca8.log)
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_column_expressions (1s) ... 9 tests were skipped
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/97cbccca-1e9a-44d9-958f-e2368266b437/python3.9__pyspark.sql.tests.connect.test_connect_plan_only__dffs5ux1.log)
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_column (1s) ... 2 tests were skipped
Starting test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/297a054f-c577-4aee-b1ff-48d48beb423c/python3.9__pyspark.sql.tests.connect.test_connect_select_ops__4e__w6gh.log)
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_function (1s) ... 5 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_basic (1s) ... 48 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_select_ops (0s) ... 2 tests were skipped
Finished test(python3.9): pyspark.sql.tests.connect.test_connect_plan_only (1s) ... 28 tests were skipped
Tests passed in 2 seconds
Skipped tests in pyspark.sql.tests.connect.test_connect_basic with python3.9:
test_channel_properties (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.002s)
test_invalid_connection_strings (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s)
test_metadata (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s)
test_sensible_defaults (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s)
test_valid_channel_creation (pyspark.sql.tests.connect.test_connect_basic.ChannelBuilderTests) ... skip (0.000s)
test_agg_with_avg (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_agg_with_two_agg_exprs (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_collect (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_count (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_create_global_temp_view (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_create_session_local_temp_view (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_crossjoin (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_deduplicate (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_drop (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_drop_na (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_empty_dataset (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_explain_string (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_fill_na (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_first (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_head (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_input_files (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_is_empty (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_is_local (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_is_streaming (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_join_condition_column_list_columns (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_limit_offset (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_print_schema (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_range (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_replace (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_repr (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_schema (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_select_expr (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_session (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_show (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_simple_datasource_read (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_simple_explain_string (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_simple_read (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_simple_udf (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_sort (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
test_sql (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_subquery_alias (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_tail (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_take (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_toDF (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_to_pandas (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_with_columns (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_with_columns_renamed (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.000s)
test_write_operations (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests) ... skip (0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_column with python3.9:
test_column_operator (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip (0.002s)
test_columns (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests) ... skip (0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_column_expressions with python3.9:
test_binary_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.002s)
test_column_alias (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_column_literals (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_float_nan_inf (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_map_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.001s)
test_null_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_numeric_literal_types (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_simple_column_expressions (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
test_uuid_literal (pyspark.sql.tests.connect.test_connect_column_expressions.SparkConnectColumnExpressionSuite) ... skip (0.000s)
Skipped tests in pyspark.sql.tests.connect.test_connect_function with python3.9:
test_aggregation_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.006s)
test_math_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.025s)
test_normal_functions (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.002s)
test_sort_with_nulls_order (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.001s)
test_sorting_functions_with_column (pyspark.sql.tests.connect.test_connect_function.SparkConnectFunctionTests) ... skip (0.000s)
Skipped tests in pyspark.sql.tests.connect.test_connect_plan_only with python3.9:
test_all_the_plans (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.013s)
test_coalesce_and_repartition (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s)
test_crossjoin (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_crosstab (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_datasource_read (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_deduplicate (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_drop (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_drop_na (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_except (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_fill_na (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_filter (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_intersect (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_join_condition (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_join_using_columns (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_limit (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_offset (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_range (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_relation_alias (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_replace (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_sample (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_simple_project (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_simple_udf (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.000s)
test_sort (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_sql_project (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s)
test_summary (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_union (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.002s)
test_unsupported_functions (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
test_write_operation (pyspark.sql.tests.connect.test_connect_plan_only.SparkConnectTestsPlanOnly) ... skip (0.001s)
Skipped tests in pyspark.sql.tests.connect.test_connect_select_ops with python3.9:
test_join_with_join_type (pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite) ... skip (0.085s)
test_select_with_columns_and_strings (pyspark.sql.tests.connect.test_connect_select_ops.SparkConnectToProtoSuite) ... skip (0.000s)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the CIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38934: [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import WriteOperation only when pandas is available
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38934:
URL: https://github.com/apache/spark/pull/38934#issuecomment-1339027484
cc @grundprinzip , @hvanhovell , @bjornjorgensen , @zhengruifeng , @HyukjinKwon , @amaliujia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #38934: [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when `pandas` is available
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #38934: [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when `pandas` is available
URL: https://github.com/apache/spark/pull/38934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38934: [SPARK-41317][CONNECT][TESTS][FOLLOWUP] Import `WriteOperation` only when `pandas` is available
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38934:
URL: https://github.com/apache/spark/pull/38934#issuecomment-1339150057
Thank you, @HyukjinKwon . Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org