You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/06 08:41:38 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

HyukjinKwon opened a new pull request #30277:
URL: https://github.com/apache/spark/pull/30277


   ### What changes were proposed in this pull request?
   
   This PR proposes to fix PySpark to officially support Python 3.9. The main codes already work. We should just note that we support Python 3.9.
   
   Also, this PR fixes some minor fixes into the test codes.
   - `Thread.isAlive` is removed in Python 3.9, and `Thread.is_alive` exists in Python 3.6+, see https://docs.python.org/3/whatsnew/3.9.html#removed
   - Fixed `TaskContextTestsWithWorkerReuse.test_barrier_with_python_worker_reuse` and `TaskContextTests.test_barrier` to be less flaky. This becomes more flaky in Python 3.9 for some reasons.
   
   NOTE that PyArrow does not support Python 3.9 yet.
   
   ### Why are the changes needed?
   
   To officially support Python 3.9.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, it officially supports Python 3.9.
   
   ### How was this patch tested?
   
   Manually ran the tests:
   
   ```
   $  ./run-tests --python-executable=python
   Running PySpark tests. Output is in /.../spark/python/unit-tests.log
   Will test against the following Python executables: ['python']
   Will test the following Python modules: ['pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming']
   python python_implementation is CPython
   python version is: Python 3.9.0
   Starting test(python): pyspark.ml.tests.test_base
   Starting test(python): pyspark.ml.tests.test_evaluation
   Starting test(python): pyspark.ml.tests.test_algorithms
   Starting test(python): pyspark.ml.tests.test_feature
   Finished test(python): pyspark.ml.tests.test_base (12s)
   Starting test(python): pyspark.ml.tests.test_image
   Finished test(python): pyspark.ml.tests.test_evaluation (15s)
   Starting test(python): pyspark.ml.tests.test_linalg
   Finished test(python): pyspark.ml.tests.test_feature (25s)
   Starting test(python): pyspark.ml.tests.test_param
   Finished test(python): pyspark.ml.tests.test_image (17s)
   Starting test(python): pyspark.ml.tests.test_persistence
   Finished test(python): pyspark.ml.tests.test_param (17s)
   Starting test(python): pyspark.ml.tests.test_pipeline
   Finished test(python): pyspark.ml.tests.test_linalg (30s)
   Starting test(python): pyspark.ml.tests.test_stat
   Finished test(python): pyspark.ml.tests.test_pipeline (6s)
   Starting test(python): pyspark.ml.tests.test_training_summary
   Finished test(python): pyspark.ml.tests.test_stat (12s)
   Starting test(python): pyspark.ml.tests.test_tuning
   Finished test(python): pyspark.ml.tests.test_algorithms (68s)
   Starting test(python): pyspark.ml.tests.test_wrapper
   Finished test(python): pyspark.ml.tests.test_persistence (51s)
   Starting test(python): pyspark.mllib.tests.test_algorithms
   Finished test(python): pyspark.ml.tests.test_training_summary (33s)
   Starting test(python): pyspark.mllib.tests.test_feature
   Finished test(python): pyspark.ml.tests.test_wrapper (19s)
   Starting test(python): pyspark.mllib.tests.test_linalg
   Finished test(python): pyspark.mllib.tests.test_feature (26s)
   Starting test(python): pyspark.mllib.tests.test_stat
   Finished test(python): pyspark.mllib.tests.test_stat (22s)
   Starting test(python): pyspark.mllib.tests.test_streaming_algorithms
   Finished test(python): pyspark.mllib.tests.test_algorithms (53s)
   Starting test(python): pyspark.mllib.tests.test_util
   Finished test(python): pyspark.mllib.tests.test_linalg (54s)
   Starting test(python): pyspark.sql.tests.test_arrow
   Finished test(python): pyspark.sql.tests.test_arrow (0s) ... 61 tests were skipped
   Starting test(python): pyspark.sql.tests.test_catalog
   Finished test(python): pyspark.mllib.tests.test_util (11s)
   Starting test(python): pyspark.sql.tests.test_column
   Finished test(python): pyspark.sql.tests.test_catalog (16s)
   Starting test(python): pyspark.sql.tests.test_conf
   Finished test(python): pyspark.sql.tests.test_column (17s)
   Starting test(python): pyspark.sql.tests.test_context
   Finished test(python): pyspark.sql.tests.test_context (6s) ... 3 tests were skipped
   Starting test(python): pyspark.sql.tests.test_dataframe
   Finished test(python): pyspark.sql.tests.test_conf (11s)
   Starting test(python): pyspark.sql.tests.test_datasources
   Finished test(python): pyspark.sql.tests.test_datasources (19s)
   Starting test(python): pyspark.sql.tests.test_functions
   Finished test(python): pyspark.sql.tests.test_dataframe (35s) ... 3 tests were skipped
   Starting test(python): pyspark.sql.tests.test_group
   Finished test(python): pyspark.sql.tests.test_functions (32s)
   Starting test(python): pyspark.sql.tests.test_pandas_cogrouped_map
   Finished test(python): pyspark.sql.tests.test_pandas_cogrouped_map (1s) ... 15 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_grouped_map
   Finished test(python): pyspark.sql.tests.test_group (19s)
   Starting test(python): pyspark.sql.tests.test_pandas_map
   Finished test(python): pyspark.sql.tests.test_pandas_grouped_map (0s) ... 21 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_udf
   Finished test(python): pyspark.sql.tests.test_pandas_map (0s) ... 6 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_udf_grouped_agg
   Finished test(python): pyspark.sql.tests.test_pandas_udf (0s) ... 6 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_udf_scalar
   Finished test(python): pyspark.sql.tests.test_pandas_udf_grouped_agg (0s) ... 13 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_udf_typehints
   Finished test(python): pyspark.sql.tests.test_pandas_udf_scalar (0s) ... 50 tests were skipped
   Starting test(python): pyspark.sql.tests.test_pandas_udf_window
   Finished test(python): pyspark.sql.tests.test_pandas_udf_typehints (0s) ... 10 tests were skipped
   Starting test(python): pyspark.sql.tests.test_readwriter
   Finished test(python): pyspark.sql.tests.test_pandas_udf_window (0s) ... 14 tests were skipped
   Starting test(python): pyspark.sql.tests.test_serde
   Finished test(python): pyspark.sql.tests.test_serde (19s)
   Starting test(python): pyspark.sql.tests.test_session
   Finished test(python): pyspark.mllib.tests.test_streaming_algorithms (120s)
   Starting test(python): pyspark.sql.tests.test_streaming
   Finished test(python): pyspark.sql.tests.test_readwriter (25s)
   Starting test(python): pyspark.sql.tests.test_types
   Finished test(python): pyspark.ml.tests.test_tuning (208s)
   Starting test(python): pyspark.sql.tests.test_udf
   Finished test(python): pyspark.sql.tests.test_session (31s)
   Starting test(python): pyspark.sql.tests.test_utils
   Finished test(python): pyspark.sql.tests.test_streaming (35s)
   Starting test(python): pyspark.streaming.tests.test_context
   Finished test(python): pyspark.sql.tests.test_types (34s)
   Starting test(python): pyspark.streaming.tests.test_dstream
   Finished test(python): pyspark.sql.tests.test_utils (14s)
   Starting test(python): pyspark.streaming.tests.test_kinesis
   Finished test(python): pyspark.streaming.tests.test_kinesis (0s) ... 2 tests were skipped
   Starting test(python): pyspark.streaming.tests.test_listener
   Finished test(python): pyspark.streaming.tests.test_listener (11s)
   Starting test(python): pyspark.tests.test_appsubmit
   Finished test(python): pyspark.sql.tests.test_udf (39s)
   Starting test(python): pyspark.tests.test_broadcast
   Finished test(python): pyspark.streaming.tests.test_context (23s)
   Starting test(python): pyspark.tests.test_conf
   Finished test(python): pyspark.tests.test_conf (15s)
   Starting test(python): pyspark.tests.test_context
   Finished test(python): pyspark.tests.test_broadcast (33s)
   Starting test(python): pyspark.tests.test_daemon
   Finished test(python): pyspark.tests.test_daemon (5s)
   Starting test(python): pyspark.tests.test_install_spark
   Finished test(python): pyspark.tests.test_context (44s)
   Starting test(python): pyspark.tests.test_join
   Finished test(python): pyspark.tests.test_appsubmit (68s)
   Starting test(python): pyspark.tests.test_profiler
   Finished test(python): pyspark.tests.test_join (7s)
   Starting test(python): pyspark.tests.test_rdd
   Finished test(python): pyspark.tests.test_profiler (9s)
   Starting test(python): pyspark.tests.test_rddbarrier
   Finished test(python): pyspark.tests.test_rddbarrier (7s)
   Starting test(python): pyspark.tests.test_readwrite
   Finished test(python): pyspark.streaming.tests.test_dstream (107s)
   Starting test(python): pyspark.tests.test_serializers
   Finished test(python): pyspark.tests.test_serializers (8s)
   Starting test(python): pyspark.tests.test_shuffle
   Finished test(python): pyspark.tests.test_readwrite (14s)
   Starting test(python): pyspark.tests.test_taskcontext
   Finished test(python): pyspark.tests.test_install_spark (65s)
   Starting test(python): pyspark.tests.test_util
   Finished test(python): pyspark.tests.test_shuffle (8s)
   Starting test(python): pyspark.tests.test_worker
   Finished test(python): pyspark.tests.test_util (5s)
   Starting test(python): pyspark.accumulators
   Finished test(python): pyspark.accumulators (5s)
   Starting test(python): pyspark.broadcast
   Finished test(python): pyspark.broadcast (6s)
   Starting test(python): pyspark.conf
   Finished test(python): pyspark.tests.test_worker (14s)
   Starting test(python): pyspark.context
   Finished test(python): pyspark.conf (4s)
   Starting test(python): pyspark.ml.classification
   Finished test(python): pyspark.tests.test_rdd (60s)
   Starting test(python): pyspark.ml.clustering
   Finished test(python): pyspark.context (21s)
   Starting test(python): pyspark.ml.evaluation
   Finished test(python): pyspark.tests.test_taskcontext (69s)
   Starting test(python): pyspark.ml.feature
   Finished test(python): pyspark.ml.evaluation (26s)
   Starting test(python): pyspark.ml.fpm
   Finished test(python): pyspark.ml.clustering (45s)
   Starting test(python): pyspark.ml.functions
   Finished test(python): pyspark.ml.fpm (24s)
   Starting test(python): pyspark.ml.image
   Finished test(python): pyspark.ml.functions (17s)
   Starting test(python): pyspark.ml.linalg.__init__
   Finished test(python): pyspark.ml.linalg.__init__ (0s)
   Starting test(python): pyspark.ml.recommendation
   Finished test(python): pyspark.ml.classification (74s)
   Starting test(python): pyspark.ml.regression
   Finished test(python): pyspark.ml.image (8s)
   Starting test(python): pyspark.ml.stat
   Finished test(python): pyspark.ml.stat (29s)
   Starting test(python): pyspark.ml.tuning
   Finished test(python): pyspark.ml.regression (53s)
   Starting test(python): pyspark.mllib.classification
   Finished test(python): pyspark.ml.tuning (35s)
   Starting test(python): pyspark.mllib.clustering
   Finished test(python): pyspark.ml.feature (103s)
   Starting test(python): pyspark.mllib.evaluation
   Finished test(python): pyspark.mllib.classification (33s)
   Starting test(python): pyspark.mllib.feature
   Finished test(python): pyspark.mllib.evaluation (21s)
   Starting test(python): pyspark.mllib.fpm
   Finished test(python): pyspark.ml.recommendation (103s)
   Starting test(python): pyspark.mllib.linalg.__init__
   Finished test(python): pyspark.mllib.linalg.__init__ (1s)
   Starting test(python): pyspark.mllib.linalg.distributed
   Finished test(python): pyspark.mllib.feature (26s)
   Starting test(python): pyspark.mllib.random
   Finished test(python): pyspark.mllib.fpm (23s)
   Starting test(python): pyspark.mllib.recommendation
   Finished test(python): pyspark.mllib.clustering (50s)
   Starting test(python): pyspark.mllib.regression
   Finished test(python): pyspark.mllib.random (13s)
   Starting test(python): pyspark.mllib.stat.KernelDensity
   Finished test(python): pyspark.mllib.stat.KernelDensity (1s)
   Starting test(python): pyspark.mllib.stat._statistics
   Finished test(python): pyspark.mllib.linalg.distributed (42s)
   Starting test(python): pyspark.mllib.tree
   Finished test(python): pyspark.mllib.stat._statistics (19s)
   Starting test(python): pyspark.mllib.util
   Finished test(python): pyspark.mllib.regression (33s)
   Starting test(python): pyspark.profiler
   Finished test(python): pyspark.mllib.recommendation (36s)
   Starting test(python): pyspark.rdd
   Finished test(python): pyspark.profiler (9s)
   Starting test(python): pyspark.resource.tests.test_resources
   Finished test(python): pyspark.mllib.tree (19s)
   Starting test(python): pyspark.serializers
   Finished test(python): pyspark.mllib.util (21s)
   Starting test(python): pyspark.shuffle
   Finished test(python): pyspark.resource.tests.test_resources (9s)
   Starting test(python): pyspark.sql.avro.functions
   Finished test(python): pyspark.shuffle (1s)
   Starting test(python): pyspark.sql.catalog
   Finished test(python): pyspark.rdd (22s)
   Starting test(python): pyspark.sql.column
   Finished test(python): pyspark.serializers (12s)
   Starting test(python): pyspark.sql.conf
   Finished test(python): pyspark.sql.conf (6s)
   Starting test(python): pyspark.sql.context
   Finished test(python): pyspark.sql.catalog (14s)
   Starting test(python): pyspark.sql.dataframe
   Finished test(python): pyspark.sql.avro.functions (15s)
   Starting test(python): pyspark.sql.functions
   Finished test(python): pyspark.sql.column (24s)
   Starting test(python): pyspark.sql.group
   Finished test(python): pyspark.sql.context (20s)
   Starting test(python): pyspark.sql.pandas.conversion
   Finished test(python): pyspark.sql.pandas.conversion (13s)
   Starting test(python): pyspark.sql.pandas.group_ops
   Finished test(python): pyspark.sql.group (36s)
   Starting test(python): pyspark.sql.pandas.map_ops
   Finished test(python): pyspark.sql.pandas.group_ops (21s)
   Starting test(python): pyspark.sql.pandas.serializers
   Finished test(python): pyspark.sql.pandas.serializers (0s)
   Starting test(python): pyspark.sql.pandas.typehints
   Finished test(python): pyspark.sql.pandas.typehints (0s)
   Starting test(python): pyspark.sql.pandas.types
   Finished test(python): pyspark.sql.pandas.types (0s)
   Starting test(python): pyspark.sql.pandas.utils
   Finished test(python): pyspark.sql.pandas.utils (0s)
   Starting test(python): pyspark.sql.readwriter
   Finished test(python): pyspark.sql.dataframe (56s)
   Starting test(python): pyspark.sql.session
   Finished test(python): pyspark.sql.functions (57s)
   Starting test(python): pyspark.sql.streaming
   Finished test(python): pyspark.sql.pandas.map_ops (12s)
   Starting test(python): pyspark.sql.types
   Finished test(python): pyspark.sql.types (10s)
   Starting test(python): pyspark.sql.udf
   Finished test(python): pyspark.sql.streaming (16s)
   Starting test(python): pyspark.sql.window
   Finished test(python): pyspark.sql.session (19s)
   Starting test(python): pyspark.streaming.util
   Finished test(python): pyspark.streaming.util (0s)
   Starting test(python): pyspark.util
   Finished test(python): pyspark.util (0s)
   Finished test(python): pyspark.sql.readwriter (24s)
   Finished test(python): pyspark.sql.udf (13s)
   Finished test(python): pyspark.sql.window (14s)
   Tests passed in 780 seconds
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] BryanCutler commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723342005


   >Hm, maybe pyarrow dev branch supports but seems like the release doesn't support (?), https://arrow.apache.org/docs/python/install.html#python-compatibility.
   
   >At least I can't install out of the box in both pip and conda.
   
   Ahh, it's available for conda in 3.9, but not pip..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30277:
URL: https://github.com/apache/spark/pull/30277#discussion_r519029346



##########
File path: python/pyspark/tests/test_taskcontext.py
##########
@@ -124,12 +124,12 @@ def f(iterator):
 
         def context_barrier(x):
             tc = BarrierTaskContext.get()
-            time.sleep(random.randint(1, 10))
+            time.sleep(random.randint(1, 5) * 2)

Review comment:
       The same question. And, is it different from `random.randint(2, 10)`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] BryanCutler commented on a change in pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #30277:
URL: https://github.com/apache/spark/pull/30277#discussion_r518879946



##########
File path: python/pyspark/tests/test_taskcontext.py
##########
@@ -124,12 +124,12 @@ def f(iterator):
 
         def context_barrier(x):
             tc = BarrierTaskContext.get()
-            time.sleep(random.randint(1, 10))
+            time.sleep(random.randint(1, 5) * 2)

Review comment:
       Isn't this basically increasing the minimum sleep time? I would think that tend to make it more flaky..




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722969456


   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130707/
   Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723339112


   Could you make a backporting PR once more in order to see the test result on `branch-3.0` please?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30277:
URL: https://github.com/apache/spark/pull/30277#discussion_r519038459



##########
File path: python/pyspark/tests/test_taskcontext.py
##########
@@ -124,12 +124,12 @@ def f(iterator):
 
         def context_barrier(x):
             tc = BarrierTaskContext.get()
-            time.sleep(random.randint(1, 10))
+            time.sleep(random.randint(1, 5) * 2)

Review comment:
       Previously it failed even when the barrior worked some times. All the tasks properly waited for each other and finished properly but it looks like it could still take around a sec when the tests run in parallel.
   
   So I increased the minimum sleep secs and the gap between each task. If barriers don't work, it will take more than 2 secs because the minimum and the gap is 2 seconds. If it works, the gap between each task will take less than 2 seconds.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30277:
URL: https://github.com/apache/spark/pull/30277


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722968958


   **[Test build #130707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130707/testReport)** for PR 30277 at commit [`40b1e40`](https://github.com/apache/spark/commit/40b1e40c06ec14df04afdfb9c8c59afe35d0e633).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722969446


   Merged build finished. Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] BryanCutler commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723642904


   I use the conda-forge channel, I think main takes a while to catch up. Can
   you try `conda install -c conda-forge pyarrow`?
   
   On Sun, Nov 8, 2020, 3:09 AM Hyukjin Kwon <no...@github.com> wrote:
   
   > Hm, seems like I can't still install in conda:
   >
   > conda install pyarrow
   > Collecting package metadata (current_repodata.json): done
   > Solving environment: failed with initial frozen solve. Retrying with flexible solve.
   > Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
   > Collecting package metadata (repodata.json): done
   > Solving environment: failed with initial frozen solve. Retrying with flexible solve.
   > Solving environment: \
   > Found conflicts! Looking for incompatible packages.
   > This can take several minutes.  Press CTRL-C to abort.
   > failed
   >
   > UnsatisfiableError: The following specifications were found
   > to be incompatible with the existing python installation in your environment:
   >
   > Specifications:
   >
   >   - pyarrow -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']
   >
   > Your python: python=3.9
   >
   > If python is on the left-most side of the chain, that's the version you've asked for.
   > When python appears to the right, that indicates that the thing on the left is somehow
   > not available for the python version you are constrained to. Note that conda will not
   > change your python version to a different minor version unless you explicitly specify
   > that.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/30277#issuecomment-723561870>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABCTA5N33O43LGCWQEJTSCLSOZ353ANCNFSM4TMMU3DA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723371742


   Ohh maybe I had to update my conda. I'll check it tomorrow and see if tests pass :).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723339613


   Sure! Thanks @dongjoon-hyun and @BryanCutler 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722954003


   I believe this can be ported back to branch-3.0.
   
   cc @BryanCutler @ueshin @jiangxb1987 can you take a look when you find some time?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722992547


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35317/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723837905


   I can confirm that the tests pass with PyArrow latest version and Python 3.9.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723764926


   @BryanCutler oh sure. it works! thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722992569


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723225505


   +1 for backporting to `branch-3.0`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30277:
URL: https://github.com/apache/spark/pull/30277#discussion_r519038459



##########
File path: python/pyspark/tests/test_taskcontext.py
##########
@@ -124,12 +124,12 @@ def f(iterator):
 
         def context_barrier(x):
             tc = BarrierTaskContext.get()
-            time.sleep(random.randint(1, 10))
+            time.sleep(random.randint(1, 5) * 2)

Review comment:
       Previously it failed even when the barrior worked some times. All the tasks properly waited for each other and finished properly but it looks like it could still take around a sec when the tests run in parallel.
   
   So I increased the minimum sleep secs and the gap between each task. If barriers don't work, it will take more than 2 secs because the minimum and the gap is 2 seconds. If it works, it will take less than 2 seconds.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722981023


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35317/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723561870


   Hm, seems like I can't still install in conda:
   
   ```
   conda install pyarrow
   Collecting package metadata (current_repodata.json): done
   Solving environment: failed with initial frozen solve. Retrying with flexible solve.
   Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
   Collecting package metadata (repodata.json): done
   Solving environment: failed with initial frozen solve. Retrying with flexible solve.
   Solving environment: \
   Found conflicts! Looking for incompatible packages.
   This can take several minutes.  Press CTRL-C to abort.
   failed
   
   UnsatisfiableError: The following specifications were found
   to be incompatible with the existing python installation in your environment:
   
   Specifications:
   
     - pyarrow -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']
   
   Your python: python=3.9
   
   If python is on the left-most side of the chain, that's the version you've asked for.
   When python appears to the right, that indicates that the thing on the left is somehow
   not available for the python version you are constrained to. Note that conda will not
   change your python version to a different minor version unless you explicitly specify
   that.
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722954696


   **[Test build #130707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130707/testReport)** for PR 30277 at commit [`40b1e40`](https://github.com/apache/spark/commit/40b1e40c06ec14df04afdfb9c8c59afe35d0e633).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723327343


   Hm, maybe pyarrow dev branch supports but seems like the release doesn't support (?), https://arrow.apache.org/docs/python/install.html#python-compatibility.
   
   At least I can't install out of the box in both pip and conda.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722992577


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35317/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723339112


   Merged to master.
   
   Could you make a backporting PR once more in order to see the test result on `branch-3.0` please?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722992569






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] BryanCutler commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-723189246


   PyArrow supports 3.9, or was it causing some test error?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722954696


   **[Test build #130707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130707/testReport)** for PR 30277 at commit [`40b1e40`](https://github.com/apache/spark/commit/40b1e40c06ec14df04afdfb9c8c59afe35d0e633).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30277: [SPARK-33371][PYTHON] Update setup.py and tests for Python 3.9

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30277:
URL: https://github.com/apache/spark/pull/30277#issuecomment-722969446






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org