You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruifeng Zheng (Jira)" <ji...@apache.org> on 2023/05/15 03:57:00 UTC
[jira] [Comment Edited] (SPARK-43501) `test_ops_on_diff_frames_*` are much slower on Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-43501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722633#comment-17722633 ]
Ruifeng Zheng edited comment on SPARK-43501 at 5/15/23 3:56 AM:
----------------------------------------------------------------
In vanilla PySpark:
{code}
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (temp output: /__w/spark/spark/python/target/bcf91e64-da92-456c-a65f-2acbb5a57c5e/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames__vhzam369.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (183s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (temp output: /__w/spark/spark/python/target/21464771-1bb4-4e2e-8a76-cc5424bc3eee/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames_groupby__3jskgr8q.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (121s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_slow (temp output: /__w/spark/spark/python/target/4ec97d5c-ff3a-474f-aaa4-a3e08b503667/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames_slow__bfsd6ob8.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_slow (167s)
{code}
In Spark Connect:
{code}
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames (temp output: /__w/spark/spark/python/target/7d09fc80-b92b-4872-ae33-62f61aa4d2d9/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames__3d49xqjs.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames (1018s)
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby (temp output: /__w/spark/spark/python/target/6542f9a4-39fb-4605-9e7b-6d9a3e7b5a63/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby__swoyoc14.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby (350s) ... 8 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow (temp output: /__w/spark/spark/python/target/7880eb8c-9299-4e1c-af32-daa48aa10715/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow__y24aawsj.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow (732s) ... 7 tests were skipped
{code}
was (Author: podongfeng):
In vanilla PySpark:
{code}
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (temp output: /__w/spark/spark/python/target/bcf91e64-da92-456c-a65f-2acbb5a57c5e/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames__vhzam369.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (183s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (temp output: /__w/spark/spark/python/target/21464771-1bb4-4e2e-8a76-cc5424bc3eee/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames_groupby__3jskgr8q.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (121s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_slow (temp output: /__w/spark/spark/python/target/4ec97d5c-ff3a-474f-aaa4-a3e08b503667/python3.9__pyspark.pandas.tests.test_ops_on_diff_frames_slow__bfsd6ob8.log)
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_slow (167s)
{code}
{code}
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames (temp output: /__w/spark/spark/python/target/7d09fc80-b92b-4872-ae33-62f61aa4d2d9/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames__3d49xqjs.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames (1018s)
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby (temp output: /__w/spark/spark/python/target/6542f9a4-39fb-4605-9e7b-6d9a3e7b5a63/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby__swoyoc14.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby (350s) ... 8 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow (temp output: /__w/spark/spark/python/target/7880eb8c-9299-4e1c-af32-daa48aa10715/python3.9__pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow__y24aawsj.log)
Finished test(python3.9): pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow (732s) ... 7 tests were skipped
{code}
> `test_ops_on_diff_frames_*` are much slower on Spark Connect
> ------------------------------------------------------------
>
> Key: SPARK-43501
> URL: https://issues.apache.org/jira/browse/SPARK-43501
> Project: Spark
> Issue Type: Test
> Components: Connect, PySpark, Tests
> Affects Versions: 3.5.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org