You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/22 06:24:20 UTC
[GitHub] [spark] LuciferYang commented on a diff in pull request #38631: [SPARK-40809] [CONNECT] [FOLLOW] Support `alias()` in Python client
LuciferYang commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1028908773
##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -301,6 +301,20 @@ def test_simple_datasource_read(self) -> None:
actualResult = pandasResult.values.tolist()
self.assertEqual(len(expectResult), len(actualResult))
+ def test_alias(self) -> None:
+ """Testing supported and unsupported alias"""
+ col0 = (
+ self.connect.range(1, 10)
+ .select(col("id").alias("name", metadata={"max": 99}))
+ .schema()
+ .names[0]
+ )
+ self.assertEqual("name", col0)
+
+ with self.assertRaises(grpc.RpcError) as exc:
+ self.connect.range(1, 10).select(col("id").alias("this", "is", "not")).collect()
+ self.assertIn("Buffer(this, is, not)", str(exc.exception))
Review Comment:
@grundprinzip The Scala 2.13 daily test has failed in recent times, which seems to be related to this assert:
```
======================================================================
FAIL [0.742s]: test_alias (pyspark.sql.tests.connect.test_connect_basic.SparkConnectTests)
Testing supported and unsupported alias
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_basic.py", line 337, in test_alias
self.assertIn("Buffer(this, is, not)", str(exc.exception))
AssertionError: 'Buffer(this, is, not)' not found in '<_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = "[INTERNAL_ERROR] Found the unresolved operator: \'Project [id#24L AS List(this, is, not)]"\n\tdebug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:15002 {created_time:"2022-11-21T19:25:59.902777537+00:00", grpc_status:2, grpc_message:"[INTERNAL_ERROR] Found the unresolved operator: \\\'Project [id#24L AS List(this, is, not)]"}"\n>'
```
https://github.com/apache/spark/actions/runs/3517345801/jobs/5895043640
This seems to be related to the difference of Scala version. Can we simplify the assertion? such as `self.assertIn("(this, is, not)", str(exc.exception))`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org