You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/08/24 12:02:48 UTC
[GitHub] [spark] LuciferYang commented on pull request #42521: [SPARK-44435][SS][CONNECT] Tests for foreachBatch and Listener
LuciferYang commented on PR #42521:
URL: https://github.com/apache/spark/pull/42521#issuecomment-1691547730
https://github.com/apache/spark/actions/runs/5962873768/job/16174987432
```
Running tests...
----------------------------------------------------------------------
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
/__w/spark/spark/python/pyspark/sql/connect/session.py:185: UserWarning: [CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.connect.execute.reattachable.senderMaxStreamDuration".
See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements'.
warnings.warn(str(e))
/__w/spark/spark/python/pyspark/sql/connect/session.py:185: UserWarning: [CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.connect.execute.reattachable.senderMaxStreamSize".
See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements'.
warnings.warn(str(e))
/__w/spark/spark/python/pyspark/sql/connect/session.py:185: UserWarning: [CANNOT_MODIFY_CONFIG] Cannot modify the value of the Spark config: "spark.connect.grpc.binding.port".
See also 'https://spark.apache.org/docs/latest/sql-migration-guide.html#ddl-statements'.
warnings.warn(str(e))
test_listener_events (pyspark.sql.tests.connect.streaming.test_parity_listener.StreamingListenerParityTests) ... Streaming query listener worker is starting with url sc://localhost:43833/;user_id= and sessionId a5a5becc-8da7-4d4b-9a7c-484cd957e3be.
[Stage 0:> (0 + 1) / 1]
[Stage 0:> (0 + 1) / 1][Stage 2:> (0 + 1) / 1]
[Stage 0:> (0 + 1) / 1]
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/connect/streaming/worker/listener_worker.py", line 99, in <module>
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/connect/streaming/worker/listener_worker.py", line 86, in main
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/connect/streaming/worker/listener_worker.py", line 77, in process
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/streaming/listener.py", line 251, in fromJson
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/streaming/listener.py", line 480, in fromJson
KeyError: 'batchDuration'
[Stage 17:> (0 + 1) / 1]
ERROR (46.372s)
======================================================================
ERROR [46.372s]: test_listener_events (pyspark.sql.tests.connect.streaming.test_parity_listener.StreamingListenerParityTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py", line 80, in test_listener_events
self.spark.read.table("listener_progress_events").collect()[0][0]
File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 1645, in collect
table, schema = self._session.client.to_table(query)
File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 833, in to_table
table, schema, _, _, _ = self._execute_and_fetch(req)
File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1257, in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(req):
File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1238, in _execute_and_fetch_as_iterator
self._handle_error(error)
File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1477, in _handle_error
self._handle_rpc_error(error)
File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1513, in _handle_rpc_error
raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `listener_progress_events` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.;
'UnresolvedRelation [listener_progress_events], [], false
----------------------------------------------------------------------
Ran 1 test in 55.000s
FAILED (errors=1)
Generating XML reports...
Generated XML report: target/test-reports/TEST-pyspark.sql.tests.connect.streaming.test_parity_listener.StreamingListenerParityTests-20230824115154.xml
Had test failures in pyspark.sql.tests.connect.streaming.test_parity_listener with python3.9; see logs.
Error: running /__w/spark/spark/python/run-tests --modules=pyspark-connect --parallelism=1 ; received return code 255
Error: Process completed with exit code 19.
```
@WweiL Are there any related PRs that have not been merged into branch-3.5? The branch-3.5 daily test failed today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org