You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maciej Szymkiewicz (Jira)" <ji...@apache.org> on 2021/12/18 12:41:00 UTC
[jira] [Created] (SPARK-37683) Nondeterministic type checker error in data tests
Maciej Szymkiewicz created SPARK-37683:
------------------------------------------
Summary: Nondeterministic type checker error in data tests
Key: SPARK-37683
URL: https://issues.apache.org/jira/browse/SPARK-37683
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.3.0
Reporter: Maciej Szymkiewicz
Occasionally, I see different errors around {{_jvm}}, {{_wrapped}} and {{_jsparkSession}} popping out in data tests
{code}
E ../../spark/python/pyspark/sql/catalog:71: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/catalog:72: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/catalog:349: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/catalog:360: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/column:920: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:400: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:407: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:413: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:602: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:604: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:620: error: Cannot determine type of "_jvm" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:621: error: Cannot determine type of "_jvm" [has-type] (diff)
E ../../spark/python/pyspark/sql/pandas/conversion:622: error: Cannot determine type of "_wrapped" [has-type] (diff)
E ../../spark/python/pyspark/sql/readwriter:115: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/readwriter:307: error: Cannot determine type of "_jvm" [has-type] (diff)
E ../../spark/python/pyspark/sql/readwriter:308: error: Cannot determine type of "_jvm" [has-type] (diff)
E ../../spark/python/pyspark/sql/session:283: error: Cannot determine type of "_jvm" [has-type] (diff)
E ../../spark/python/pyspark/sql/session:284: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/streaming:377: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/udf:225: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/udf:456: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/udf:509: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/udf:510: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
E ../../spark/python/pyspark/sql/udf:533: error: Cannot determine type of "_jsparkSession" [has-type] (diff)
{code}
The set of errors is not always the same, so it indicates so issues with type checker.
I am going to add {{casts}} and / or annotations for these lines, so we don't get flaky tests.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org