You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "itholic (via GitHub)" <gi...@apache.org> on 2023/06/20 06:42:40 UTC
[GitHub] [spark] itholic commented on a diff in pull request #41670: [WIP][SPARK-43631][CONNECT][PS] Enable Series.interpolate with Spark Connect
itholic commented on code in PR #41670:
URL: https://github.com/apache/spark/pull/41670#discussion_r1234805869
##########
python/pyspark/pandas/series.py:
##########
@@ -2257,27 +2257,44 @@ def _interpolate(
return self._psdf.copy()._psser_for(self._column_label)
scol = self.spark.column
- sql_utils = SparkContext._active_spark_context._jvm.PythonSQLUtils
- last_non_null = PySparkColumn(sql_utils.lastNonNull(scol._jc))
- null_index = PySparkColumn(sql_utils.nullIndex(scol._jc))
+ if is_remote():
+ from pyspark.sql.connect.functions import _invoke_function_over_columns
+ last_non_null = _invoke_function_over_columns(
+ "last_non_null",
+ scol, # type: ignore[arg-type]
+ )
+ null_index = _invoke_function_over_columns(
+ "null_index",
+ scol, # type: ignore[arg-type]
+ )
+ else:
+ sql_utils = SparkContext._active_spark_context._jvm.PythonSQLUtils
+ last_non_null = PySparkColumn(
+ sql_utils.lastNonNull(scol._jc) # type: ignore[assignment]
+ )
+ null_index = PySparkColumn(sql_utils.nullIndex(scol._jc)) # type: ignore[assignment]
Review Comment:
Hi, @zhengruifeng Could you happen to help reviewing this PR when you find some time??
I made some modifications to remove JVM dependencies from `last_non_null` and `null_index` based on other similar PRs, but it still doesn't work properly in Spark Connect as follows:
```python
>>> import pyspark.pandas as ps
>>> import numpy as np
>>> psser = ps.Series([1, np.nan, 3], name="a")
>>> psser.interpolate()
/.../spark/python/pyspark/sql/connect/expressions.py:945: UserWarning: WARN WindowExpression: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
warnings.warn(
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/pandas/series.py", line 2232, in interpolate
return self._interpolate(
File "/.../spark/python/pyspark/pandas/series.py", line 2372, in _interpolate
self._psdf._internal.with_new_spark_column(self._column_label, scol) # TODO: dtype?
File "/.../spark/python/pyspark/pandas/internal.py", line 1420, in with_new_spark_column
return self.with_new_columns(
File "/.../spark/python/pyspark/pandas/internal.py", line 1368, in with_new_columns
return self.copy(
File "/.../spark/python/pyspark/pandas/internal.py", line 1486, in copy
return InternalFrame(
File "/.../spark/python/pyspark/pandas/internal.py", line 743, in __init__
schema = spark_frame.select(data_spark_columns).schema
File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 1665, in schema
return self._session.client.schema(query)
File "/.../spark/python/pyspark/sql/connect/client/core.py", line 783, in schema
schema = self._analyze(method="schema", plan=plan).schema
File "/.../spark/python/pyspark/sql/connect/client/core.py", line 964, in _analyze
self._handle_error(error)
File "/.../spark/python/pyspark/sql/connect/client/core.py", line 1212, in _handle_error
self._handle_rpc_error(error)
File "/.../spark/python/pyspark/sql/connect/client/core.py", line 1248, in _handle_rpc_error
raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.AnalysisException: Invalid call to dataType on unresolved object
```
Looking at the error message, it seems that somewhere the column is not being resolved correctly. Is there any possible cause that you can speculate about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org