You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2023/04/17 00:49:00 UTC
[jira] [Created] (SPARK-43158) Set lowerbound of pandas version in 3.4.0
Hyukjin Kwon created SPARK-43158:
------------------------------------
Summary: Set lowerbound of pandas version in 3.4.0
Key: SPARK-43158
URL: https://issues.apache.org/jira/browse/SPARK-43158
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 3.4.0
Reporter: Hyukjin Kwon
{code}
df.toPandas
{code}
fails with
{code}
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[14], line 1
----> 1 df.toPandas()
File /srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251, in PandasConversionMixin.toPandas(self)
248 should_check_timedelta = is_timedelta64_dtype(t) and len(pdf) == 0
250 if (t is not None and not is_timedelta64_dtype(t)) or should_check_timedelta:
--> 251 series = series.astype(t, copy=False)
253 with catch_warnings():
254 from pandas.errors import PerformanceWarning
File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/generic.py:6324, in NDFrame.astype(self, dtype, copy, errors)
6317 results = [
6318 self.iloc[:, i].astype(dtype, copy=copy)
6319 for i in range(len(self.columns))
6320 ]
6322 else:
6323 # else, only a single dtype is given
-> 6324 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
6325 return self._constructor(new_data).__finalize__(self, method="astype")
6327 # GH 33113: handle empty frame or series
File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:451, in BaseBlockManager.astype(self, dtype, copy, errors)
448 elif using_copy_on_write():
449 copy = False
--> 451 return self.apply(
452 "astype",
453 dtype=dtype,
454 copy=copy,
455 errors=errors,
456 using_cow=using_copy_on_write(),
457 )
File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
350 applied = b.apply(f, **kwargs)
351 else:
--> 352 applied = getattr(b, f)(**kwargs)
353 result_blocks = extend_blocks(applied, result_blocks)
355 out = type(self).from_blocks(result_blocks, self.axes)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org