You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2023/04/17 00:57:00 UTC
[jira] [Updated] (SPARK-43158) Set upperbound of pandas version in binder integrations
[ https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-43158:
---------------------------------
Summary: Set upperbound of pandas version in binder integrations (was: Set upperbound of pandas version in binder integartions)
> Set upperbound of pandas version in binder integrations
> -------------------------------------------------------
>
> Key: SPARK-43158
> URL: https://issues.apache.org/jira/browse/SPARK-43158
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark, PySpark
> Affects Versions: 3.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
>
> {code}
> df.toPandas
> {code}
> fails with
> {code}
> ---------------------------------------------------------------------------
> TypeError Traceback (most recent call last)
> Cell In[14], line 1
> ----> 1 df.toPandas()
> File /srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251, in PandasConversionMixin.toPandas(self)
> 248 should_check_timedelta = is_timedelta64_dtype(t) and len(pdf) == 0
> 250 if (t is not None and not is_timedelta64_dtype(t)) or should_check_timedelta:
> --> 251 series = series.astype(t, copy=False)
> 253 with catch_warnings():
> 254 from pandas.errors import PerformanceWarning
> File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/generic.py:6324, in NDFrame.astype(self, dtype, copy, errors)
> 6317 results = [
> 6318 self.iloc[:, i].astype(dtype, copy=copy)
> 6319 for i in range(len(self.columns))
> 6320 ]
> 6322 else:
> 6323 # else, only a single dtype is given
> -> 6324 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
> 6325 return self._constructor(new_data).__finalize__(self, method="astype")
> 6327 # GH 33113: handle empty frame or series
> File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:451, in BaseBlockManager.astype(self, dtype, copy, errors)
> 448 elif using_copy_on_write():
> 449 copy = False
> --> 451 return self.apply(
> 452 "astype",
> 453 dtype=dtype,
> 454 copy=copy,
> 455 errors=errors,
> 456 using_cow=using_copy_on_write(),
> 457 )
> File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
> 350 applied = b.apply(f, **kwargs)
> 351 else:
> --> 352 applied = getattr(b, f)(**kwargs)
> 353 result_blocks = extend_blocks(applied, result_blocks)
> 355 out = type(self).from_blocks(result_blocks, self.axes)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org