You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yikun Jiang (Jira)" <ji...@apache.org> on 2021/07/07 08:19:00 UTC
[jira] [Updated] (SPARK-36031) Keep same behavior with pandas for
operations of series with nan
[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yikun Jiang updated SPARK-36031:
--------------------------------
Description:
There are many operations for series doen't follow the pandas, such as:
>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser.astype(int)
Traceback (most recent call last):
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-30-1ca2ff8756d2>", line 1, in <module>
pser.astype(int)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/generic.py", line 5548, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 604, in astype
return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 409, in apply
applied = getattr(b, f)(**kwargs)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 595, in astype
values = astype_nansafe(vals1d, dtype, copy=True)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 968, in astype_nansafe
raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
ValueError: Cannot convert non-finite values (NA or inf) to integer
>>> psser.astype(int)
0 1.0
1 2.0
2 NaN
dtype: float64
>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser ** False
pser ** False
Out[6]:
0 1.0
1 1.0
2 1.0
dtype: float64
>>> psser ** False
0 1.0
1 1.0
2 NaN
dtype: float64
was:
There are many operations for series doen't follow the pandas, such as:
>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser.astype(int)
Traceback (most recent call last):
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-30-1ca2ff8756d2>", line 1, in <module>
pser.astype(int)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/generic.py", line 5548, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 604, in astype
return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 409, in apply
applied = getattr(b, f)(**kwargs)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 595, in astype
values = astype_nansafe(vals1d, dtype, copy=True)
File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 968, in astype_nansafe
raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
ValueError: Cannot convert non-finite values (NA or inf) to integer
>>> psser.astype(int)
0 1.0
1 2.0
2 NaN
dtype: float64
>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser ** False
pser ** False
Out[6]:
0 1.0
1 1.0
2 1.0
dtype: float64
>>> psser ** False
0 1.0
1 1.0
2 NaN
dtype: float64
> Keep same behavior with pandas for operations of series with nan
> -----------------------------------------------------------------
>
> Key: SPARK-36031
> URL: https://issues.apache.org/jira/browse/SPARK-36031
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.2.0, 3.3.0
> Reporter: Yikun Jiang
> Priority: Major
>
> There are many operations for series doen't follow the pandas, such as:
> >>> pser = pd.Series([1, 2, np.nan], dtype=float)
> >>> psser = ps.from_pandas(pser)
> >>> pser.astype(int)
> Traceback (most recent call last):
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
> exec(code_obj, self.user_global_ns, self.user_ns)
> File "<ipython-input-30-1ca2ff8756d2>", line 1, in <module>
> pser.astype(int)
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/generic.py", line 5548, in astype
> new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 604, in astype
> return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 409, in apply
> applied = getattr(b, f)(**kwargs)
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 595, in astype
> values = astype_nansafe(vals1d, dtype, copy=True)
> File "/Users/jiangyikun/venv36/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 968, in astype_nansafe
> raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
> ValueError: Cannot convert non-finite values (NA or inf) to integer
> >>> psser.astype(int)
> 0 1.0
> 1 2.0
> 2 NaN
> dtype: float64
> >>> pser = pd.Series([1, 2, np.nan], dtype=float)
> >>> psser = ps.from_pandas(pser)
> >>> pser ** False
> pser ** False
> Out[6]:
> 0 1.0
> 1 1.0
> 2 1.0
> dtype: float64
> >>> psser ** False
> 0 1.0
> 1 1.0
> 2 NaN
> dtype: float64
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org