You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2019/11/13 17:52:00 UTC
[jira] [Resolved] (SPARK-29875) Avoid to use deprecated
pyarrow.open_stream API in Spark 2.4.x
[ https://issues.apache.org/jira/browse/SPARK-29875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-29875.
-----------------------------------
Fix Version/s: 2.4.5
Resolution: Fixed
Issue resolved by pull request 26501
[https://github.com/apache/spark/pull/26501]
> Avoid to use deprecated pyarrow.open_stream API in Spark 2.4.x
> --------------------------------------------------------------
>
> Key: SPARK-29875
> URL: https://issues.apache.org/jira/browse/SPARK-29875
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.4
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Minor
> Fix For: 2.4.5
>
>
> In Spark 2.4.x, if we use PyArrow higher then 0.12.0, it shows a bunch of warnings as below:
> {code}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> @pandas_udf("integer", PandasUDFType.SCALAR) # doctest: +SKIP
> def add_one(x):
> return x + 1
> spark.range(100).select(add_one("id")).collect()
> {code}
> {code}
> UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org