You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2023/08/25 03:34:00 UTC
[jira] [Resolved] (SPARK-44946) toPandas() gives FutureWarning when containing columns of datatype timestamp
[ https://issues.apache.org/jira/browse/SPARK-44946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-44946.
----------------------------------
Resolution: Cannot Reproduce
> toPandas() gives FutureWarning when containing columns of datatype timestamp
> ----------------------------------------------------------------------------
>
> Key: SPARK-44946
> URL: https://issues.apache.org/jira/browse/SPARK-44946
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.4.1
> Environment: For my test, I ran it in a docker container:
> * Python version: python 3.10 (base image python:3.10-slim-bullseye)
> * Java: openjdk-17-jre-headless
> * Spark: 3.4.1
> * pandas: 1.5.3
> * pyarrow: 11.0.0
> Reporter: Matthias Roels
> Priority: Major
>
> When converting a Spark DataFrame into a pandas DataFrame, we get a FutureWarning when the DataFrame contains columns of type {{timestamp. }}
> Reproducible example (that you can run locally):
> {code:java}
> from datetime import datetime
> from pyspark.sql import SparkSession
> import pandas as pd
> spark = SparkSession.builder.getOrCreate()
> df = pd.DataFrame({"foo": [datetime(2023, 1, 1), datetime(2023, 1, 1)]})
> df_sp = spark.createDataFrame(df)
> test = df_sp.toPandas()
> // warning logs:
> /usr/local/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251: FutureWarning: Passing unit-less datetime64 dtype to .astype is deprecated and will raise in a future version. Pass 'datetime64[ns]' instead
> {code}
> Note that if we enable arrow (by setting {{{}config("spark.sql.execution.arrow.pyspark.enabled", "true"){}}}), this warning is gone! Although I admit that I have seen it popping up once with arrow enabled too, but I could not create a reproducible example out of that.
>
> Note that this basically means that I cannot use Spark with pandas 2.0 without Arrow enabled...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org