[jira] [Created] (SPARK-40154) PySpark: DataFrame.cache docstring gives wrong storage level

["Paul Staab (Jira)" <ji...@apache.org>]

Paul Staab created SPARK-40154:
----------------------------------

             Summary: PySpark: DataFrame.cache docstring gives wrong storage level
                 Key: SPARK-40154
                 URL: https://issues.apache.org/jira/browse/SPARK-40154
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.3.0
            Reporter: Paul Staab


The docstring of the `DataFrame.cache()` methods currently states that it uses a serialized storage level
{code:java}
Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK`).
[...]
-        The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.{code}
while `DataFrame.persists()` states that it uses deserialized storage level
{code:java}
If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
[...]
The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala in 3.0.{code}
 

However, in practice both `.cache()` and `.persist()` use deserialized storage levels:


{code:java}
import pyspark
from pyspark.sql import SparkSession
from pyspark import StorageLevel
print(pyspark.__version__)
# 3.3.0

spark = SparkSession.builder.master("local[2]").getOrCreate()

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.cache()
df.count()
# Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.persist()
df.count()
# Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.persist(StorageLevel.MEMORY_AND_DISK)
df.count()
# Storage level in Spark UI: "Disk Memory Serialized 1x Replicated"{code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org