[jira] [Created] (SPARK-44226) dropDuplicates prevents correct metadata caching

["Darren Cheung (Jira)" <ji...@apache.org>]

Darren Cheung created SPARK-44226:
-------------------------------------

             Summary: dropDuplicates prevents correct metadata caching
                 Key: SPARK-44226
                 URL: https://issues.apache.org/jira/browse/SPARK-44226
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.3.2
            Reporter: Darren Cheung


{quote}given a df, applying dropDuplicates (dD) AND after writing+reading it to s3 (cache), the metadata is deleted

applying just dD or just cache preserves metadata

however, the issue is fixed by recreating the df after dD

{{spark.createDataFrame(dedupedPositiveUsersDf.rdd, dedupedPositiveUsersDf.schema)}}

this leads to the conclusion that there is some issue with dD (affects the df in some unknown way that prevents metadata caching)

unsure why this is

```

val dfWithVector = spark.createDataFrame(

spark.sparkContext.parallelize(denseData),

StructType(schema)

)

println("metadata before:" + dfWithVector.schema("features").metadata)

var dfDroppedDuplicates = dfWithVector.dropDuplicates("id")

println("metadata after duplicates dropped:" + droppedDuplicatesWithOverwrittenSchema.schema("features").metadata)

dfDroppedDuplicates.write.mode("overwrite").parquet(outputLocation)

val dfRead = spark.read.parquet(outputLocation)

println("metadata after caching: " + dfRead.schema("features").metadata)

```
{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org