You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ted Chester Jenks (Jira)" <ji...@apache.org> on 2025/04/08 19:16:00 UTC

[jira] [Created] (SPARK-51746) Data dropped when aggregate on CSV corrupt_record column

Ted Chester Jenks created SPARK-51746:
-----------------------------------------

             Summary: Data dropped when aggregate on CSV corrupt_record column
                 Key: SPARK-51746
                 URL: https://issues.apache.org/jira/browse/SPARK-51746
             Project: Spark
          Issue Type: Bug
          Components: Input/Output
    Affects Versions: 3.5.5
            Reporter: Ted Chester Jenks


When parsing a simple, invalid CSV into Spark (3.5.5) I am able to drop data during aggregation.

The data:
{noformat}
col1,col2,col3
something,corrupt
not,so,bad
bad {noformat}
Reading this with permissive parsing works as expected. However, the following results in the corrupt_record data to be dropped:
{noformat}
df.groupBy("col1").agg(collect_list("corrupt_record").alias("corrupt_record")){noformat}
The bug is fixed if one caches before doing the aggregate.

This is similar to when you try to perform other operations on a corrupt_record column, e.g. [https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/cpd-one-workload-migration-spark-corrupt-csv.html|https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/cpd-one-workload-migration-spark-corrupt-csv.html.]. However, in other cases an informative error is produced, rather than data silently dropped. I think that an error would be better here too.

 

!image-2025-04-08-15-11-11-498.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org