You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Łukasz Rogalski (Jira)" <ji...@apache.org> on 2022/07/22 11:58:00 UTC
[jira] [Updated] (SPARK-39842) Inconsistent parsing of unusually encoded CSV
[ https://issues.apache.org/jira/browse/SPARK-39842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Łukasz Rogalski updated SPARK-39842:
------------------------------------
Description:
MCVE with PySpark code (but inherently in Spark itself, not PySpark):
{code:java}
spark = SparkSession.builder.appName("...").getOrCreate()
min_schema = StructType(
[
StructField("dummy_col", StringType(), True),
StructField("record_id", IntegerType(), nullable=False),
StructField("dummy_after", StringType(), nullable=False),
]
)
df = (
spark.read.option("mode", "FAILFAST")
.option("quote", '"')
.option("escape", '"')
.option("inferSchema", "false")
.option("multiline", "true")
.option("ignoreLeadingWhiteSpace", "true")
.option("ignoreTrailingWhiteSpace", "true")
.schema(min_schema)
.csv(f'min_repro.csv', header=True)
) {code}
min_repro.csv:
{code:java}
dummy_col,record_id,dummy_after
"",1,", Unusual value with comma included"
B,2,"Unusual value with escaped quote and comma ""like, this" {code}
Spark behaves inconsistently
1. Collect works fine:
{code:java}
df.collect()
[Row(dummy_col=None, record_id=1, dummy_after=', Unusual value with comma included'),
Row(dummy_col='B', record_id=2, dummy_after='Unusual value with escaped quote and comma "like, this')] {code}
2. Trivial query on same DF fails
{code:java}
if df.count() != df.select('record_id').distinct().count():
pass {code}
Error:
{code:java}
Py4JJavaError: An error occurred while calling o357.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage 17.0 (TID 13, localhost, executor driver): org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST.
...
Caused by: java.lang.NumberFormatException: For input string: "Unusual value with comma included""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) {code}
I understand that input file encoding is unusual, but Spark still shouldn't parse it differently between one or other dataframe method.
was:
MCVE with PySpark code (but inherently in Spark itself, not PySpark):
{code:java}
spark = SparkSession.builder.appName("...").getOrCreate()
min_schema = StructType(
[
StructField("dummy_col", StringType(), True),
StructField("record_id", IntegerType(), nullable=False),
StructField("dummy_after", StringType(), nullable=False),
]
)
df = (
spark.read.option("mode", "FAILFAST")
.option("quote", '"')
.option("escape", '"')
.option("inferSchema", "false")
.option("multiline", "true")
.option("ignoreLeadingWhiteSpace", "true")
.option("ignoreTrailingWhiteSpace", "true")
.schema(min_schema)
.csv(f'min_repro.csv', header=True)
) {code}
min_repro.csv:
{code:java}
dummy_col,record_id,dummy_after
"",1,", Unusual value with comma included"
B,2,"Unusual value with escaped quote and comma ""like, this" {code}
Spark behaves inconsistently
1. Collect works fine:
{code:java}
df.collect()
[Row(dummy_col=None, record_id=1, dummy_after=', Unusual value with comma included'),
Row(dummy_col='B', record_id=2, dummy_after='Unusual value with escaped quote and comma "like, this')] {code}
2. Trivial query on same DF fails
{code:java}
if df.count() != df.select('record_id').distinct().count():
pass {code}
Error:
{code:java}
Py4JJavaError: An error occurred while calling o357.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage 17.0 (TID 13, localhost, executor driver): org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST.
...
Caused by: java.lang.NumberFormatException: For input string: "Unusual value with comma included""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) {code}
I understand that input file encoding is unusual, but Spark still shouldn't parse it differently between one or other dataframe method.
> Inconsistent parsing of unusually encoded CSV
> ---------------------------------------------
>
> Key: SPARK-39842
> URL: https://issues.apache.org/jira/browse/SPARK-39842
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 3.2.2
> Environment: pyspark 3.3.2, local mode
> Reporter: Łukasz Rogalski
> Priority: Major
>
> MCVE with PySpark code (but inherently in Spark itself, not PySpark):
> {code:java}
> spark = SparkSession.builder.appName("...").getOrCreate()
> min_schema = StructType(
> [
> StructField("dummy_col", StringType(), True),
> StructField("record_id", IntegerType(), nullable=False),
> StructField("dummy_after", StringType(), nullable=False),
> ]
> )
> df = (
> spark.read.option("mode", "FAILFAST")
> .option("quote", '"')
> .option("escape", '"')
> .option("inferSchema", "false")
> .option("multiline", "true")
> .option("ignoreLeadingWhiteSpace", "true")
> .option("ignoreTrailingWhiteSpace", "true")
> .schema(min_schema)
> .csv(f'min_repro.csv', header=True)
> ) {code}
> min_repro.csv:
> {code:java}
> dummy_col,record_id,dummy_after
> "",1,", Unusual value with comma included"
> B,2,"Unusual value with escaped quote and comma ""like, this" {code}
> Spark behaves inconsistently
>
> 1. Collect works fine:
> {code:java}
> df.collect()
> [Row(dummy_col=None, record_id=1, dummy_after=', Unusual value with comma included'),
> Row(dummy_col='B', record_id=2, dummy_after='Unusual value with escaped quote and comma "like, this')] {code}
> 2. Trivial query on same DF fails
> {code:java}
> if df.count() != df.select('record_id').distinct().count():
> pass {code}
> Error:
> {code:java}
> Py4JJavaError: An error occurred while calling o357.count.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage 17.0 (TID 13, localhost, executor driver): org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST.
> ...
> Caused by: java.lang.NumberFormatException: For input string: "Unusual value with comma included""
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) {code}
>
> I understand that input file encoding is unusual, but Spark still shouldn't parse it differently between one or other dataframe method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org