You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "azmatsiddique (via GitHub)" <gi...@apache.org> on 2026/03/15 07:36:28 UTC
[PR] Spark 46959 [spark]
azmatsiddique opened a new pull request, #54812:
URL: https://github.com/apache/spark/pull/54812
### What changes were proposed in this pull request?
This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (`""`) when the `escape` option is set to an empty string (`""`).
Previously, if `escape=""` was used, mid-line empty quoted strings were correctly resolved to an empty string, but the last column resolved to a literal `"` character. This occurred because Spark Maps `escape=""` to `\u0000`, which univocity’s parser relies on. At the end of a line, without a trailing delimiter, univocity misinterprets the second `"` as an escaped quote rather than a closing quote.
The fix introduces a post-processing step in `UnivocityParser.parseLine` to detect this specific condition (a single quote character as the last token when `escape` is `\u0000`) and replace it with the configured `emptyValueInRead`.
### Why are the changes needed?
To ensure consistent parsing of CSV data regardless of whether an empty quoted string appears in the middle of a line or at the end of a line.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations (e.g. `escape=""`, `quote="\""`, `sep=";"`).
### How was this patch tested?
Added a new regression test in [CSVSuite.scala](cci:7://file:///Users/azmatsiddique/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala:0:0-0:0):
`"SPARK-46959: CSV reader reads data inconsistently depending on column position"`
The test verifies that an empty quoted string behaves identically in the mid-line position (column c) and the end-of-line position (column d) when configured with `escape=""` and `nullValue=""`.
Verified that both `CSVv1Suite` and `CSVv2Suite` pass without regressions.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Spark 46959 [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #54812: Spark 46959
URL: https://github.com/apache/spark/pull/54812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Spark 46959 [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54812:
URL: https://github.com/apache/spark/pull/54812#issuecomment-4784903417
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org