You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/04/18 05:25:36 UTC
[PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
yadavay-amzn opened a new pull request, #55405:
URL: https://github.com/apache/spark/pull/55405
### What changes were proposed in this pull request?
Clarify the documentation for `nullValue` and `emptyValue` CSV reader options to explain how they differ and when each applies.
### Why are the changes needed?
The current descriptions of `nullValue` and `emptyValue` are very similar and do not explain the distinction between the two. This causes confusion for users trying to handle null and empty values in CSV files.
### Does this PR introduce _any_ user-facing change?
No code change. Documentation only.
### How was this patch tested?
Documentation change only.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55405: [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options
URL: https://github.com/apache/spark/pull/55405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "waterlx (via GitHub)" <gi...@apache.org>.
waterlx commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-4349329997
> > @yadavay-amzn Thanks for the patch! May you review my comment to see if it makes any sense to you?
>
> @waterlx replied to JIRA comment asking for feedback from the previous developers who worked on this.
>
> Looked to me like PR #22234 is unrelated to [SPARK-56429](https://issues.apache.org/jira/browse/SPARK-56429), so we still need this PR to fix the issue. May be I'm missing something?
@yadavay-amzn Sorry if I made you confused and I did not quite get you.
Yes, PR #22234 does not relate to this PR you are working on, and I do think your PR is great to remove the vagueness of current doc, especially for new comers (like me).
I am not sure if my second comment made you confused, sorry for that if it did. Please allow me to explain:
There are 2 cases if specifing`.options("nullValue", "NA")`
(1) `NA` in CSV will be read as null.
(2) `NA` quoted by the "quote" specification, for example `"NA"`, in CSV, will be also read as null. "quote" is also an option which be specified when reading CSV. See https://spark.apache.org/docs/latest/sql-data-sources-csv.html
My point is may you consider adding (2) into your PR?. Making any sense to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-4347607837
> @yadavay-amzn Thanks for the patch! May you review my comment to see if it makes any sense to you?
@waterlx replied to JIRA comment asking for feedback from the previous developers who worked on this.
Looked to me like PR #22234 is unrelated to SPARK-56429, so we still need this PR to fix the issue. May be I'm missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-4426081808
@waterlx Good point — updated the `nullValue` description to note that matching also applies when the value is enclosed by the `quote` character. Thanks for the suggestion!
@MaxGekk @HyukjinKwon Would appreciate a review when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "waterlx (via GitHub)" <gi...@apache.org>.
waterlx commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-4305713545
@yadavay-amzn Thanks for the patch! May you review my comment to see if it makes any sense to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "waterlx (via GitHub)" <gi...@apache.org>.
waterlx commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-4349385959
@mmolimar It is regarding #22234 about `nullValue` when reading CSV. I happened to find that the `nullValue` specification quoted by `quote` specification, will be also read as null. Say `.options("nullValue", "NA")` makes `"NA"` (`quote` defaults to a double quote) being read as null, in additional to `NA`. May you comment if it works as your original design?
@MaxGekk @HyukjinKwon May you review this PR by @yadavay-amzn, since you also worked on or reviewed #22234. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56429][DOCS] Clarify differences between nullValue and emptyValue CSV options [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55405:
URL: https://github.com/apache/spark/pull/55405#issuecomment-5349625701
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org