You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/02/04 07:28:13 UTC
[PR] [SPARK-46971][SQL] When the `compression` is null, a `NullPointException` should not be thrown [spark]
panbingkun opened a new pull request, #45015:
URL: https://github.com/apache/spark/pull/45015
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
- Add new UT.
- Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46971][SQL] When the `compression` is null, a `NullPointException` should not be thrown [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45015:
URL: https://github.com/apache/spark/pull/45015#discussion_r1477698620
##########
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:
##########
@@ -118,7 +119,11 @@ private[sql] class AvroOptions(
* taken into account. If the former one is not set too, the `snappy` codec is used by default.
*/
val compression: String = {
- parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ val v = parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ if (v == null) {
Review Comment:
This problem is actually here and there, not only `AvroOptions`. I believe a lot of `CSVOptions` also has this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46971][SQL] When the `compression` is null, a `NullPointException` should not be thrown [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45015:
URL: https://github.com/apache/spark/pull/45015#discussion_r1482479360
##########
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:
##########
@@ -118,7 +119,11 @@ private[sql] class AvroOptions(
* taken into account. If the former one is not set too, the `snappy` codec is used by default.
*/
val compression: String = {
- parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ val v = parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ if (v == null) {
Review Comment:
Would there be compatibility issues if null values are prohibited in `CaseInsensitiveMap`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46971][SQL] When the `compression` is null, a `NullPointException` should not be thrown [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45015:
URL: https://github.com/apache/spark/pull/45015#discussion_r1478853391
##########
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:
##########
@@ -118,7 +119,11 @@ private[sql] class AvroOptions(
* taken into account. If the former one is not set too, the `snappy` codec is used by default.
*/
val compression: String = {
- parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ val v = parameters.get(COMPRESSION).getOrElse(SQLConf.get.avroCompressionCodec)
+ if (v == null) {
Review Comment:
+1 for @HyukjinKwon 's comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org