You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "harshmotw-db (via GitHub)" <gi...@apache.org> on 2024/04/30 22:07:22 UTC
[PR] [SPARK-45891][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
harshmotw-db opened a new pull request, #46311:
URL: https://github.com/apache/spark/pull/46311
### What changes were proposed in this pull request?
Added a check in the `is_variant_null` expression where the length of the value field is verified to be greater than zero. If the length is zero, a `MALFORMED_VARIANT` exception is thrown.
### Why are the changes needed?
Earlier, `is_variant_null` was simply checking if the first byte of a variant value was zero. However, if the value field is empty, the first byte logically doesn't exist and therefore, it could result in undefined behavior. Such a case should ideally never be seen but it could appear in the case of data corruption.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Additional unit test to check if the zero-length variant throws an exception.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #46311:
URL: https://github.com/apache/spark/pull/46311#issuecomment-2089353288
Yep. Merged to master for Apache Spark 4.0.0. Thank you, @harshmotw-db .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #46311:
URL: https://github.com/apache/spark/pull/46311#discussion_r1585892152
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -58,9 +59,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
- // Variant NULL is denoted by basic_type == 0 and val_header == 0
- variantValue(0) == 0
- }
+ if(variantValue.isEmpty) {
Review Comment:
I guess this fails at `Scala` linter? Could you double-check this, @harshmotw-db ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #46311:
URL: https://github.com/apache/spark/pull/46311#discussion_r1585892540
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -58,9 +59,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
- // Variant NULL is denoted by basic_type == 0 and val_header == 0
- variantValue(0) == 0
- }
+ if(variantValue.isEmpty) {
+ throw new SparkRuntimeException("MALFORMED_VARIANT", Map.empty)
Review Comment:
Maybe, shall we use `QueryExecutionErrors`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #46311:
URL: https://github.com/apache/spark/pull/46311#discussion_r1585910343
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -58,9 +59,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
- // Variant NULL is denoted by basic_type == 0 and val_header == 0
- variantValue(0) == 0
- }
+ if(variantValue.isEmpty) {
Review Comment:
Resolved
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -58,9 +59,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
- // Variant NULL is denoted by basic_type == 0 and val_header == 0
- variantValue(0) == 0
- }
+ if(variantValue.isEmpty) {
+ throw new SparkRuntimeException("MALFORMED_VARIANT", Map.empty)
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #46311: [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression
URL: https://github.com/apache/spark/pull/46311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #46311:
URL: https://github.com/apache/spark/pull/46311#discussion_r1585892540
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -58,9 +59,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
- // Variant NULL is denoted by basic_type == 0 and val_header == 0
- variantValue(0) == 0
- }
+ if(variantValue.isEmpty) {
+ throw new SparkRuntimeException("MALFORMED_VARIANT", Map.empty)
Review Comment:
Shall we use `QueryExecutionErrors`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on PR #46311:
URL: https://github.com/apache/spark/pull/46311#issuecomment-2089347930
Thanks for the reviews @cloud-fan @HyukjinKwon @dongjoon-hyun. The tests are completed and it is ready to be merged now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org