You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/22 07:25:11 UTC
[GitHub] [spark] rednaxelafx opened a new pull request, #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
rednaxelafx opened a new pull request, #37252:
URL: https://github.com/apache/spark/pull/37252
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Update the `UnsafeRow` structural integrity check in `UnsafeRowUtils. validateStructuralIntegrity` to handle a special case with null variable-length DecimalType value.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The check should follow the format that `UnsafeRowWriter` produces. In general, `UnsafeRowWriter` clears out a field with zero when the field is set to be null, c.f. `UnsafeRowWriter.setNullAt(ordinal)` and `UnsafeRow.setNullAt(ordinal)`.
But there's a special case for `DecimalType` values: this is the only type that is both:
- can be fixed-length or variable-length, depending on the precision, and
- is mutable in `UnsafeRow`.
To support a variable-length `DecimalType` to be mutable in `UnsafeRow`, the `UnsafeRowWriter` always leaves a 16-byte space in the variable-length section of the `UnsafeRow` (tail end of the row), regardless of whether the `Decimal` value being written is null or not. In the fixed-length part of the field, it would be an "OffsetAndSize", and the `offset` part always points to the start offset of the variable-length part of the field, while the `size` part will either be `0` for the null value, or `1` to at most `16` for non-null values.
When `setNullAt(ordinal)` is called instead of passing a null value to `write(int, Decimal, int, int)`, however, the `offset` part gets zero'd out and this field stops being mutable. There's a comment on `UnsafeRow.setDecimal` that mentions to keep this field able to support updates, `setNullAt(ordinal)` cannot be called, but there's no code enforcement of that.
So we need to recognize that in the structural integrity check and allow variable-length `DecimalType` to have non-zero field even for null.
Note that for non-null values, the existing check does conform to the format from `UnsafeRowWriter`. It's only null value of variable-length `DecimalType` that'd trigger a bug, which can affect Structured Streaming's checkpoint file read where this check is applied.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, previously the `UnsafeRow` structural integrity validation will return false positive for correct data, when there's a null value in a variable-length `DecimalType` field. The fix will no longer return false positive.
Because the Structured Streaming checkpoint file validation uses this check, previously a good checkpoint file may be rejected by the check, and the only workaround is to disable the check; with the fix, the correct checkpoint file will be allowed to load.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Added new test case in `UnsafeRowUtilsSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1197515372
Thank you very much, @HeartSaVioR !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37252:
URL: https://github.com/apache/spark/pull/37252#discussion_r927462656
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:
##########
@@ -74,13 +79,36 @@ object UnsafeRowUtils {
if ((row.getLong(index) >> 32) != 0L) return false
case _ =>
}
- case (_, index) if row.isNullAt(index) =>
- if (row.getLong(index) != 0L) return false
+ case (field, index) if row.isNullAt(index) =>
+ field.dataType match {
+ case dt: DecimalType if !UnsafeRow.isFixedLength(dt) =>
+ // See special case in UnsafeRowWriter.write(int, Decimal, int, int) and
+ // UnsafeRow.setDecimal(int, Decimal, int).
+ // A variable-length Decimal may be marked as null while having non-zero offset and
+ // zero length. This allows the field to be updated (i.e. mutable variable-length data)
+ val (offset, size) = getOffsetAndSize(row, index)
+ if (size != 0 || // size must be zero
+ offset != 0 && // offsetAndSize may be entirely zero if row.setNullAt(index),
+ // otherwise range check offset
+ (offset < bitSetWidthInBytes + 8 * row.numFields ||
+ offset + size > rowSizeInBytes)) {
Review Comment:
```suggestion
// Check the integrity of the DecimalType in UnsafeRow:
// 1. size must be zero
// 2. ...
// 3. ...
if (size != 0 || offset != 0 && (offset < bitSetWidthInBytes + 8 * row.numFields || offset + size > rowSizeInBytes)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37252:
URL: https://github.com/apache/spark/pull/37252#discussion_r928117562
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:
##########
@@ -74,13 +79,36 @@ object UnsafeRowUtils {
if ((row.getLong(index) >> 32) != 0L) return false
case _ =>
}
- case (_, index) if row.isNullAt(index) =>
- if (row.getLong(index) != 0L) return false
+ case (field, index) if row.isNullAt(index) =>
+ field.dataType match {
+ case dt: DecimalType if !UnsafeRow.isFixedLength(dt) =>
+ // See special case in UnsafeRowWriter.write(int, Decimal, int, int) and
+ // UnsafeRow.setDecimal(int, Decimal, int).
+ // A variable-length Decimal may be marked as null while having non-zero offset and
+ // zero length. This allows the field to be updated (i.e. mutable variable-length data)
+ val (offset, size) = getOffsetAndSize(row, index)
+ if (size != 0 || // size must be zero
+ offset != 0 && // offsetAndSize may be entirely zero if row.setNullAt(index),
+ // otherwise range check offset
+ (offset < bitSetWidthInBytes + 8 * row.numFields ||
+ offset + size > rowSizeInBytes)) {
Review Comment:
Yes. sure. I just give a code reference!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1193073290
I can see that the test relevant to this PR passed the PR test action:
```
2022-07-22T19:01:28.3310666Z [info] UnsafeRowUtilsSuite:
2022-07-22T19:01:28.3311658Z [info] - UnsafeRow format invalidation (3 milliseconds)
2022-07-22T19:01:28.3312437Z [info] - Handle special case for null variable-length Decimal (5 milliseconds)
```
But there are other unrelated errors failing build/test. Could somebody please help with the test infra? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on a diff in pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on code in PR #37252:
URL: https://github.com/apache/spark/pull/37252#discussion_r927878312
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:
##########
@@ -74,13 +79,36 @@ object UnsafeRowUtils {
if ((row.getLong(index) >> 32) != 0L) return false
case _ =>
}
- case (_, index) if row.isNullAt(index) =>
- if (row.getLong(index) != 0L) return false
+ case (field, index) if row.isNullAt(index) =>
+ field.dataType match {
+ case dt: DecimalType if !UnsafeRow.isFixedLength(dt) =>
+ // See special case in UnsafeRowWriter.write(int, Decimal, int, int) and
+ // UnsafeRow.setDecimal(int, Decimal, int).
+ // A variable-length Decimal may be marked as null while having non-zero offset and
+ // zero length. This allows the field to be updated (i.e. mutable variable-length data)
+ val (offset, size) = getOffsetAndSize(row, index)
+ if (size != 0 || // size must be zero
+ offset != 0 && // offsetAndSize may be entirely zero if row.setNullAt(index),
+ // otherwise range check offset
+ (offset < bitSetWidthInBytes + 8 * row.numFields ||
+ offset + size > rowSizeInBytes)) {
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1197495648
Thanks! Merging to master/3.3/3.2 (3.1 is considered as EOL since 3.1.1 was released in Jan 2021)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1196167832
@rednaxelafx Could you please rebase once? The community seems to make progress of the GA issue and it'd be nice if we participate to verify the effort. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1197494336
Yes, recent test failure came from ImageFileFormatSuite which doesn't seem to be related. Let me merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1197112149
@HeartSaVioR I've rebased to latest master yesterday, but it seems like it's still running into unrelated test failures (but the symptom seem to be different from before).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1193189931
I'll merge this once we got the confirmation in https://github.com/apache/spark/pull/37248#issuecomment-1193118952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
URL: https://github.com/apache/spark/pull/37252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on a diff in pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on code in PR #37252:
URL: https://github.com/apache/spark/pull/37252#discussion_r927846790
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:
##########
@@ -74,13 +79,36 @@ object UnsafeRowUtils {
if ((row.getLong(index) >> 32) != 0L) return false
case _ =>
}
- case (_, index) if row.isNullAt(index) =>
- if (row.getLong(index) != 0L) return false
+ case (field, index) if row.isNullAt(index) =>
+ field.dataType match {
+ case dt: DecimalType if !UnsafeRow.isFixedLength(dt) =>
+ // See special case in UnsafeRowWriter.write(int, Decimal, int, int) and
+ // UnsafeRow.setDecimal(int, Decimal, int).
+ // A variable-length Decimal may be marked as null while having non-zero offset and
+ // zero length. This allows the field to be updated (i.e. mutable variable-length data)
+ val (offset, size) = getOffsetAndSize(row, index)
+ if (size != 0 || // size must be zero
+ offset != 0 && // offsetAndSize may be entirely zero if row.setNullAt(index),
+ // otherwise range check offset
+ (offset < bitSetWidthInBytes + 8 * row.numFields ||
+ offset + size > rowSizeInBytes)) {
Review Comment:
Thank you very much for the suggestion! The suggested version of Line 94 is going to exceed 100 character line width limit, so please allow me to still break the predicate into multiple lines. But I'll extract the inlilne comments to above the `if` as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37252: [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37252:
URL: https://github.com/apache/spark/pull/37252#issuecomment-1193081431
Same here - https://github.com/apache/spark/pull/37248#issuecomment-1193081390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org