You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/04/19 20:55:55 UTC
[GitHub] [iceberg] RussellSpitzer opened a new pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
RussellSpitzer opened a new pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495
Previously the boxed value of contains NaN would be null if the table
was made before NaN stats were implemented. This leads to an NPE when
being unboxed. To fix this we use the Boxed boolean and on null report
"true" since we do not know if the partition contains any NaN values. In
addition we move this below the "allValuesNull" check just in case that
would have allowed us to ignore the manifest since the values are all null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616231093
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -60,7 +60,7 @@ boolean canContain(Object value) {
}
if (NaNUtil.isNaN(value)) {
- return containsNaN;
+ return containsNaN == null ? true : containsNaN;
Review comment:
I'd consider doing this in the constructor to avoid calling the ternary operation in each `canContain`.
```
this.containsNaN = summary.containsNaN() == null || summary.containsNaN();
```
Then this will also be a one line change. I'll leave this up to you, @RussellSpitzer @yyanyy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer merged pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
RussellSpitzer merged pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616264801
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -60,7 +60,7 @@ boolean canContain(Object value) {
}
if (NaNUtil.isNaN(value)) {
- return containsNaN;
+ return containsNaN == null ? true : containsNaN;
Review comment:
I'd rather keep it explicit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616216979
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -59,16 +59,16 @@ boolean canContain(Object value) {
return containsNull;
}
- if (NaNUtil.isNaN(value)) {
- return containsNaN;
- }
-
// if lower bound is null, then there are no non-null values
if (lowerBound == null) {
// the value is non-null, so it cannot match
return false;
Review comment:
Fixed !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616194414
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -59,16 +59,16 @@ boolean canContain(Object value) {
return containsNull;
}
- if (NaNUtil.isNaN(value)) {
- return containsNaN;
- }
-
// if lower bound is null, then there are no non-null values
if (lowerBound == null) {
// the value is non-null, so it cannot match
return false;
Review comment:
I think @yyanyy is correct. It was my bad for suggesting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616182133
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -59,16 +59,16 @@ boolean canContain(Object value) {
return containsNull;
}
- if (NaNUtil.isNaN(value)) {
- return containsNaN;
- }
-
// if lower bound is null, then there are no non-null values
if (lowerBound == null) {
// the value is non-null, so it cannot match
return false;
Review comment:
I think if the input is NaN and the file is written with the new code (`NaN` flips `containsNaN` to true without touching upper/lower bound) then I think we will return false here which wouldn't be correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#issuecomment-822843292
LGTM. I'd probably do the check once in the constructor but I am not sure it is a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2495: Core: Fix NPE caused by Unboxing a Null in ManifestFileUtil (#2492)
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2495:
URL: https://github.com/apache/iceberg/pull/2495#discussion_r616212799
##########
File path: core/src/main/java/org/apache/iceberg/util/ManifestFileUtil.java
##########
@@ -59,16 +59,16 @@ boolean canContain(Object value) {
return containsNull;
}
- if (NaNUtil.isNaN(value)) {
- return containsNaN;
- }
-
// if lower bound is null, then there are no non-null values
if (lowerBound == null) {
// the value is non-null, so it cannot match
return false;
Review comment:
Ok I think i understand the issue here,
Previously we would report "null" for the lowerBound when we meant "NaN" because we weren't recording NaNs, this means if we see a missing lowerBound we either could have a mix of NaN and Null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org