You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/09 10:18:12 UTC
[GitHub] [iceberg] lcspinter opened a new pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
lcspinter opened a new pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740
The following casting exceptions were observed when joining two Iceberg tables on Tez:
- java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
- java.time.OffsetDateTime cannot be cast to org.apache.hadoop.hive.common.type.Timestamp
- java.nio.HeapByteBuffer cannot be cast to [B
- java.time.LocalDate cannot be cast to org.apache.hadoop.hive.common.type.Date
Doesn't happen with Iceberg table - non-Iceberg table joins, or with order by queries on a single table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520409425
##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -56,7 +56,15 @@ public DateWritableV2 getPrimitiveWritableObject(Object o) {
@Override
public Object copyObject(Object o) {
- return o == null ? null : new Date((Date) o);
+ if (o == null) {
+ return null;
+ }
+
+ if (o instanceof Date) {
+ return new Date((Date) o);
+ } else {
+ return o;
+ }
Review comment:
On Tez, this is called with LocalDate. On MR we get Date, which is just a wrapper around LocalDate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520963787
##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -56,7 +56,15 @@ public DateWritableV2 getPrimitiveWritableObject(Object o) {
@Override
public Object copyObject(Object o) {
- return o == null ? null : new Date((Date) o);
+ if (o == null) {
+ return null;
+ }
+
+ if (o instanceof Date) {
+ return new Date((Date) o);
+ } else {
+ return o;
+ }
Review comment:
Nevermind, I see that you did.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520963671
##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -56,7 +56,15 @@ public DateWritableV2 getPrimitiveWritableObject(Object o) {
@Override
public Object copyObject(Object o) {
- return o == null ? null : new Date((Date) o);
+ if (o == null) {
+ return null;
+ }
+
+ if (o instanceof Date) {
+ return new Date((Date) o);
+ } else {
+ return o;
+ }
Review comment:
Why not support both representations that are possibly passed in by Hive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520192582
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandlerBaseTest.java
##########
@@ -211,6 +212,11 @@ public void testJoinTables() throws IOException {
Assert.assertArrayEquals(new Object[] {0L, "Alice", 100L, 11.11d}, rows.get(0));
Assert.assertArrayEquals(new Object[] {0L, "Alice", 101L, 22.22d}, rows.get(1));
Assert.assertArrayEquals(new Object[] {1L, "Bob", 102L, 33.33d}, rows.get(2));
+
+ joinTables("decimaltable", "decimal_col", Types.DecimalType.of(3, 1));
+ joinTables("timestamptable", "timestamp_col", Types.TimestampType.withZone());
+ joinTables("binarytable", "binary_col", Types.BinaryType.get());
+ joinTables("datetable", "date_col", Types.DateType.get());
Review comment:
Could you rewrite this as a new test case with a list of types to test?
We consider it a best practice to start new test cases rather than adding to existing, complete cases. Each test method is run independently so you see more of the failures that way. By making longer test methods with more than one case, failures can prevent other tests from even running.
I think it would be fine to use a loop over types in a single new case, since most of the code is the same for these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520191141
##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -56,7 +56,15 @@ public DateWritableV2 getPrimitiveWritableObject(Object o) {
@Override
public Object copyObject(Object o) {
- return o == null ? null : new Date((Date) o);
+ if (o == null) {
+ return null;
+ }
+
+ if (o instanceof Date) {
+ return new Date((Date) o);
+ } else {
+ return o;
+ }
Review comment:
Why would this be called to copy a non-Date object?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on a change in pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740#discussion_r520608445
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandlerBaseTest.java
##########
@@ -211,6 +212,11 @@ public void testJoinTables() throws IOException {
Assert.assertArrayEquals(new Object[] {0L, "Alice", 100L, 11.11d}, rows.get(0));
Assert.assertArrayEquals(new Object[] {0L, "Alice", 101L, 22.22d}, rows.get(1));
Assert.assertArrayEquals(new Object[] {1L, "Bob", 102L, 33.33d}, rows.get(2));
+
+ joinTables("decimaltable", "decimal_col", Types.DecimalType.of(3, 1));
+ joinTables("timestamptable", "timestamp_col", Types.TimestampType.withZone());
+ joinTables("binarytable", "binary_col", Types.BinaryType.get());
+ joinTables("datetable", "date_col", Types.DateType.get());
Review comment:
Thanks @rdblue for the review. I created a new test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1740: Fix casting issues when joining two Iceberg tables together on Tez
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1740:
URL: https://github.com/apache/iceberg/pull/1740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org