You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/20 14:32:12 UTC

[GitHub] [iceberg] lcspinter opened a new pull request #2126: Hive: Add write test for all supported types.

lcspinter opened a new pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561073191



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -88,9 +91,10 @@
 
   private static final List<Type> SUPPORTED_TYPES =
           ImmutableList.of(Types.BooleanType.get(), Types.IntegerType.get(), Types.LongType.get(),
-                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(), Types.TimestampType.withZone(),
+                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(),
+                  Types.TimestampType.withZone(),

Review comment:
       nit: why is this in a new line?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561940969



##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,12 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    Date date = (Date) o;

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561786578



##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,12 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    Date date = (Date) o;

Review comment:
       nit of the nit: new line after if block 😄 

##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDecimalObjectInspector.java
##########
@@ -80,6 +80,12 @@ public Object copyObject(Object o) {
 
   @Override
   public BigDecimal convert(Object o) {
-    return o == null ? null : ((HiveDecimal) o).bigDecimalValue();
+    if (o == null) {
+      return null;
+    }
+
+    BigDecimal result = ((HiveDecimal) o).bigDecimalValue();
+    result = result.setScale(scale());

Review comment:
       Can we add a comment here?

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {

Review comment:
       Can we use Assume here too?

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }

Review comment:
       Am I understanding correctly when I think this conversion is for creating timestamps for which the toString is expected for Hive? Is it ok to set a Timestamp to a field of a Record where the type is Types.TimestampType?
   
   Maybe it would be better to have a `Map<Long, String> forValuesClause`, and move every transformation here (Timestamp, Boolean, and maybe later Fixed/Binary), and later just concatenate it for the query?

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());

Review comment:
       In my recent reviews I have learned that in Iceberg we should not create ArrayLists directly but should use the guava methods for that.
   
   nit: Lists.newArrayListWithCapacity(expected.size())

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,37 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(expected.stream()
+                      // in hive2 every boolean value in apostrophes is translated to true
+                      .map(r -> String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       Would it make sense to move the quotation marks to the `getStringValueForInsert` method as well?
   That would encapsulate the type related stuff entirely and it might be easier to understand the test code.
   
   What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-764858662


   Thank @pvary @marton-bod for the review!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-766857493


   Thanks for the fix @lcspinter and @marton-bod for the review!
   Please do not forget to add different scale/precision tests for the Decimal later.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561941231



##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDecimalObjectInspector.java
##########
@@ -80,6 +80,12 @@ public Object copyObject(Object o) {
 
   @Override
   public BigDecimal convert(Object o) {
-    return o == null ? null : ((HiveDecimal) o).bigDecimalValue();
+    if (o == null) {
+      return null;
+    }
+
+    BigDecimal result = ((HiveDecimal) o).bigDecimalValue();
+    result = result.setScale(scale());

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] marton-bod commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561108864



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(String.join(",", records.stream().map(r ->
+                  String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),
+                      r.get(1))).collect(Collectors.toList())));

Review comment:
       I think you can use `Collectors.joining` here and then you wouldn't need the extra `String.join` step

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(String.join(",", records.stream().map(r ->
+                  String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       Can you please add a small clarifying comment as to why we need the special case for the boolean type?

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();

Review comment:
       very nit: you can set the initialCapacity to be `expected.size()`

##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,8 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    return o == null ? null : LocalDate.of(((Date) o).getYear(), ((Date) o).getMonth(), ((Date) o).getDay());

Review comment:
       nit: can we cast to Date only once and reuse that? might help slightly with readability too

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -88,9 +91,10 @@
 
   private static final List<Type> SUPPORTED_TYPES =
           ImmutableList.of(Types.BooleanType.get(), Types.IntegerType.get(), Types.LongType.get(),
-                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(), Types.TimestampType.withZone(),
+                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(),
+                  Types.TimestampType.withZone(),
                   Types.TimestampType.withoutZone(), Types.StringType.get(), Types.BinaryType.get(),
-                  Types.DecimalType.of(3, 1), Types.UUIDType.get(), Types.FixedType.ofLength(5),
+                  Types.DecimalType.of(3, 1), Types.UUIDType.get(), Types.FixedType.ofLength(11),

Review comment:
       Can we write different decimal scales/precisions? If so, can we add a second one here, that has a different scale to test?

##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimeObjectInspector.java
##########
@@ -49,8 +50,11 @@ public Text getPrimitiveWritableObject(Object o) {
   }
 
   @Override
-  public Object convert(Object o) {
-    return o == null ? null : o.toString();
+  public LocalTime convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    return LocalTime.parse(o.toString());

Review comment:
       Can we rely on the string representation of `o` to be parsed correctly here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561943057



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {

Review comment:
       I would not use Assume in this case, because when the condition fails, it would exit the test method, skipping the rest of the loop. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561796626



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }

Review comment:
       Am I understanding correctly when I think this conversion is for creating timestamps for which the toString is expected for Hive? Is it ok to set a Timestamp to a field of a Record where the type is Types.TimestampType?
   
   Maybe it would be better to have a `Map<Long, String> forValuesClause`, and move every transformation here (Timestamp, Boolean, and maybe later Fixed/Binary), and later just concatenate it for the query?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-767016624


   Thanks for reviewing, @pvary! Glad to see you can commit now!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary merged pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
pvary merged pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561786958



##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDecimalObjectInspector.java
##########
@@ -80,6 +80,12 @@ public Object copyObject(Object o) {
 
   @Override
   public BigDecimal convert(Object o) {
-    return o == null ? null : ((HiveDecimal) o).bigDecimalValue();
+    if (o == null) {
+      return null;
+    }
+
+    BigDecimal result = ((HiveDecimal) o).bigDecimalValue();
+    result = result.setScale(scale());

Review comment:
       Can we add a comment here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-767016624


   Thanks for reviewing, @pvary! Glad to see you can commit now!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-766857493


   Thanks for the fix @lcspinter and @marton-bod for the review!
   Please do not forget to add different scale/precision tests for the Decimal later.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561951605



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,37 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(expected.stream()
+                      // in hive2 every boolean value in apostrophes is translated to true
+                      .map(r -> String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       Would it make sense to move the quotation marks to the `getStringValueForInsert` method as well?
   That would encapsulate the type related stuff entirely and it might be easier to understand the test code.
   
   What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561818263



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());

Review comment:
       In my recent reviews I have learned that in Iceberg we should not create ArrayLists directly but should use the guava methods for that.
   
   nit: Lists.newArrayListWithCapacity(expected.size())




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561940969



##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,12 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    Date date = (Date) o;

Review comment:
       fixed

##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDecimalObjectInspector.java
##########
@@ -80,6 +80,12 @@ public Object copyObject(Object o) {
 
   @Override
   public BigDecimal convert(Object o) {
-    return o == null ? null : ((HiveDecimal) o).bigDecimalValue();
+    if (o == null) {
+      return null;
+    }
+
+    BigDecimal result = ((HiveDecimal) o).bigDecimalValue();
+    result = result.setScale(scale());

Review comment:
       Done

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {

Review comment:
       I would not use Assume in this case, because when the condition fails, it would exit the test method, skipping the rest of the loop. 

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }

Review comment:
       I created a getStringValueForInsert which gives back to correct string representation for certain types.

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());

Review comment:
       Noted.

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,37 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(expected.stream()
+                      // in hive2 every boolean value in apostrophes is translated to true
+                      .map(r -> String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       It's a good idea. I will change the code respectively. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561944125



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }

Review comment:
       I created a getStringValueForInsert which gives back to correct string representation for certain types.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary merged pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
pvary merged pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561973648



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,37 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(expected.stream()
+                      // in hive2 every boolean value in apostrophes is translated to true
+                      .map(r -> String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       It's a good idea. I will change the code respectively. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561106918



##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimeObjectInspector.java
##########
@@ -49,8 +50,11 @@ public Text getPrimitiveWritableObject(Object o) {
   }
 
   @Override
-  public Object convert(Object o) {
-    return o == null ? null : o.toString();
+  public LocalTime convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    return LocalTime.parse(o.toString());

Review comment:
       Fixed.

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -88,9 +91,10 @@
 
   private static final List<Type> SUPPORTED_TYPES =
           ImmutableList.of(Types.BooleanType.get(), Types.IntegerType.get(), Types.LongType.get(),
-                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(), Types.TimestampType.withZone(),
+                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(),
+                  Types.TimestampType.withZone(),

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561072166



##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimeObjectInspector.java
##########
@@ -49,8 +50,11 @@ public Text getPrimitiveWritableObject(Object o) {
   }
 
   @Override
-  public Object convert(Object o) {
-    return o == null ? null : o.toString();
+  public LocalTime convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    return LocalTime.parse(o.toString());

Review comment:
       nit: Could you make this a one liner as all of the others?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561151289



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -88,9 +91,10 @@
 
   private static final List<Type> SUPPORTED_TYPES =
           ImmutableList.of(Types.BooleanType.get(), Types.IntegerType.get(), Types.LongType.get(),
-                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(), Types.TimestampType.withZone(),
+                  Types.FloatType.get(), Types.DoubleType.get(), Types.DateType.get(),
+                  Types.TimestampType.withZone(),
                   Types.TimestampType.withoutZone(), Types.StringType.get(), Types.BinaryType.get(),
-                  Types.DecimalType.of(3, 1), Types.UUIDType.get(), Types.FixedType.ofLength(5),
+                  Types.DecimalType.of(3, 1), Types.UUIDType.get(), Types.FixedType.ofLength(11),

Review comment:
       I would rather not add a new entry to this list. If we would like to test the many flavours of the Decimal type, I guess we should do it in a dedicated method/class. 

##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,8 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    return o == null ? null : LocalDate.of(((Date) o).getYear(), ((Date) o).getMonth(), ((Date) o).getDay());

Review comment:
       Fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561788393



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {

Review comment:
       Can we use Assume here too?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561786578



##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergDateObjectInspectorHive3.java
##########
@@ -69,4 +69,12 @@ public Object copyObject(Object o) {
     }
   }
 
+  @Override
+  public LocalDate convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    Date date = (Date) o;

Review comment:
       nit of the nit: new line after if block 😄 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561152557



##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimeObjectInspector.java
##########
@@ -49,8 +50,11 @@ public Text getPrimitiveWritableObject(Object o) {
   }
 
   @Override
-  public Object convert(Object o) {
-    return o == null ? null : o.toString();
+  public LocalTime convert(Object o) {
+    if (o == null) {
+      return null;
+    }
+    return LocalTime.parse(o.toString());

Review comment:
       Sorry, I was wrong about the type of `o`. It is not Text, but String.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-763646722


   @marton-bod @pvary If you have time, could you please review this change? Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561147058



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(String.join(",", records.stream().map(r ->
+                  String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),
+                      r.get(1))).collect(Collectors.toList())));

Review comment:
       Good point! Done.

##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();
+      if (type == Types.TimestampType.withoutZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.valueOf((LocalDateTime) r.get(1))));
+      } else if (type == Types.TimestampType.withZone()) {
+        expected.forEach(r -> records.add(r.copy()));
+        records.forEach(r -> r.set(1, Timestamp.from(((OffsetDateTime) r.get(1)).toInstant())));
+      } else {
+        records.addAll(expected);
+      }
+
+      Table table = testTables.createTable(shell, tableName, schema, fileFormat, ImmutableList.of());
+      StringBuilder query = new StringBuilder("INSERT INTO ").append(tableName).append(" VALUES")
+              .append(String.join(",", records.stream().map(r ->
+                  String.format(type == Types.BooleanType.get() ? "(%s,%s)" : "(%s,'%s')", r.get(0),

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on pull request #2126: Hive: Fix writing of Date, Decimal, Time and UUID types.

Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-767374079


   > Thanks for reviewing, @pvary! Glad to see you can commit now!
   
   Thanks for helping through the first steps! 😄 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#issuecomment-764858662


   Thank @pvary @marton-bod for the review!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561944354



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +299,46 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(5))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>(expected.size());

Review comment:
       Noted.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] lcspinter commented on a change in pull request #2126: Hive: Add write test for all supported types.

Posted by GitBox <gi...@apache.org>.
lcspinter commented on a change in pull request #2126:
URL: https://github.com/apache/iceberg/pull/2126#discussion_r561151494



##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -296,6 +300,45 @@ public void testInsert() throws IOException {
     HiveIcebergTestUtils.validateData(table, new ArrayList<>(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS), 0);
   }
 
+  @Test
+  public void testInsertSupportedTypes() throws IOException {
+    Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+    for (int i = 0; i < SUPPORTED_TYPES.size(); i++) {
+      Type type = SUPPORTED_TYPES.get(i);
+      // TODO: remove this filter when issue #1881 is resolved
+      if (type == Types.UUIDType.get() && fileFormat == FileFormat.PARQUET) {
+        continue;
+      }
+      // TODO: remove this filter when we figure out how we could test binary types
+      if (type.equals(Types.BinaryType.get()) || type.equals(Types.FixedType.ofLength(11))) {
+        continue;
+      }
+      String tableName = type.typeId().toString().toLowerCase() + "_table_" + i;
+      String columnName = type.typeId().toString().toLowerCase() + "_column";
+
+      Schema schema = new Schema(required(1, "id", Types.LongType.get()), required(2, columnName, type));
+      List<Record> expected = TestHelper.generateRandomRecords(schema, 5, 0L);
+      List<Record> records = new ArrayList<>();

Review comment:
       Fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org