You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/04/27 18:02:40 UTC
[GitHub] [hive] pvary opened a new pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
pvary opened a new pull request #2228:
URL: https://github.com/apache/hive/pull/2228
### What changes were proposed in this pull request?
Wrap the Record before calculating partitions
### Why are the changes needed?
Timestamp/Date needs values converted to long/int respectively to calculate partitions
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added new unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622242175
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergTestUtils.java
##########
@@ -107,6 +110,8 @@
PrimitiveObjectInspectorFactory.writableStringObjectInspector
));
+ private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd hh:mm:ss");
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622249484
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).bucket("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testTruncateTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).truncate("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
Review comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622227503
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).bucket("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testTruncateTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).truncate("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
Review comment:
I would like to check that the data is correct with the Java API and read through SQL is also important
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622183265
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).bucket("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testTruncateTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).truncate("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
Review comment:
Never mind, I guess the new `validateData` reads the data by SQL query, not java API. But then, do we want to keep run both methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622181150
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).bucket("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testTruncateTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).truncate("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
Review comment:
What is the difference between the two `validateData` methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622227816
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergTestUtils.java
##########
@@ -107,6 +110,8 @@
PrimitiveObjectInspectorFactory.writableStringObjectInspector
));
+ private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd hh:mm:ss");
Review comment:
Not anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622240406
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
Review comment:
renamed.
Good catch! Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622242549
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).bucket("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testTruncateTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.StringType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).truncate("part_field", 2).build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, "Part1")
+ .add(2L, "Part2")
+ .add(3L, "Art3")
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
Review comment:
Maybe name one of them `validateDataWithSQL`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622242014
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
Review comment:
Renamed.
Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary merged pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #2228:
URL: https://github.com/apache/hive/pull/2228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622185131
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
Review comment:
nit: I guess here and in the truncate transform, this is not a schema with a date anymore
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -759,6 +763,108 @@ public void testMultilevelIdentityPartitionedWrite() throws IOException {
HiveIcebergTestUtils.validateData(table, records, 0);
}
+ @Test
+ public void testYearTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.DateType.get()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).year("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDate.of(2020, 1, 21))
+ .add(2L, LocalDate.of(2020, 1, 22))
+ .add(3L, LocalDate.of(2019, 1, 21))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testMonthTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).month("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(1)))
+ .add(2L, OffsetDateTime.of(2017, 11, 22, 11, 30, 7, 0, ZoneOffset.ofHours(2)))
+ .add(3L, OffsetDateTime.of(2017, 11, 23, 11, 30, 7, 0, ZoneOffset.ofHours(3)))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testDayTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).day("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testHourTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
+ optional(1, "id", Types.LongType.get()),
+ optional(2, "part_field", Types.TimestampType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schemaWithDate).hour("part_field").build();
+ List<Record> records = TestHelper.RecordsBuilder.newInstance(schemaWithDate)
+ .add(1L, LocalDateTime.of(2019, 2, 22, 9, 44, 54))
+ .add(2L, LocalDateTime.of(2019, 2, 22, 10, 44, 54))
+ .add(3L, LocalDateTime.of(2019, 2, 23, 9, 44, 54))
+ .build();
+ Table table = testTables.createTable(shell, "part_test", schemaWithDate, spec, fileFormat, records);
+ HiveIcebergTestUtils.validateData(table, records, 0);
+
+ HiveIcebergTestUtils.validateData(shell, "part_test", records, "id");
+ }
+
+ @Test
+ public void testBucketTransform() throws IOException {
+ Schema schemaWithDate = new Schema(
Review comment:
nit: I guess here and in the truncate transform, this is not a schema with a date anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #2228: HIVE-25062 Iceberg: Fix date partition transform insert issue
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2228:
URL: https://github.com/apache/hive/pull/2228#discussion_r622181750
##########
File path: iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergTestUtils.java
##########
@@ -107,6 +110,8 @@
PrimitiveObjectInspectorFactory.writableStringObjectInspector
));
+ private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd hh:mm:ss");
Review comment:
Is this used anywhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org