You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/02/17 12:57:04 UTC
[GitHub] [iceberg] marton-bod opened a new pull request #2246: Hive: skip projection pushdown for output tables
marton-bod opened a new pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246
When inserting data into an Iceberg table from a Hive table, where the source table has the same columns as the target table, the query fails. This is due to the fact that after the SELECT query completes on the source table, the read column values from the source table will be stuck in the config under `hive.io.file.readcolumn.names`, and will be read by the Serde when writing out the data into the target table and therefore will create the wrong objectinspector. To solve this issue, we should skip column projection pushdown for output tables that we write to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577595509
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -395,6 +395,25 @@ public void testInsertFromSelectWithProjection() throws IOException {
HiveIcebergTestUtils.validateData(table, expected, 0);
}
+ @Test
+ public void testInsertFromHiveTableWithSameColumnNames() throws IOException {
+ Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+
+ shell.executeStatement(
+ "CREATE TABLE hive_customers(customer_id bigint, first_name string) PARTITIONED BY (last_name string)");
+ shell.executeStatement(
+ "INSERT INTO hive_customers VALUES (0, 'Alice', 'Brown'), (1, 'Bob', 'Green'), (2, 'Trudy', 'Pink')");
Review comment:
Can we use `testTables.createTable(TestHiveShell shell, String tableName, Schema schema, PartitionSpec spec, FileFormat fileFormat, List<Record> records)` here?
Or it would be even better if we create a version of this method where we do not need to execute a Hive query to insert the data. We do not test the Hive `INSERT INTO` here and adding records directly to the table is faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577590629
##########
File path: mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java
##########
@@ -88,18 +88,24 @@ public void initialize(@Nullable Configuration configuration, Properties serDePr
}
}
- configuration.setBoolean(InputFormatConfig.CASE_SENSITIVE, false);
- String[] selectedColumns = ColumnProjectionUtils.getReadColumnNames(configuration);
- // When same table is joined multiple times, it is possible some selected columns are duplicated,
- // in this case wrong recordStructField position leads wrong value or ArrayIndexOutOfBoundException
- String[] distinctSelectedColumns = Arrays.stream(selectedColumns).distinct().toArray(String[]::new);
- Schema projectedSchema = distinctSelectedColumns.length > 0 ?
- tableSchema.caseInsensitiveSelect(distinctSelectedColumns) : tableSchema;
- // the input split mapper handles does not belong to this table
- // it is necessary to ensure projectedSchema equals to tableSchema,
- // or we cannot find selectOperator's column from inspector
- if (projectedSchema.columns().size() != distinctSelectedColumns.length) {
+ Schema projectedSchema;
+ if (serDeProperties.get(HiveIcebergStorageHandler.WRITE_KEY) != null) {
+ // when writing out data, we should not do projection pushdown
projectedSchema = tableSchema;
+ } else {
+ configuration.setBoolean(InputFormatConfig.CASE_SENSITIVE, false);
Review comment:
I think we should leave this config setting outside the if statement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
marton-bod commented on pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#issuecomment-781285453
Thanks for the merge @pvary!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577627150
##########
File path: hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java
##########
@@ -56,7 +56,7 @@
public class TestHiveMetastore {
private static final String DEFAULT_DATABASE_NAME = "default";
- private static final int DEFAULT_POOL_SIZE = 5;
+ private static final int DEFAULT_POOL_SIZE = 10;
Review comment:
We don't need to increase this anymore, so changed it back to 5.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary merged pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577626975
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -395,6 +395,25 @@ public void testInsertFromSelectWithProjection() throws IOException {
HiveIcebergTestUtils.validateData(table, expected, 0);
}
+ @Test
+ public void testInsertFromHiveTableWithSameColumnNames() throws IOException {
+ Assume.assumeTrue("Tez write is not implemented yet", executionEngine.equals("mr"));
+
+ shell.executeStatement(
+ "CREATE TABLE hive_customers(customer_id bigint, first_name string) PARTITIONED BY (last_name string)");
+ shell.executeStatement(
+ "INSERT INTO hive_customers VALUES (0, 'Alice', 'Brown'), (1, 'Bob', 'Green'), (2, 'Trudy', 'Pink')");
Review comment:
I switched to using Iceberg tables to make it simpler. Originally I used Hive tables in the test because with Hive, `SELECT *` does not put the partition column into `hive.io.file.readcolumn.names` which leads to this unexpected failure, while this behaviour regarding partition columns is not present in Iceberg. However, we can reproduce the issue by using an Iceberg source table as well (with the same columns as the target table), just not selecting all of its columns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577627150
##########
File path: hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java
##########
@@ -56,7 +56,7 @@
public class TestHiveMetastore {
private static final String DEFAULT_DATABASE_NAME = "default";
- private static final int DEFAULT_POOL_SIZE = 5;
+ private static final int DEFAULT_POOL_SIZE = 10;
Review comment:
Changed it back to 5.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #2246: Hive: skip projection pushdown for output tables
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2246:
URL: https://github.com/apache/iceberg/pull/2246#discussion_r577596118
##########
File path: hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java
##########
@@ -56,7 +56,7 @@
public class TestHiveMetastore {
private static final String DEFAULT_DATABASE_NAME = "default";
- private static final int DEFAULT_POOL_SIZE = 5;
+ private static final int DEFAULT_POOL_SIZE = 10;
Review comment:
Do we need this change if we do not use insert to add data to the source table in the test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org