You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/08/10 11:49:32 UTC
[GitHub] [hive] klcopp opened a new pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
klcopp opened a new pull request #1384:
URL: https://github.com/apache/hive/pull/1384
This is the second PR for this change.
### Why are the changes needed?
Hive reads insert-only tables truncated by Impala as if the truncation hadn't happened.
### Does this PR introduce _any_ user-facing change?
Yes, getAcidState will not filter out files with names beginning with "_empty"
### How was this patch tested?
Unit test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1384:
URL: https://github.com/apache/hive/pull/1384#discussion_r467938892
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java
##########
@@ -481,6 +482,61 @@ public void testOperationsOnCompletedTxnComponentsForMmTable() throws Exception
verifyDirAndResult(0, true);
}
+ /**
+ * Impala truncates insert-only tables by writing a base directory (like insert overwrite) containing an empty file
+ * named "_empty". Generally in Hive files beginning with an underscore are hidden, so here we make sure that Hive
+ * reads these bases correctly.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testImpalaTruncatedMmTable() throws Exception {
+ FileSystem fs = FileSystem.get(hiveConf);
+ FileStatus[] status;
+
+ Path tblLocation = new Path(TEST_WAREHOUSE_DIR + "/" +
+ (TableExtended.MMTBL).toString().toLowerCase());
+
+ // 1. Insert two rows to an MM table
+ runStatementOnDriver("insert into " + TableExtended.MMTBL + "(a,b) values(1,2)");
+ runStatementOnDriver("insert into " + TableExtended.MMTBL + "(a,b) values(3,4)");
+ status = fs.listStatus(tblLocation, FileUtils.STAGING_DIR_PATH_FILTER);
+ // There should be 2 delta dirs in the location
+ Assert.assertEquals(2, status.length);
+ for (int i = 0; i < status.length; i++) {
+ Assert.assertTrue(status[i].getPath().getName().matches("delta_.*"));
+ }
+
+ // 2. Simulate Impala truncating the table: write a base dir (base_0000003) containing an empty file.
+ // Hive will name the empty file "000000_0"
+ runStatementOnDriver("insert overwrite table " + TableExtended.MMTBL + " select * from "
+ + TableExtended.MMTBL + " where 1=2");
+ status = fs.listStatus(tblLocation, FileUtils.STAGING_DIR_PATH_FILTER);
+ // There should be 2 delta dirs, plus 1 base dir in the location
+ Assert.assertEquals(3, status.length);
+ int baseCount = 0;
+ int deltaCount = 0;
+ for (int i = 0; i < status.length; i++) {
+ String dirName = status[i].getPath().getName();
+ if (dirName.matches("delta_.*")) {
+ deltaCount++;
+ } else {
+ baseCount++;
+ }
+ }
+ Assert.assertEquals(2, deltaCount);
+ Assert.assertEquals(1, baseCount);
+
+ // rename empty file to "_empty"
+ Path basePath = new Path(tblLocation, "base_0000003");
+ Assert.assertTrue("Rename failed",
+ fs.rename(new Path(basePath, "000000_0"), new Path(basePath, "_empty")));
+
+ // 3. Verify query result. Selecting from a truncated table should return nothing.
+ List<String> rs = runStatementOnDriver("select a,b from " + TableExtended.MMTBL + " order by a,b");
+ Assert.assertEquals(Collections.emptyList(), rs);
+ }
Review comment:
Could you extend the test to run a compaction and read back the data, just to make sure compaction also handles this file properly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1384:
URL: https://github.com/apache/hive/pull/1384#discussion_r467937985
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##########
@@ -220,6 +220,10 @@ public boolean accept(Path p) {
if (name.startsWith(OrcAcidVersion.ACID_FORMAT)) {
return true;
}
+ // Don't filter out empty files
+ if (name.startsWith("_empty")) {
Review comment:
extract to a constant, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
Posted by GitBox <gi...@apache.org>.
pvargacl commented on pull request #1384:
URL: https://github.com/apache/hive/pull/1384#issuecomment-671536986
LGTM. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp closed pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
Posted by GitBox <gi...@apache.org>.
klcopp closed pull request #1384:
URL: https://github.com/apache/hive/pull/1384
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp commented on a change in pull request #1384: HIVE-24021: Read insert-only tables truncated by Impala correctly
Posted by GitBox <gi...@apache.org>.
klcopp commented on a change in pull request #1384:
URL: https://github.com/apache/hive/pull/1384#discussion_r467952862
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java
##########
@@ -481,6 +482,61 @@ public void testOperationsOnCompletedTxnComponentsForMmTable() throws Exception
verifyDirAndResult(0, true);
}
+ /**
+ * Impala truncates insert-only tables by writing a base directory (like insert overwrite) containing an empty file
+ * named "_empty". Generally in Hive files beginning with an underscore are hidden, so here we make sure that Hive
+ * reads these bases correctly.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testImpalaTruncatedMmTable() throws Exception {
+ FileSystem fs = FileSystem.get(hiveConf);
+ FileStatus[] status;
+
+ Path tblLocation = new Path(TEST_WAREHOUSE_DIR + "/" +
+ (TableExtended.MMTBL).toString().toLowerCase());
+
+ // 1. Insert two rows to an MM table
+ runStatementOnDriver("insert into " + TableExtended.MMTBL + "(a,b) values(1,2)");
+ runStatementOnDriver("insert into " + TableExtended.MMTBL + "(a,b) values(3,4)");
+ status = fs.listStatus(tblLocation, FileUtils.STAGING_DIR_PATH_FILTER);
+ // There should be 2 delta dirs in the location
+ Assert.assertEquals(2, status.length);
+ for (int i = 0; i < status.length; i++) {
+ Assert.assertTrue(status[i].getPath().getName().matches("delta_.*"));
+ }
+
+ // 2. Simulate Impala truncating the table: write a base dir (base_0000003) containing an empty file.
+ // Hive will name the empty file "000000_0"
+ runStatementOnDriver("insert overwrite table " + TableExtended.MMTBL + " select * from "
+ + TableExtended.MMTBL + " where 1=2");
+ status = fs.listStatus(tblLocation, FileUtils.STAGING_DIR_PATH_FILTER);
+ // There should be 2 delta dirs, plus 1 base dir in the location
+ Assert.assertEquals(3, status.length);
+ int baseCount = 0;
+ int deltaCount = 0;
+ for (int i = 0; i < status.length; i++) {
+ String dirName = status[i].getPath().getName();
+ if (dirName.matches("delta_.*")) {
+ deltaCount++;
+ } else {
+ baseCount++;
+ }
+ }
+ Assert.assertEquals(2, deltaCount);
+ Assert.assertEquals(1, baseCount);
+
+ // rename empty file to "_empty"
+ Path basePath = new Path(tblLocation, "base_0000003");
+ Assert.assertTrue("Rename failed",
+ fs.rename(new Path(basePath, "000000_0"), new Path(basePath, "_empty")));
+
+ // 3. Verify query result. Selecting from a truncated table should return nothing.
+ List<String> rs = runStatementOnDriver("select a,b from " + TableExtended.MMTBL + " order by a,b");
+ Assert.assertEquals(Collections.emptyList(), rs);
+ }
Review comment:
Great idea!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org