You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/08/24 13:47:30 UTC
[GitHub] [hive] ArkoSharma opened a new pull request #1422: HIVE-24064 : Disable materialized view replication
ArkoSharma opened a new pull request #1422:
URL: https://github.com/apache/hive/pull/1422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1422: HIVE-24064 : Disable materialized view replication
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1422:
URL: https://github.com/apache/hive/pull/1422#discussion_r479922802
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -542,6 +549,23 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
if (ev.getEventId() <= resumeFrom) {
continue;
}
+
+ //disable materialized-view replication if not configured
+ if(!isMaterializedViewsReplEnabled()){
+ String tblName = ev.getTableName();
+ if(tblName != null) {
+ try {
+ HiveWrapper.Tuple<Table> tabletuple = new HiveWrapper(hiveDb, dbName).table(tblName, conf);
Review comment:
you can use hiveDb directly instead of using HiveWrapper
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #1422: HIVE-24064 : Disable materialized view replication
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1422:
URL: https://github.com/apache/hive/pull/1422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1422: HIVE-24064 : Disable materialized view replication
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1422:
URL: https://github.com/apache/hive/pull/1422#discussion_r477138389
##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -2609,6 +2609,109 @@ public void testViewsReplication() throws IOException {
verifyIfTableNotExist(replDbName, "virtual_view", metaStoreClientMirror);
}
+ @Test
+ public void testMaterializedViewsReplication() throws Exception {
+ boolean verifySetup_tmp = verifySetupSteps;
+ verifySetupSteps = true;
+ String testName = "materializedviewsreplication";
+ String testName2 = testName + "2";
+ String dbName = createDB(testName, driver);
+ String dbName2 = createDB(testName2, driver); //for creating multi-db materialized view
+ String replDbName = dbName + "_dupe";
+
+ run("CREATE TABLE " + dbName + ".unptned(a string) STORED AS TEXTFILE", driver);
+ run("CREATE TABLE " + dbName2 + ".unptned(a string) STORED AS TEXTFILE", driver);
+ run("CREATE TABLE " + dbName + ".ptned(a string) partitioned by (b int) STORED AS TEXTFILE", driver);
+
+ String[] unptn_data = new String[]{ "eleven", "twelve" };
+ String[] ptn_data_1 = new String[]{ "thirteen", "fourteen", "fifteen"};
+ String[] ptn_data_2 = new String[]{ "fifteen", "sixteen", "seventeen"};
+ String[] empty = new String[]{};
+
+ String unptn_locn = new Path(TEST_PATH, testName + "_unptn").toUri().getPath();
+ String ptn_locn_1 = new Path(TEST_PATH, testName + "_ptn1").toUri().getPath();
+ String ptn_locn_2 = new Path(TEST_PATH, testName + "_ptn2").toUri().getPath();
+
+ createTestDataFile(unptn_locn, unptn_data);
+ createTestDataFile(ptn_locn_1, ptn_data_1);
+ createTestDataFile(ptn_locn_2, ptn_data_2);
+
+ verifySetup("SELECT a from " + dbName + ".ptned", empty, driver);
+ verifySetup("SELECT * from " + dbName + ".unptned", empty, driver);
+ verifySetup("SELECT * from " + dbName2 + ".unptned", empty, driver);
+
+ run("LOAD DATA LOCAL INPATH '" + unptn_locn + "' OVERWRITE INTO TABLE " + dbName + ".unptned", driver);
+ run("LOAD DATA LOCAL INPATH '" + unptn_locn + "' OVERWRITE INTO TABLE " + dbName2 + ".unptned", driver);
+ verifySetup("SELECT * from " + dbName + ".unptned", unptn_data, driver);
+ verifySetup("SELECT * from " + dbName2 + ".unptned", unptn_data, driver);
+
+ run("LOAD DATA LOCAL INPATH '" + ptn_locn_1 + "' OVERWRITE INTO TABLE " + dbName + ".ptned PARTITION(b=1)", driver);
+ verifySetup("SELECT a from " + dbName + ".ptned WHERE b=1", ptn_data_1, driver);
+ run("LOAD DATA LOCAL INPATH '" + ptn_locn_2 + "' OVERWRITE INTO TABLE " + dbName + ".ptned PARTITION(b=2)", driver);
+ verifySetup("SELECT a from " + dbName + ".ptned WHERE b=2", ptn_data_2, driver);
+
+
+ run("CREATE MATERIALIZED VIEW " + dbName + ".mat_view_boot disable rewrite stored as textfile AS SELECT a FROM " + dbName + ".ptned where b=1", driver);
+ verifySetup("SELECT a from " + dbName + ".mat_view_boot", ptn_data_1, driver);
+
+ run("CREATE MATERIALIZED VIEW " + dbName + ".mat_view_boot2 disable rewrite stored as textfile AS SELECT t1.a FROM " + dbName + ".unptned as t1 join " + dbName2 + ".unptned as t2 on t1.a = t2.a", driver);
+ verifySetup("SELECT a from " + dbName + ".mat_view_boot2", unptn_data, driver);
+
+ Tuple bootstrapDump = bootstrapLoadAndVerify(dbName, replDbName);
+
+ verifyRun("SELECT * from " + replDbName + ".unptned", unptn_data, driverMirror);
+ verifyRun("SELECT a from " + replDbName + ".ptned where b=1", ptn_data_1, driverMirror);
+
+ //verify source MVs are not on replica
+ verifyIfTableNotExist(replDbName, "mat_view_boot", metaStoreClientMirror);
+ verifyIfTableNotExist(replDbName, "mat_view_boot2", metaStoreClientMirror);
+
+ //test alter materialized view with rename
+ run("ALTER TABLE " + dbName + ".mat_view_boot RENAME TO " + dbName + ".mat_view_rename", driver);
+
+ //verify rename, i.e. new MV exists and old MV does not exist
+ verifyIfTableNotExist(dbName, "mat_view_boot", metaStoreClient);
+ verifyIfTableExist(dbName, "mat_view_rename", metaStoreClient);
+ //verifySetup("SELECT a from " + dbName + ".mat_view_rename", ptn_data_1, driver);
Review comment:
check why this is failing
##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -2609,6 +2609,109 @@ public void testViewsReplication() throws IOException {
verifyIfTableNotExist(replDbName, "virtual_view", metaStoreClientMirror);
}
+ @Test
+ public void testMaterializedViewsReplication() throws Exception {
Review comment:
test for MV on top of transactional tables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1422: HIVE-24064 : Disable materialized view replication
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1422:
URL: https://github.com/apache/hive/pull/1422#discussion_r476198220
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -542,6 +549,24 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
if (ev.getEventId() <= resumeFrom) {
continue;
}
+
+ //disable materialized-view replication if configured
+ String tblName = null;
+ tblName = ev.getTableName();
+ if(tblName != null) {
+ try {
+ Table table = null;
+ HiveWrapper.Tuple<Table> TableTuple = new HiveWrapper(hiveDb, dbName).table(tblName, conf);
Review comment:
rename the variable. should start with small letter
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -540,6 +540,8 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
REPL_DUMP_METADATA_ONLY("hive.repl.dump.metadata.only", false,
"Indicates whether replication dump only metadata information or data + metadata. \n"
+ "This config makes hive.repl.include.external.tables config ineffective."),
+ REPL_MATERIALIZED_VIEWS_ENABLED("hive.repl.materialized.views.enabled", false,
Review comment:
rename to REPL_INCLUDE_MATERIALIZED_VIEWS to match the other config naming
##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -136,7 +136,7 @@
protected static final Logger LOG = LoggerFactory.getLogger(TestReplicationScenarios.class);
private ArrayList<String> lastResults;
- private final boolean VERIFY_SETUP_STEPS = false;
+ private boolean VERIFY_SETUP_STEPS = false;
Review comment:
can be removed. the value is overwritten inside the tests so not a constant
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -542,6 +549,24 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
if (ev.getEventId() <= resumeFrom) {
continue;
}
+
+ //disable materialized-view replication if configured
+ String tblName = null;
+ tblName = ev.getTableName();
+ if(tblName != null) {
+ try {
+ Table table = null;
+ HiveWrapper.Tuple<Table> TableTuple = new HiveWrapper(hiveDb, dbName).table(tblName, conf);
+ table = TableTuple != null ? TableTuple.object : null;
Review comment:
can be initialised in the same statement
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -542,6 +549,24 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
if (ev.getEventId() <= resumeFrom) {
continue;
}
+
+ //disable materialized-view replication if configured
+ String tblName = null;
+ tblName = ev.getTableName();
Review comment:
can be initialised in the same statement
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##########
@@ -542,6 +549,24 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
if (ev.getEventId() <= resumeFrom) {
continue;
}
+
+ //disable materialized-view replication if configured
+ String tblName = null;
+ tblName = ev.getTableName();
+ if(tblName != null) {
+ try {
+ Table table = null;
+ HiveWrapper.Tuple<Table> TableTuple = new HiveWrapper(hiveDb, dbName).table(tblName, conf);
+ table = TableTuple != null ? TableTuple.object : null;
+ if (TableTuple != null && TableType.MATERIALIZED_VIEW.equals(TableTuple.object.getTableType()) && !isMaterializedViewsReplEnabled()) {
Review comment:
isMaterializedViewsReplEnabled check can be done at the top. no processing is needed if its enabled
##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -2518,11 +2518,16 @@ public void testRenamePartitionedTableAcrossDatabases() throws IOException {
@Test
public void testViewsReplication() throws IOException {
+ boolean verify_setup_tmp = VERIFY_SETUP_STEPS;
Review comment:
you can add a new test just for MV.
Create MV with same DB and diff DB
Test after bootstrap, replica shouldn't have MV.
Create MV with same DB and diff DB
Test after incremental, replica shouldn't have MV.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org