You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/04/12 09:45:52 UTC
[GitHub] [hive] hmangla98 opened a new pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
hmangla98 opened a new pull request #2167:
URL: https://github.com/apache/hive/pull/2167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r615736521
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -211,7 +211,7 @@ private void stopWorkers() {
}
private List<AnalyzeWork> processOneTable(TableName fullTableName)
- throws MetaException, NoSuchTxnException, NoSuchObjectException {
+ throws MetaException, NoSuchTxnException, NoSuchObjectException, TException {
Review comment:
is the TException needed?>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hmangla98 commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r613250003
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -140,7 +140,7 @@
public static final String RANGER_CONFIGURATION_RESOURCE_NAME = "ranger-hive-security.xml";
- public static final String TARGET_OF_REPLICATION = "repl.target.for";
+ public static final String TARGET_OF_REPLICATION = ReplConst.TARGET_OF_REPLICATION;
Review comment:
I think repl.target.for was only defined in ReplUtils before this patch. In this path, it is moved to ReplConst class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r613181869
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -226,8 +227,8 @@ private void stopWorkers() {
// on the target again
// 2. updating stats requires a writeId to be created. Hence writeIds on source and target
// can get out of sync when stats are updated. That can cause consistency issues.
- String replTrgtParam = table.getParameters().get(ReplConst.REPL_TARGET_TABLE_PROPERTY);
- if (replTrgtParam != null && !replTrgtParam.isEmpty()) {
+ HiveMetaStoreClient msc = new HiveMetaStoreClient(conf);
+ if (ReplUtils.isTargetOfReplication(msc.getDatabase(table.getCatName(), table.getDbName()))) {
Review comment:
can use rs directly instead of msc
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -140,7 +140,7 @@
public static final String RANGER_CONFIGURATION_RESOURCE_NAME = "ranger-hive-security.xml";
- public static final String TARGET_OF_REPLICATION = "repl.target.for";
+ public static final String TARGET_OF_REPLICATION = ReplConst.TARGET_OF_REPLICATION;
Review comment:
you dont need another constant TARGET_OF_REPLICATION is its already defined in ReplConst class
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
##########
@@ -88,9 +90,13 @@ private static boolean partitionDiscoveryEnabled(Map<String, String> params) {
params.get(DISCOVER_PARTITIONS_TBLPROPERTY).equalsIgnoreCase("true");
}
- private static boolean tblBeingReplicatedInto(Map<String, String> params) {
- return params != null && params.containsKey(ReplConst.REPL_TARGET_TABLE_PROPERTY) &&
- !params.get(ReplConst.REPL_TARGET_TABLE_PROPERTY).trim().isEmpty();
+ public boolean isTargetOfReplication(Database db) {
+ assert (db != null);
+ Map<String, String> m = db.getParameters();
Review comment:
rename to params
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -226,8 +227,8 @@ private void stopWorkers() {
// on the target again
// 2. updating stats requires a writeId to be created. Hence writeIds on source and target
// can get out of sync when stats are updated. That can cause consistency issues.
- String replTrgtParam = table.getParameters().get(ReplConst.REPL_TARGET_TABLE_PROPERTY);
- if (replTrgtParam != null && !replTrgtParam.isEmpty()) {
+ HiveMetaStoreClient msc = new HiveMetaStoreClient(conf);
+ if (ReplUtils.isTargetOfReplication(msc.getDatabase(table.getCatName(), table.getDbName()))) {
Review comment:
in case of table level replication is anything need to be handled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hmangla98 commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r615549345
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -226,8 +227,8 @@ private void stopWorkers() {
// on the target again
// 2. updating stats requires a writeId to be created. Hence writeIds on source and target
// can get out of sync when stats are updated. That can cause consistency issues.
- String replTrgtParam = table.getParameters().get(ReplConst.REPL_TARGET_TABLE_PROPERTY);
- if (replTrgtParam != null && !replTrgtParam.isEmpty()) {
+ HiveMetaStoreClient msc = new HiveMetaStoreClient(conf);
+ if (ReplUtils.isTargetOfReplication(msc.getDatabase(table.getCatName(), table.getDbName()))) {
Review comment:
In case of table level replication also, repl.target.for would be set for that db. Hence, these tasks would be disabled for these tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha merged pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
aasha merged pull request #2167:
URL: https://github.com/apache/hive/pull/2167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hmangla98 commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r613250210
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
##########
@@ -88,9 +90,13 @@ private static boolean partitionDiscoveryEnabled(Map<String, String> params) {
params.get(DISCOVER_PARTITIONS_TBLPROPERTY).equalsIgnoreCase("true");
}
- private static boolean tblBeingReplicatedInto(Map<String, String> params) {
- return params != null && params.containsKey(ReplConst.REPL_TARGET_TABLE_PROPERTY) &&
- !params.get(ReplConst.REPL_TARGET_TABLE_PROPERTY).trim().isEmpty();
+ public boolean isTargetOfReplication(Database db) {
+ assert (db != null);
+ Map<String, String> m = db.getParameters();
Review comment:
Will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hmangla98 commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r615737849
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -211,7 +211,7 @@ private void stopWorkers() {
}
private List<AnalyzeWork> processOneTable(TableName fullTableName)
- throws MetaException, NoSuchTxnException, NoSuchObjectException {
+ throws MetaException, NoSuchTxnException, NoSuchObjectException, TException {
Review comment:
will check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #2167: HIVE-25002: modify condition for target of replication in statsUpdaterThread and PartitionManagementTask
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #2167:
URL: https://github.com/apache/hive/pull/2167#discussion_r615736279
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -140,7 +140,7 @@
public static final String RANGER_CONFIGURATION_RESOURCE_NAME = "ranger-hive-security.xml";
- public static final String TARGET_OF_REPLICATION = "repl.target.for";
+ public static final String TARGET_OF_REPLICATION = ReplConst.TARGET_OF_REPLICATION;
Review comment:
then use the same constant everywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org