You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/11/03 14:02:32 UTC
[GitHub] [hive] pvargacl opened a new pull request #1647: HIVE-24329: Add Notification event for compaction commit
pvargacl opened a new pull request #1647:
URL: https://github.com/apache/hive/pull/1647
### What changes were proposed in this pull request?
Add new notification event type for commit compaction.
### Why are the changes needed?
The new event can be used by file metadata cache implementations to invalidate the cache content
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r524248762
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -1162,6 +1172,55 @@ protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, Tx
TxnDbUtil.getEpochFn(dbProduct) + " WHERE \"CQ_TXN_ID\" = " + txnid);
}
}
+
+ protected CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException {
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r524244952
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -1409,6 +1407,23 @@ public void commitTxn(CommitTxnRequest rqst)
}
}
+ /**
+ * Create Notifiaction Events on txn commit
+ * @param dbConn DatabaseConnection
+ * @param txnid committed txn
+ * @param txnType transaction type
+ * @throws MetaException ex
+ */
+ protected void createCommitNotificationEvent(Connection dbConn, long txnid, Optional<TxnType> txnType)
+ throws MetaException, SQLException {
+ if (transactionalListeners != null) {
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.COMMIT_TXN, new CommitTxnEvent(txnid, txnType.get()), dbConn, sqlGenerator);
+ }
+ }
+
+ protected abstract CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException;
Review comment:
It was a leftover, deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r524232515
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CommitCompactionEvent.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.events;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hive.metastore.IHMSHandler;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.TxnType;
+import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
+
+/**
+ * CommitCompactionEvent
+ * Event generated for compaction commit operation
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Stable
+public class CommitCompactionEvent extends ListenerEvent {
+
+ private final Long txnId;
+ private final Long compactionId;
+ private final CompactionType type;
+ private final String dbname;
Review comment:
The filemetadata cache will have to invalidate the cache of a given partition based on the event payload. It most likely will need to use the dbName as part of the key, to identify the right table. When it matters the event contains this info (see AllocWriteIdEvent)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521381364
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/CommitCompactionMessage.java
##########
@@ -0,0 +1,47 @@
+/* * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.messaging;
+
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.WriteEventInfo;
+
+import java.util.List;
+
+/**
+ * Message sent when a compaction commit is done.
+ */
+public abstract class CommitCompactionMessage extends EventMessage {
+
+ protected CommitCompactionMessage() {
+ super(EventType.COMMIT_COMPACTION);
+ }
+
+ public abstract Long getTxnId();
+
+ public abstract Long getCompactionId();
+
+ public abstract CompactionType getType();
+
+ public abstract String getDbname();
Review comment:
Same as above, do we actually need db name here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521386568
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -1409,6 +1407,23 @@ public void commitTxn(CommitTxnRequest rqst)
}
}
+ /**
+ * Create Notifiaction Events on txn commit
+ * @param dbConn DatabaseConnection
+ * @param txnid committed txn
+ * @param txnType transaction type
+ * @throws MetaException ex
+ */
+ protected void createCommitNotificationEvent(Connection dbConn, long txnid, Optional<TxnType> txnType)
+ throws MetaException, SQLException {
+ if (transactionalListeners != null) {
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.COMMIT_TXN, new CommitTxnEvent(txnid, txnType.get()), dbConn, sqlGenerator);
+ }
+ }
+
+ protected abstract CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException;
Review comment:
Why do we have compaction related stuff in TxnHandler? It's an abstract class already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521393727
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -278,7 +279,7 @@
"SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
- private List<TransactionalMetaStoreEventListener> transactionalListeners;
+ protected List<TransactionalMetaStoreEventListener> transactionalListeners;
Review comment:
can you wrap method with if check, so you wouldn't need to change access modifier
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r525992636
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONCommitCompactionMessage.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.messaging.json;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.collect.Lists;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.WriteEventInfo;
+import org.apache.hadoop.hive.metastore.messaging.CommitCompactionMessage;
+import org.apache.hadoop.hive.metastore.messaging.CommitTxnMessage;
+import org.apache.hadoop.hive.metastore.messaging.MessageBuilder;
+
+import java.util.List;
+
+/**
+ * JSON implementation of CommitCompactionMessage
+ */
+public class JSONCommitCompactionMessage extends CommitCompactionMessage {
+
+ @JsonProperty
+ private Long txnid;
+
+ @JsonProperty
+ private Long timestamp;
+
+ @JsonProperty
+ private String server;
+
+ @JsonProperty
+ private String servicePrincipal;
+
+ @JsonProperty
+ private Long compactionId;
+
+ @JsonProperty
+ private CompactionType type;
+
+ @JsonProperty
+ private String dbname;
+
+ @JsonProperty
+ private String tableName;
+
+ @JsonProperty
+ private String partName;
+
+ /**
+ * Default constructor, needed for Jackson.
+ */
+ public JSONCommitCompactionMessage() {
+ }
+
+ public JSONCommitCompactionMessage(String server, String servicePrincipal, Long timestamp, Long txnid,
Review comment:
passed the event as parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ merged pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ merged pull request #1647:
URL: https://github.com/apache/hive/pull/1647
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r524248581
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##########
@@ -8373,6 +8374,11 @@ public void commit_txn(CommitTxnRequest rqst) throws TException {
if (listeners != null && !listeners.isEmpty()) {
MetaStoreListenerNotifier.notifyEvent(listeners, EventType.COMMIT_TXN,
new CommitTxnEvent(rqst.getTxnid(), this));
+ CompactionInfo compactionInfo = getTxnHandler().getCompactionByTxnId(rqst.getTxnid());
Review comment:
This does a little bit different thing. In TxnHandler we notify the transactional eventListeners using the same jdbc connection from the txnhandler. The event will be committed together with the "hive transaction commit". This on the other hand notifies the "normal" event listeners. The file metadata cache strictly needs only the first one, but every other transactional event is fired as normal event also, so I don't want to break this pattern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521424854
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##########
@@ -8373,6 +8374,11 @@ public void commit_txn(CommitTxnRequest rqst) throws TException {
if (listeners != null && !listeners.isEmpty()) {
MetaStoreListenerNotifier.notifyEvent(listeners, EventType.COMMIT_TXN,
new CommitTxnEvent(rqst.getTxnid(), this));
+ CompactionInfo compactionInfo = getTxnHandler().getCompactionByTxnId(rqst.getTxnid());
Review comment:
I think this overlap with code in CompactionTxnHandler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521383280
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -1162,6 +1172,55 @@ protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, Tx
TxnDbUtil.getEpochFn(dbProduct) + " WHERE \"CQ_TXN_ID\" = " + txnid);
}
}
+
+ protected CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException {
Review comment:
why it is protected?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvargacl commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
pvargacl commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r527538920
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -1176,6 +1186,55 @@ protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, Tx
TxnDbUtil.getEpochFn(dbProduct) + " WHERE \"CQ_TXN_ID\" = " + txnid);
}
}
+
+ private CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException {
+ CompactionInfo info = null;
Review comment:
good catch, fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521383280
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -1162,6 +1172,55 @@ protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, Tx
TxnDbUtil.getEpochFn(dbProduct) + " WHERE \"CQ_TXN_ID\" = " + txnid);
}
}
+
+ protected CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException {
Review comment:
why is it protected?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521382402
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONCommitCompactionMessage.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.messaging.json;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.collect.Lists;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.WriteEventInfo;
+import org.apache.hadoop.hive.metastore.messaging.CommitCompactionMessage;
+import org.apache.hadoop.hive.metastore.messaging.CommitTxnMessage;
+import org.apache.hadoop.hive.metastore.messaging.MessageBuilder;
+
+import java.util.List;
+
+/**
+ * JSON implementation of CommitCompactionMessage
+ */
+public class JSONCommitCompactionMessage extends CommitCompactionMessage {
+
+ @JsonProperty
+ private Long txnid;
+
+ @JsonProperty
+ private Long timestamp;
+
+ @JsonProperty
+ private String server;
+
+ @JsonProperty
+ private String servicePrincipal;
+
+ @JsonProperty
+ private Long compactionId;
+
+ @JsonProperty
+ private CompactionType type;
+
+ @JsonProperty
+ private String dbname;
+
+ @JsonProperty
+ private String tableName;
+
+ @JsonProperty
+ private String partName;
+
+ /**
+ * Default constructor, needed for Jackson.
+ */
+ public JSONCommitCompactionMessage() {
+ }
+
+ public JSONCommitCompactionMessage(String server, String servicePrincipal, Long timestamp, Long txnid,
Review comment:
Maybe use builder to avoid constructor with huge number of params?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521379969
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CommitCompactionEvent.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.events;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.hive.metastore.IHMSHandler;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.TxnType;
+import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
+
+/**
+ * CommitCompactionEvent
+ * Event generated for compaction commit operation
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Stable
+public class CommitCompactionEvent extends ListenerEvent {
+
+ private final Long txnId;
+ private final Long compactionId;
+ private final CompactionType type;
+ private final String dbname;
Review comment:
I haven't seen that other events contained db information. Is it required here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r527057925
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -1176,6 +1186,55 @@ protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, Tx
TxnDbUtil.getEpochFn(dbProduct) + " WHERE \"CQ_TXN_ID\" = " + txnid);
}
}
+
+ private CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException {
+ CompactionInfo info = null;
Review comment:
Wouldn't it be better if we wrap this with Optional?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521424854
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##########
@@ -8373,6 +8374,11 @@ public void commit_txn(CommitTxnRequest rqst) throws TException {
if (listeners != null && !listeners.isEmpty()) {
MetaStoreListenerNotifier.notifyEvent(listeners, EventType.COMMIT_TXN,
new CommitTxnEvent(rqst.getTxnid(), this));
+ CompactionInfo compactionInfo = getTxnHandler().getCompactionByTxnId(rqst.getTxnid());
Review comment:
I think, this has overlap with createCommitNotificationEvent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #1647: HIVE-24329: Add Notification event for compaction commit
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #1647:
URL: https://github.com/apache/hive/pull/1647#discussion_r521386568
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -1409,6 +1407,23 @@ public void commitTxn(CommitTxnRequest rqst)
}
}
+ /**
+ * Create Notifiaction Events on txn commit
+ * @param dbConn DatabaseConnection
+ * @param txnid committed txn
+ * @param txnType transaction type
+ * @throws MetaException ex
+ */
+ protected void createCommitNotificationEvent(Connection dbConn, long txnid, Optional<TxnType> txnType)
+ throws MetaException, SQLException {
+ if (transactionalListeners != null) {
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.COMMIT_TXN, new CommitTxnEvent(txnid, txnType.get()), dbConn, sqlGenerator);
+ }
+ }
+
+ protected abstract CompactionInfo getCompactionByTxnId(Connection dbConn, long txnid) throws SQLException, MetaException;
Review comment:
why do we have compaction related stuff in TxnHandler?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org