You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/12 15:02:53 UTC
[GitHub] [hive] pvary opened a new pull request #1101: HIVE-23683: Add queue time to compaction
pvary opened a new pull request #1101:
URL: https://github.com/apache/hive/pull/1101
Added new QUEUE_TIME to the compaction tables.
Created upgrade scripts
Initialized the data with current timestamp
Made sure that the data is kept during the lifecycle
Updated HMS API to return the data
Created tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440465947
##########
File path: ql/src/test/queries/clientpositive/sysdb.q
##########
@@ -1,5 +1,6 @@
--! qt:dataset:src,part,srcbucket:ONLY
--! qt:sysdb
+--! qt:replace:/(initiated\s+NULL\s+NULL\s+)[0-9]*(\s+NULL)/$1#Masked#$2/
Review comment:
Done
##########
File path: ql/src/test/queries/clientpositive/dbtxnmgr_showlocks.q
##########
@@ -1,3 +1,4 @@
+ --! qt:replace:/(initiated\s+---\s+---\s+)[0-9]*(\s+---)/$1#Masked#$2/
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp commented on a change in pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
klcopp commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440640809
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
##########
@@ -81,6 +81,8 @@ private void writeHeader(DataOutputStream os) throws IOException {
os.write(Utilities.tabCode);
os.writeBytes("Worker");
os.write(Utilities.tabCode);
+ os.writeBytes("Queue Time");
Review comment:
Sounds good to me!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary merged pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #1101:
URL: https://github.com/apache/hive/pull/1101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp commented on a change in pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
klcopp commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440343846
##########
File path: ql/src/test/queries/clientpositive/dbtxnmgr_showlocks.q
##########
@@ -1,3 +1,4 @@
+ --! qt:replace:/(initiated\s+---\s+---\s+)[0-9]*(\s+---)/$1#Masked#$2/
Review comment:
Nit: maybe add a comment here about what's going on, to save someone time down the road
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
##########
@@ -81,6 +81,8 @@ private void writeHeader(DataOutputStream os) throws IOException {
os.write(Utilities.tabCode);
os.writeBytes("Worker");
os.write(Utilities.tabCode);
+ os.writeBytes("Queue Time");
Review comment:
"Queue time" sounds like "time spent in queue." Something like "Entered queue" sounds more like a timestamp to me.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
##########
@@ -116,6 +117,7 @@ public String toString() {
"partName:" + partName + "," +
"state:" + state + "," +
"type:" + type + "," +
+ "queueTime:" + queueTime + "," +
Review comment:
(Opportunity: "start" is missing from this method)
##########
File path: ql/src/test/queries/clientpositive/sysdb.q
##########
@@ -1,5 +1,6 @@
--! qt:dataset:src,part,srcbucket:ONLY
--! qt:sysdb
+--! qt:replace:/(initiated\s+NULL\s+NULL\s+)[0-9]*(\s+NULL)/$1#Masked#$2/
Review comment:
Nit: maybe add a comment here about what's going on, to save someone time down the road
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1101:
URL: https://github.com/apache/hive/pull/1101#discussion_r440467302
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
##########
@@ -81,6 +81,8 @@ private void writeHeader(DataOutputStream os) throws IOException {
os.write(Utilities.tabCode);
os.writeBytes("Worker");
os.write(Utilities.tabCode);
+ os.writeBytes("Queue Time");
Review comment:
How about enqueue time? Short, easy to understand...
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
##########
@@ -116,6 +117,7 @@ public String toString() {
"partName:" + partName + "," +
"state:" + state + "," +
"type:" + type + "," +
+ "queueTime:" + queueTime + "," +
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on pull request #1101: HIVE-23683: Add queue time to compaction
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #1101:
URL: https://github.com/apache/hive/pull/1101#issuecomment-644675940
Thanks @klcopp and @laszlopinter86 for the reviews!
> One more: have you considered displaying a human-readable timestamp (maybe specifying UTC) in show compactions output?
I decided against it, because every timestamp now is printed in epoch time in the show compactions output, and also I think this might break some user scripts if we change this behavior. Maybe in another jira with its' own config would be beneficial to have this.
> Also I recommend running the metastore upgrade tests if you haven't yet.
That was big help in identifying issues (happily not in this script) - See: https://issues.apache.org/jira/browse/HIVE-23697
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org