You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "fvaleri (via GitHub)" <gi...@apache.org> on 2023/05/21 08:58:17 UTC

[GitHub] [kafka] fvaleri opened a new pull request, #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

fvaleri opened a new pull request, #13738:
URL: https://github.com/apache/kafka/pull/13738

   When running kafka-metadata-quorum script to get the quorum replication status, the LastFetchTimestamp and LastCaughtUpTimestamp output is not human-readable.
   
   I will be convenient to add an optional flag (-hr, --human-readable) to enable a human-readable format showing the delay in ms (i.e. 366 ms ago).
   
   This dealy is computed as (now - timestamp), where they are both represented as Unix time (UTC based).
   
   ```sh
   bin/kafka-metadata-quorum.sh --bootstrap-server :9092 -hr describe --replication
   NodeId  LogEndOffset    Lag LastFetchTimestamp  LastCaughtUpTimestamp   Status     
   2       541             0   3 ms ago            3 ms ago                Leader     
   3       541             0   366 ms ago          366 ms ago              Follower   
   4       541             0   366 ms ago          366 ms ago              Follower
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon merged pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon merged PR #13738:
URL: https://github.com/apache/kafka/pull/13738


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #13738:
URL: https://github.com/apache/kafka/pull/13738#issuecomment-1569434420

   @fvaleri testRelativeTimeMs seems to fail regularly. There is an example here: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13782/1/tests. Could you take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on PR #13738:
URL: https://github.com/apache/kafka/pull/13738#issuecomment-1562817801

   @showuon all your comments should be addressed with the latest commit. 
   
   I removed the `-hr` abbreviation and updated the KIP because argparse4j supports them by default. For example you can do the following:
   
   ```sh
   $ bin/kafka-metadata-quorum.sh --bootstrap-server :9092 describe --re --hu
   NodeId	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status  	
   2     	2032        	0  	3 ms ago          	3 ms ago             	Leader  	
   3     	2032        	0  	207 ms ago        	207 ms ago           	Follower	
   4     	2032        	0  	207 ms ago        	207 ms ago           	Follower
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1205438624


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {
+        Instant instant = Instant.ofEpochMilli(timestampMs);

Review Comment:
   Renamed to lastTimestamp.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {
+        Instant instant = Instant.ofEpochMilli(timestampMs);
+        Instant now = Instant.now();
+        if (!(instant.isAfter(Instant.EPOCH) && instant.isBefore(now))) {
+            throw new KafkaException("Invalid timestamp, possible drift in system clock");

Review Comment:
   Sure, I reworked that code to include more details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1207971633


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -191,13 +191,16 @@ private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leade
         }).collect(Collectors.toList());
     }
 
-    private static long durationMs(long timestampMs) {
-        Instant instant = Instant.ofEpochMilli(timestampMs);
+    /* test */ static long delayMs(long timestampMs, String desc) {

Review Comment:
   Sure, I like relativeTimeMs. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1200248662


##########
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandTest.java:
##########
@@ -172,4 +173,34 @@ public void testDescribeQuorumInZkMode() {
         );
 
     }
+
+    @ClusterTests({
+        @ClusterTest(clusterType = Type.CO_KRAFT, brokers = 1, controllers = 1),
+    })
+    public void testHumanReadableTimestamps() {
+        String replicationOutput0 = ToolsTestUtils.captureStandardOut(() ->

Review Comment:
   please add a test to assert that `-hr` fails gracefully if used without `--replication`



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -136,17 +143,21 @@ private static void addDescribeParser(Subparsers subparsers) {
             .addArgument("--replication")
             .help("Detailed information about the status of replication")
             .action(Arguments.storeTrue());
+        replicationArgs
+            .addArgument("-hr", "--human-readable")

Review Comment:
   This doesn't work without `--replication`, right? In which case, this should probably be a subParser of replication (similar to how we handle subparsing for --describe)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1208376541


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -96,14 +100,18 @@ static void execute(String... args) throws Exception {
                 if (namespace.getBoolean("status") && namespace.getBoolean("replication")) {
                     throw new TerseException("Only one of --status or --replication should be specified with describe sub-command");
                 } else if (namespace.getBoolean("replication")) {
-                    handleDescribeReplication(admin);
+                    boolean humanReadable = Optional.of(namespace.getBoolean("human_readable")).orElse(false);
+                    handleDescribeReplication(admin, humanReadable);
                 } else if (namespace.getBoolean("status")) {
+                    if (namespace.getBoolean("human_readable")) {
+                        throw new TerseException("The option -hr/--human-readable is only supported along with --replication");

Review Comment:
   Since we don't have `-hr` option, this message should be updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on PR #13738:
URL: https://github.com/apache/kafka/pull/13738#issuecomment-1556784078

   @machi1990 ok.
   
   @divijvaidya thanks for the great review. I went too fast with this PR, now it should be in a better shape.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1199800198


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +163,28 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean readable) {
+        return infos.map(info -> {
+            final double nowMs = System.currentTimeMillis();

Review Comment:
   Consider using `Time.SYSTEM`



##########
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandTest.java:
##########
@@ -172,4 +173,22 @@ public void testDescribeQuorumInZkMode() {
         );
 
     }
+
+    @ClusterTests({
+        @ClusterTest(clusterType = Type.CO_KRAFT, brokers = 1, controllers = 1),
+    })
+    public void testHumanReadableTimestamps() {

Review Comment:
   this test doesn't test the business logic that we introduced in the code, i.e. the part about ensuring X in "X ms ago" is what is desired is missing.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +163,28 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean readable) {
+        return infos.map(info -> {
+            final double nowMs = System.currentTimeMillis();
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                valueOf(readable ? format("%.0f ms ago", nowMs - info.lastFetchTimestamp().getAsLong())

Review Comment:
   please expect skew/drift in system clock. For such situations we should handle the case when difference is negative.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -79,6 +82,10 @@ static void execute(String... args) throws Exception {
             .addArgument("--command-config")
             .type(Arguments.fileType())
             .help("Property file containing configs to be passed to Admin Client.");
+        parser

Review Comment:
   this should probably go inside `addDescribeParser` since we want to parse for this only when --describe is called



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -92,11 +99,12 @@ static void execute(String... args) throws Exception {
             props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, namespace.getString("bootstrap_server"));
             admin = Admin.create(props);
 
+            boolean readable = Optional.of(namespace.getBoolean("human_readable")).orElse(false);

Review Comment:
   we only use this for --describe and replication, please move inside `handleDescribeReplication()`



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +163,28 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean readable) {
+        return infos.map(info -> {
+            final double nowMs = System.currentTimeMillis();

Review Comment:
   `System.currentTimeMillis()` returns a long. https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#currentTimeMillis
   
   Using long return type here will makes subtraction easier later on in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1205438962


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {

Review Comment:
   Done.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -131,22 +140,27 @@ private static void addDescribeParser(Subparsers subparsers) {
             .addArgument("--status")
             .help("A short summary of the quorum status and the other provides detailed information about the status of replication.")
             .action(Arguments.storeTrue());
+
         ArgumentGroup replicationArgs = describeParser.addArgumentGroup("Replication");
         replicationArgs
             .addArgument("--replication")
             .help("Detailed information about the status of replication")
             .action(Arguments.storeTrue());
+        replicationArgs
+            .addArgument("-hr", "--human-readable")
+            .help("Print human-readable timestamps")

Review Comment:
   What about "Human-readable output"? It's short, clear and allows reuse. In the future and even in other tools, we could use this flag to make some fields readable or filter out less important details that make the output hard to read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1207886510


##########
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandErrorTest.java:
##########
@@ -45,4 +51,14 @@ public void testDescribeOptions() {
                 MetadataQuorumCommand.mainNoExit("--bootstrap-server", "localhost:9092", "describe", "--status", "--replication")));
     }
 
+    @Test
+    public void testDelayMs() {
+        long nowMs = Instant.now().toEpochMilli();
+        assertTrue(MetadataQuorumCommand.delayMs(nowMs, "test") >= 0);
+        long invalidEpochMs = Instant.EPOCH.minus(1, ChronoUnit.DAYS).toEpochMilli();
+        assertThrows(KafkaException.class, () -> MetadataQuorumCommand.delayMs(invalidEpochMs, "test"));
+        long futureTimestampMs = Instant.now().plus(1, ChronoUnit.DAYS).toEpochMilli();
+        assertThrows(KafkaException.class, () -> MetadataQuorumCommand.delayMs(futureTimestampMs, "test"));
+    }

Review Comment:
   Thanks for adding it. Looks good.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -191,13 +191,16 @@ private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leade
         }).collect(Collectors.toList());
     }
 
-    private static long durationMs(long timestampMs) {
-        Instant instant = Instant.ofEpochMilli(timestampMs);
+    /* test */ static long delayMs(long timestampMs, String desc) {

Review Comment:
   nit: In Kafka, we usually put the test comment on one line above, i.e.
   ```
   // visible for testing
   static long delayMs(long timestampMs, String desc) {
   ```



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -191,13 +191,16 @@ private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leade
         }).collect(Collectors.toList());
     }
 
-    private static long durationMs(long timestampMs) {
-        Instant instant = Instant.ofEpochMilli(timestampMs);
+    /* test */ static long delayMs(long timestampMs, String desc) {
+        Instant lastTimestamp = Instant.ofEpochMilli(timestampMs);

Review Comment:
   `lastTimestamp` LGTM. Thanks.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -191,13 +191,16 @@ private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leade
         }).collect(Collectors.toList());
     }
 
-    private static long durationMs(long timestampMs) {
-        Instant instant = Instant.ofEpochMilli(timestampMs);
+    /* test */ static long delayMs(long timestampMs, String desc) {

Review Comment:
   Also, the method `delayMs` doesn't look right to me. I did a google search, it looks like the `relative time` is the correct name for it. Check [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/RelativeTimeFormat/format) and [here](https://momentjs.com/docs/#/displaying/fromnow/). So, maybe `relativeTimeMs`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1208444130


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -96,14 +100,18 @@ static void execute(String... args) throws Exception {
                 if (namespace.getBoolean("status") && namespace.getBoolean("replication")) {
                     throw new TerseException("Only one of --status or --replication should be specified with describe sub-command");
                 } else if (namespace.getBoolean("replication")) {
-                    handleDescribeReplication(admin);
+                    boolean humanReadable = Optional.of(namespace.getBoolean("human_readable")).orElse(false);
+                    handleDescribeReplication(admin, humanReadable);
                 } else if (namespace.getBoolean("status")) {
+                    if (namespace.getBoolean("human_readable")) {
+                        throw new TerseException("The option -hr/--human-readable is only supported along with --replication");

Review Comment:
   Thanks for catching this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1205206409


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {
+        Instant instant = Instant.ofEpochMilli(timestampMs);
+        Instant now = Instant.now();
+        if (!(instant.isAfter(Instant.EPOCH) && instant.isBefore(now))) {
+            throw new KafkaException("Invalid timestamp, possible drift in system clock");

Review Comment:
   Could we add `now` and `timestampMs` into the error message?



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {

Review Comment:
   Could we add a test for this method? It looks like we only test the output format, not test the output duration is correct.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +169,35 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean humanReadable) {
+        return infos.map(info -> {
+            String lastFetchTimestamp = !info.lastFetchTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastFetchTimestamp().getAsLong())) :
+                    valueOf(info.lastFetchTimestamp().getAsLong());
+            String lastCaughtUpTimestamp = !info.lastCaughtUpTimestamp().isPresent() ? "-1" :
+                humanReadable ? format("%d ms ago", durationMs(info.lastCaughtUpTimestamp().getAsLong())) :
+                    valueOf(info.lastCaughtUpTimestamp().getAsLong());
+            return Stream.of(
                 info.replicaId(),
                 info.logEndOffset(),
                 leader.logEndOffset() - info.logEndOffset(),
-                info.lastFetchTimestamp().orElse(-1),
-                info.lastCaughtUpTimestamp().orElse(-1),
+                lastFetchTimestamp,
+                lastCaughtUpTimestamp,
                 status
-            ).map(r -> r.toString()).collect(Collectors.toList())
-        ).collect(Collectors.toList());
+            ).map(r -> r.toString()).collect(Collectors.toList());
+        }).collect(Collectors.toList());
+    }
+
+    private static long durationMs(long timestampMs) {
+        Instant instant = Instant.ofEpochMilli(timestampMs);

Review Comment:
   This variable name is hard to understand, I don't know which name is better, maybe `timestampBefore`?



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -131,22 +140,27 @@ private static void addDescribeParser(Subparsers subparsers) {
             .addArgument("--status")
             .help("A short summary of the quorum status and the other provides detailed information about the status of replication.")
             .action(Arguments.storeTrue());
+
         ArgumentGroup replicationArgs = describeParser.addArgumentGroup("Replication");
         replicationArgs
             .addArgument("--replication")
             .help("Detailed information about the status of replication")
             .action(Arguments.storeTrue());
+        replicationArgs
+            .addArgument("-hr", "--human-readable")
+            .help("Print human-readable timestamps")

Review Comment:
   Since we're not printing timestamp with `-hr` option, maybe change to:
   nit: Print human-readable timestamps duration. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] machi1990 commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "machi1990 (via GitHub)" <gi...@apache.org>.
machi1990 commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1199737177


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -155,17 +163,28 @@ private static void handleDescribeReplication(Admin admin) throws ExecutionExcep
         );
     }
 
-    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader, Stream<QuorumInfo.ReplicaState> infos, String status) {
-        return infos.map(info ->
-            Stream.of(
+    private static List<List<String>> quorumInfoToRows(QuorumInfo.ReplicaState leader,
+                                                       Stream<QuorumInfo.ReplicaState> infos,
+                                                       String status,
+                                                       boolean readable) {

Review Comment:
   ```suggestion
                                                          boolean humanReadable) {
   ```
   nit: suggestion to change `readable` to `humanReadable`, if accepted, it has to be applied throughout: the caller and the callee of this method.



##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -138,15 +146,15 @@ private static void addDescribeParser(Subparsers subparsers) {
             .action(Arguments.storeTrue());
     }
 
-    private static void handleDescribeReplication(Admin admin) throws ExecutionException, InterruptedException {
+    private static void handleDescribeReplication(Admin admin, boolean readable) throws ExecutionException, InterruptedException {

Review Comment:
   same comment as https://github.com/apache/kafka/pull/13738/files#r1199737177



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1200336739


##########
tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java:
##########
@@ -136,17 +143,21 @@ private static void addDescribeParser(Subparsers subparsers) {
             .addArgument("--replication")
             .help("Detailed information about the status of replication")
             .action(Arguments.storeTrue());
+        replicationArgs
+            .addArgument("-hr", "--human-readable")

Review Comment:
   It doesn't work with --status, but I see what you mean. We should raise an error, rather than silently ignore the --hr option.



##########
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandTest.java:
##########
@@ -172,4 +173,34 @@ public void testDescribeQuorumInZkMode() {
         );
 
     }
+
+    @ClusterTests({
+        @ClusterTest(clusterType = Type.CO_KRAFT, brokers = 1, controllers = 1),
+    })
+    public void testHumanReadableTimestamps() {
+        String replicationOutput0 = ToolsTestUtils.captureStandardOut(() ->

Review Comment:
   Yes, this is related to the above. Will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on a diff in pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #13738:
URL: https://github.com/apache/kafka/pull/13738#discussion_r1200336786


##########
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandTest.java:
##########
@@ -172,4 +173,34 @@ public void testDescribeQuorumInZkMode() {
         );
 
     }
+
+    @ClusterTests({
+        @ClusterTest(clusterType = Type.CO_KRAFT, brokers = 1, controllers = 1),
+    })
+    public void testHumanReadableTimestamps() {
+        String replicationOutput0 = ToolsTestUtils.captureStandardOut(() ->

Review Comment:
   Yes, this is related to the addArgument comment. Will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13738:
URL: https://github.com/apache/kafka/pull/13738#issuecomment-1566387485

   Failed tests are unrelated:
   ```
       Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testReplication()
       Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector
       Build / JDK 11 and Scala 2.13 / integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor()
       Build / JDK 8 and Scala 2.12 / org.apache.kafka.tools.MetadataQuorumCommandErrorTest.testRelativeTimeMs()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] fvaleri commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on PR #13738:
URL: https://github.com/apache/kafka/pull/13738#issuecomment-1565265034

   Failing tests are unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org