You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mdedetrich (via GitHub)" <gi...@apache.org> on 2023/05/08 15:14:19 UTC

[GitHub] [kafka] mdedetrich opened a new pull request, #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

mdedetrich opened a new pull request, #13689:
URL: https://github.com/apache/kafka/pull/13689

   Current implementation of TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed will immediately throw an exception if TopicBasedRemoteLogMetadataManager is not initialized. Rather we should instead try and wait for TopicBasedRemoteLogMetadataManager to initialize with a timeout as this is expected behaviour of users.
   
   The solution to this is to use a typical `CountDownLatch` as is done elsewhere within Kafka 
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1187745268


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -76,6 +78,11 @@ public class TopicBasedRemoteLogMetadataManager implements RemoteLogMetadataMana
     private final boolean startConsumerThread;
 
     private Thread initializationThread;
+
+    private static final long SHUTDOWN_TIMEOUT_SECONDS = 60L;
+
+    private CountDownLatch initializeLatch;

Review Comment:
   Thanks for the tip! Definitely can do this, I was just mimicking the style in the rest of the Kafka codebase. Will see what the reviewers respond with.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1194911490


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Thanks for the response. While I understand the problem regarding holding up the threads because the number of calling threads may be limited unless I am missing something in practice this shouldn't be a big concern because in most cases you would either get a single block or non at all (i.e. if the methods are called after `configure` is initialized then `initializeLatch.await` won't block anything and there won't be any context switching either).
   
   If we also think about this problem in another context, i.e. from the perspective of Kafka users/creators of the `RemoteLogMetadataManager` with custom plugins this behaviour is not exactly satisfactory if we for example take into account how other plugin based ecosystems like connect work. Such behaviour is going to create lots of workarounds/hacks (even in our case of using Tiered Storage, with this change `TieredStorageTestHarness` fails to run properly when running in different Kafka versions because unless you put random sleeps because it has no idea when `RemoteLogMetadataManager` has been initialized since all fields/data for initialization is package private). So how long something takes to initialize is a block box aside from `TopicBasedRemoteLogMetadataManagerHarness` which can peek into package private. In summary I don't really think throwing an exception immediately is overall better net behaviour than blocking until initialization from a user perspective.
   
   If blocking the thread is such a blocker (pun intended!) would it be more amicable to adjust the interface so its possible for plugin authors (and others) to figure out when any `RemoteLogMetadataManager` has been initialized? If not blocking the thread is so critical then to me its appropriate then to expose the intialization so users can actually handle this case comprehensively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1193008914


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Right, that was the reason for failing the requests immediately. We do not want to hold on to those threads until the RLMM initialises. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #13689:
URL: https://github.com/apache/kafka/pull/13689#issuecomment-1713350729

   I am closing this PR since https://github.com/apache/kafka/pull/14127 solves this issue in another way, i.e. throws a specific exception which forces the client to retry.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1187572164


##########
storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerHarness.java:
##########
@@ -104,27 +103,10 @@ public void onPartitionLeadershipChanges(Set<TopicIdPartition> leaderPartitions,
         log.debug("TopicBasedRemoteLogMetadataManager configs after adding overridden properties: {}", configs);
 
         topicBasedRemoteLogMetadataManager.configure(configs);
-        try {
-            waitUntilInitialized(60_000);
-        } catch (TimeoutException e) {
-            throw new KafkaException(e);
-        }
-
         topicBasedRemoteLogMetadataManager.onPartitionLeadershipChanges(topicIdPartitions, Collections.emptySet());
     }
 
     // Visible for testing.
-    public void waitUntilInitialized(long waitTimeMs) throws TimeoutException {

Review Comment:
   This was removed because the entire point of this PR is to make explicit waiting redundant. The fact that this was only possible in a test because of package private is the underlying reason behind this change, in other words when running `TopicBasedRemoteLogMetadataManager` normally within a broker it was too easy to call methods before the`TopicBasedRemoteLogMetadataManager` was finished initializing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #13689:
URL: https://github.com/apache/kafka/pull/13689#issuecomment-1538539482

   @satishd @junrao Pinging you because you done previous PR's regarding Tiered Storage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] junrao commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1192604258


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Hmm, if we wait here, it could tie up the caller thread for 60 secs. Some of the caller threads may be very limited. So, it's probably better to fail immediately instead of block?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #13689:
URL: https://github.com/apache/kafka/pull/13689#issuecomment-1546701743

   Sorry, missed this PR notification earlier. Replied a comment [here](https://github.com/apache/kafka/pull/13689#discussion_r1193008914). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich closed pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich closed pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing
URL: https://github.com/apache/kafka/pull/13689


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1194911490


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Thanks for the response. While I understand the problem regarding holding up the threads because the number of calling threads may be limited unless I am missing something in practice this shouldn't be a big concern because in most cases you would either get a single block or non at all (i.e. if the methods are called after `configure` is initialized then `initializeLatch.await` won't block anything and there won't be any context switching either).
   
   If we also think about this problem in another context, i.e. from the perspective of Kafka users/creators of the `RemoteLogMetadataManager` with custom plugins this behaviour is not exactly satisfactory if we for example take into account how other plugin based ecosystems like connect work. Such behaviour is going to create lots of workarounds/hacks (even in our case of using Tiered Storage, with this change `TieredStorageTestHarness` fails to run properly when running in different Kafka versions because unless you put random sleeps because it has no idea when `RemoteLogMetadataManager` has been initialized since all fields/data for initialization is package private). So how long something takes to initialize is a block box aside from TopicBasedRemoteLogMetadataManagerHarness which can peek into package private.
   
   If blocking the thread is such a blocker (pun intended!) would it be more amicable to adjust the interface so its possible for plugin authors (and others) to figure out when any `RemoteLogMetadataManager` has been initialized? If not blocking the thread is so critical then to me its appropriate then to expose the intialization so users can actually handle this case comprehensively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] reta commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "reta (via GitHub)" <gi...@apache.org>.
reta commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1187596223


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -76,6 +78,11 @@ public class TopicBasedRemoteLogMetadataManager implements RemoteLogMetadataMana
     private final boolean startConsumerThread;
 
     private Thread initializationThread;
+
+    private static final long SHUTDOWN_TIMEOUT_SECONDS = 60L;
+
+    private CountDownLatch initializeLatch;

Review Comment:
   Just a suggestion, but I think you could simplify tracking the initialization flow by folding  `initializeLatch` & `initialized` into a `CompletableFuture` (which could also handle the case when close is called while the initialization is still in progress).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1194911490


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Thanks for the response. While I understand the problem regarding holding up the threads because the number of calling threads may be limited unless I am missing something in practice this shouldn't be a big concern because in most cases you would either get a single block or non at all (i.e. if the methods are called after `configure` is initialized then `initializeLatch.await` won't block anything and there won't be any context switching either).
   
   If we also think about this problem in another context, i.e. from the perspective of Kafka users/creators of the `RemoteLogMetadataManager` with custom plugins this behaviour is not exactly satisfactory if we for example take into account how other plugin based ecosystems like connect work. Such behaviour is going to create lots of workarounds/hacks (even in our case of using Tiered Storage, with this change `TieredStorageTestHarness` fails to run properly when running in different Kafka versions because unless you put random sleeps because it has no idea when `RemoteLogMetadataManager` has been initialized since all fields/data for initialization is package private). So how long something takes to initialize is a block box aside from `TopicBasedRemoteLogMetadataManagerHarness` which can peek into package private.
   
   If blocking the thread is such a blocker (pun intended!) would it be more amicable to adjust the interface so its possible for plugin authors (and others) to figure out when any `RemoteLogMetadataManager` has been initialized? If not blocking the thread is so critical then to me its appropriate then to expose the intialization so users can actually handle this case comprehensively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mdedetrich commented on a diff in pull request #13689: KAFKA-14975: Wait for TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed to initialize rather than throwing

Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #13689:
URL: https://github.com/apache/kafka/pull/13689#discussion_r1200746117


##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java:
##########
@@ -489,14 +498,28 @@ public boolean isInitialized() {
         return initialized.get();
     }
 
-    private void ensureInitializedAndNotClosed() {
+    private void ensureInitializedAndNotClosed() throws RemoteStorageException {
         if (initializationFailed) {
             // If initialization is failed, shutdown the broker.
             throw new FatalExitError();
         }
-        if (closing.get() || !initialized.get()) {
-            throw new IllegalStateException("This instance is in invalid state, initialized: " + initialized +
-                                                    " close: " + closing);
+
+        if (!initialized.get()) {
+            log.info("Waiting for TopicBasedRemoteLogMetadataManager to initialize");
+            try {
+                if (!initializeLatch.await(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {

Review Comment:
   Another alternative option would be to make the `configure` method blocking, this means we don't have to deal with the issue of the other methods with the `RemoteLogMetadataManager` blocking scarce caller threads however I suspect that there is an underlying reason why `configure` is not blocking in the first place?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org