You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Denovo1998 (via GitHub)" <gi...@apache.org> on 2026/10/14 13:40:55 UTC

[PR] [fix][broker] Clean up orphan schema ledgers after write failures [pulsar]

Denovo1998 opened a new pull request, #26902:
URL: https://github.com/apache/pulsar/pull/26902

   Related to #26705.
   
   ### Motivation
   
   When `BookkeeperSchemaStorage` creates a ledger successfully but adding the schema entry fails, the ledger is never referenced by a schema locator. The write-failure path does not close or delete that ledger, so repeated failed registrations can leave orphan schema ledgers that normal schema deletion cannot discover. This affects both initial schema creation and compatible schema updates.
   
   ### Modifications
   
   - Make a best-effort close attempt after a failed write.
   - Wait for ledger close before returning the position used to publish the schema locator.
   - Route write and close failures through the existing best-effort ledger deletion helper, wait for the deletion callback, and preserve the original failure when cleanup also fails.
   - Add deterministic regression tests through the public schema registration path, covering initial creation, updates, unchanged locators, successful retries, and delayed or failed cleanup.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   Seven new fault-injection cases use the existing broker, metadata store, and `PulsarMockBookKeeper` fixtures. The initial-creation and update regressions both fail on the unpatched code because an extra schema ledger remains. Controlled futures verify that the request waits for close and deletion callbacks and retains the original write error.
   
   Local validation on 2026-10-12, based on `be41491e7d9e`, passed all 28 scoped tests and `quickCheck`. Validation has not been rerun on the newly synchronized master.
   
   ```shell
   ./gradlew :pulsar-broker:test \
     --tests org.apache.pulsar.broker.service.schema.SchemaServiceTest \
     --tests org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorageTest \
     --tests org.apache.pulsar.schema.SchemaTest.testConcurrentCreateSchemaNoOrphanLedger \
     --tests org.apache.pulsar.schema.SchemaTest.testConcurrentUpdateSchemaNoOrphanLedger \
     -PtestRetryCount=0 -PtestFailFast=false \
     -PtestMaxParallelForks=1 --max-workers=2
   
   ./gradlew quickCheck --max-workers=2
   ```
   
   ### Does this pull request potentially affect one of the following parts:
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Clean up orphan schema ledgers after write failures [pulsar]

Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on code in PR #26902:
URL: https://github.com/apache/pulsar/pull/26902#discussion_r4283254472


##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/SchemaServiceTest.java:
##########
@@ -215,6 +223,129 @@ public void writeReadBackDeleteSchemaEntry() throws Exception {
         assertNull(schemaRegistryService.getSchema(schemaId1).get());
     }
 
+    @DataProvider
+    public Object[][] schemaWriteFailureCases() {
+        return new Object[][] {{false}, {true}};
+    }
+
+    @Test(dataProvider = "schemaWriteFailureCases")
+    public void testSchemaWriteFailureDeletesOrphanLedger(boolean existingSchema) throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        if (existingSchema) {
+            putSchema(schemaId, schemaData1, version(0));
+        }
+        String path = "/schemas/" + schemaId;
+        byte[] locatorBefore = pulsar.getLocalMetadataStore().get(path).get(10, TimeUnit.SECONDS)
+                .map(result -> result.getValue()).orElse(null);
+
+        bookKeeper.addEntryFailAfter(0, BKException.Code.WriteException);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData2, SchemaCompatibilityStrategy.FULL);
+
+        assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                .hasRootCauseInstanceOf(SchemaException.class)
+                .rootCause().hasMessageContaining("operation=Failed to add entry");
+        byte[] locatorAfter = pulsar.getLocalMetadataStore().get(path).get(10, TimeUnit.SECONDS)
+                .map(result -> result.getValue()).orElse(null);
+        assertThat(locatorAfter).isEqualTo(locatorBefore);
+        assertThat(schemaLedgers(bookKeeper, schemaId))
+                .as("A failed schema write must not leave an unreferenced ledger")
+                .hasSize(existingSchema ? 1 : 0);
+
+        putSchema(schemaId, schemaData2, version(existingSchema ? 1 : 0));
+        assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(existingSchema ? 2 : 1);
+    }
+
+    @DataProvider
+    public Object[][] schemaCleanupFailureCases() {
+        return new Object[][] {{true, true}, {true, false}, {false, true}, {false, false}};
+    }
+
+    @Test(dataProvider = "schemaCleanupFailureCases")
+    public void testSchemaWriteFailurePreservesExceptionDuringCleanup(boolean closeSucceeds, boolean deleteSucceeds)
+            throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        bookKeeper.addEntryFailAfter(0, BKException.Code.WriteException);
+        // Creation is the first operation; the add failure has its own queue.
+        CompletableFuture<Void> close = bookKeeper.promiseAfter(1);
+        CompletableFuture<Void> deletion = bookKeeper.promiseAfter(2);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData1, SchemaCompatibilityStrategy.FULL);
+        try {
+            Awaitility.await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
+                assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(1);
+                assertThat(schemaLedgers(bookKeeper, schemaId).get(0).getAsyncCloseCount()).hasValue(1);
+            });
+            PulsarMockLedgerHandle ledger = schemaLedgers(bookKeeper, schemaId).get(0);
+            assertThat(put).isNotDone();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+
+            if (closeSucceeds) {
+                close.complete(null);
+            } else {
+                close.completeExceptionally(BKException.create(BKException.Code.NoSuchLedgerExistsException));
+            }
+            Awaitility.await().atMost(Duration.ofSeconds(10)).until(() -> deletion.getNumberOfDependents() > 0);
+            assertThat(put).isNotDone();
+            if (deleteSucceeds) {
+                deletion.complete(null);
+            } else {
+                deletion.completeExceptionally(BKException.create(BKException.Code.UnexpectedConditionException));
+            }
+            assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                    .hasRootCauseInstanceOf(SchemaException.class)
+                    .rootCause().hasMessage(BookkeeperSchemaStorage.bkException("Failed to add entry",
+                            BKException.Code.WriteException, ledger.getId(), -1, false).getMessage());
+            assertThat(ledger.getAsyncCloseCount()).hasValue(1);
+            assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(deleteSucceeds ? 0 : 1);
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+        } finally {
+            close.complete(null);
+            deletion.complete(null);
+        }
+    }
+
+    @Test
+    public void testSchemaCloseFailureDeletesOrphanLedger() throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        // Creation and the successful add precede close.
+        CompletableFuture<Void> close = bookKeeper.promiseAfter(2);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData1, SchemaCompatibilityStrategy.FULL);
+        try {
+            Awaitility.await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
+                assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(1);
+                assertThat(schemaLedgers(bookKeeper, schemaId).get(0).getAsyncCloseCount()).hasValue(1);
+            });
+            assertThat(put).isNotDone();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+            close.completeExceptionally(BKException.create(BKException.Code.WriteException));
+
+            assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                    .hasRootCauseInstanceOf(BKException.BKWriteException.class);
+            assertThat(schemaLedgers(bookKeeper, schemaId)).isEmpty();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Clean up orphan schema ledgers after write failures [pulsar]

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26902:
URL: https://github.com/apache/pulsar/pull/26902#discussion_r4279502942


##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/SchemaServiceTest.java:
##########
@@ -215,6 +223,129 @@ public void writeReadBackDeleteSchemaEntry() throws Exception {
         assertNull(schemaRegistryService.getSchema(schemaId1).get());
     }
 
+    @DataProvider
+    public Object[][] schemaWriteFailureCases() {
+        return new Object[][] {{false}, {true}};
+    }
+
+    @Test(dataProvider = "schemaWriteFailureCases")
+    public void testSchemaWriteFailureDeletesOrphanLedger(boolean existingSchema) throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        if (existingSchema) {
+            putSchema(schemaId, schemaData1, version(0));
+        }
+        String path = "/schemas/" + schemaId;
+        byte[] locatorBefore = pulsar.getLocalMetadataStore().get(path).get(10, TimeUnit.SECONDS)
+                .map(result -> result.getValue()).orElse(null);
+
+        bookKeeper.addEntryFailAfter(0, BKException.Code.WriteException);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData2, SchemaCompatibilityStrategy.FULL);
+
+        assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                .hasRootCauseInstanceOf(SchemaException.class)
+                .rootCause().hasMessageContaining("operation=Failed to add entry");
+        byte[] locatorAfter = pulsar.getLocalMetadataStore().get(path).get(10, TimeUnit.SECONDS)
+                .map(result -> result.getValue()).orElse(null);
+        assertThat(locatorAfter).isEqualTo(locatorBefore);
+        assertThat(schemaLedgers(bookKeeper, schemaId))
+                .as("A failed schema write must not leave an unreferenced ledger")
+                .hasSize(existingSchema ? 1 : 0);
+
+        putSchema(schemaId, schemaData2, version(existingSchema ? 1 : 0));
+        assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(existingSchema ? 2 : 1);
+    }
+
+    @DataProvider
+    public Object[][] schemaCleanupFailureCases() {
+        return new Object[][] {{true, true}, {true, false}, {false, true}, {false, false}};
+    }
+
+    @Test(dataProvider = "schemaCleanupFailureCases")
+    public void testSchemaWriteFailurePreservesExceptionDuringCleanup(boolean closeSucceeds, boolean deleteSucceeds)
+            throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        bookKeeper.addEntryFailAfter(0, BKException.Code.WriteException);
+        // Creation is the first operation; the add failure has its own queue.
+        CompletableFuture<Void> close = bookKeeper.promiseAfter(1);
+        CompletableFuture<Void> deletion = bookKeeper.promiseAfter(2);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData1, SchemaCompatibilityStrategy.FULL);
+        try {
+            Awaitility.await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
+                assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(1);
+                assertThat(schemaLedgers(bookKeeper, schemaId).get(0).getAsyncCloseCount()).hasValue(1);
+            });
+            PulsarMockLedgerHandle ledger = schemaLedgers(bookKeeper, schemaId).get(0);
+            assertThat(put).isNotDone();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+
+            if (closeSucceeds) {
+                close.complete(null);
+            } else {
+                close.completeExceptionally(BKException.create(BKException.Code.NoSuchLedgerExistsException));
+            }
+            Awaitility.await().atMost(Duration.ofSeconds(10)).until(() -> deletion.getNumberOfDependents() > 0);
+            assertThat(put).isNotDone();
+            if (deleteSucceeds) {
+                deletion.complete(null);
+            } else {
+                deletion.completeExceptionally(BKException.create(BKException.Code.UnexpectedConditionException));
+            }
+            assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                    .hasRootCauseInstanceOf(SchemaException.class)
+                    .rootCause().hasMessage(BookkeeperSchemaStorage.bkException("Failed to add entry",
+                            BKException.Code.WriteException, ledger.getId(), -1, false).getMessage());
+            assertThat(ledger.getAsyncCloseCount()).hasValue(1);
+            assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(deleteSucceeds ? 0 : 1);
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+        } finally {
+            close.complete(null);
+            deletion.complete(null);
+        }
+    }
+
+    @Test
+    public void testSchemaCloseFailureDeletesOrphanLedger() throws Exception {
+        String schemaId = "public/default/" + UUID.randomUUID();
+        PulsarMockBookKeeper bookKeeper = pulsarTestContext.getMockBookKeeper();
+        // Creation and the successful add precede close.
+        CompletableFuture<Void> close = bookKeeper.promiseAfter(2);
+        CompletableFuture<SchemaVersion> put = schemaRegistryService.putSchemaIfAbsent(
+                schemaId, schemaData1, SchemaCompatibilityStrategy.FULL);
+        try {
+            Awaitility.await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
+                assertThat(schemaLedgers(bookKeeper, schemaId)).hasSize(1);
+                assertThat(schemaLedgers(bookKeeper, schemaId).get(0).getAsyncCloseCount()).hasValue(1);
+            });
+            assertThat(put).isNotDone();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();
+            close.completeExceptionally(BKException.create(BKException.Code.WriteException));
+
+            assertThatThrownBy(() -> put.get(10, TimeUnit.SECONDS))
+                    .hasRootCauseInstanceOf(BKException.BKWriteException.class);
+            assertThat(schemaLedgers(bookKeeper, schemaId)).isEmpty();
+            assertThat(pulsar.getLocalMetadataStore().exists("/schemas/" + schemaId).get(10, TimeUnit.SECONDS))
+                    .isFalse();

Review Comment:
   Optional, non-blocking: Consider also failing deletion after a successful write and failed close, asserting that the close exception remains the reported failure. The current cleanup-failure matrix starts with a write failure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Clean up orphan schema ledgers after write failures [pulsar]

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #26902:
URL: https://github.com/apache/pulsar/pull/26902


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org