[GitHub] athanatos commented on a change in pull request #1044: AuditorPeriodicCheckTest: add test for entries with failed bookie writes

[GitBox <gi...@apache.org>]

athanatos commented on a change in pull request #1044: AuditorPeriodicCheckTest: add test for entries with failed bookie writes
URL: https://github.com/apache/bookkeeper/pull/1044#discussion_r163420990
 
 

 ##########
 File path: bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorPeriodicCheckTest.java
 ##########
 @@ -348,4 +354,114 @@ public void run() {
         t.join();
         assertFalse("Shouldn't have thrown exception", exceptionCaught.get());
     }
+
+    private BookieSocketAddress replaceBookieWithWriteFailingBookie(LedgerHandle lh) throws Exception {
+        int bookieIdx = -1;
+        Long entryId = LedgerHandleAdapter.getLedgerMetadata(lh).getEnsembles().firstKey();
+        ArrayList<BookieSocketAddress> curEnsemble = LedgerHandleAdapter
+                .getLedgerMetadata(lh).getEnsembles().get(entryId);
+
+        // Identify a bookie in the current ledger ensemble to be replaced
+        BookieSocketAddress replacedBookie = null;
+        for (int i = 0; i < numBookies; i++) {
+            if (curEnsemble.contains(bs.get(i).getLocalAddress())) {
+                bookieIdx = i;
+                replacedBookie = bs.get(i).getLocalAddress();
+                break;
+            }
+        }
+        assertNotEquals("Couldn't find ensemble bookie in bookie list", -1, bookieIdx);
+
+        LOG.info("Killing bookie " + bs.get(bookieIdx).getLocalAddress());
+        ServerConfiguration conf = killBookie(bookieIdx);
+        Bookie writeFailingBookie = new Bookie(conf) {
+            @Override
+            public void addEntry(ByteBuf entry, WriteCallback cb,
+                             Object ctx, byte[] masterKey)
+                             throws IOException, BookieException {
+                try {
+                    LOG.info("Failing write to entry ");
+                    // sleep a bit so that writes to other bookies succeed before
+                    // the client hears about the failure on this bookie. If the
+                    // client gets ack-quorum number of acks first, it won't care
+                    // about any failures and won't reform the ensemble.
+                    Thread.sleep(100);
+                    throw new IOException();
+                } catch (InterruptedException ie) {
+                    // ignore, only interrupted if shutting down,
+                    // and an exception would spam the logs
+                    Thread.currentThread().interrupt();
+                }
+            }
+        };
+        bsConfs.add(conf);
+        bs.add(startBookie(conf, writeFailingBookie));
+        return replacedBookie;
+    }
+
+    /*
+     * Validates that the periodic ledger check will fix entries with a failed write.
+     */
+    @Test
+    public void testFailedWriteRecovery() throws Exception {
+        LedgerManagerFactory mFactory = LedgerManagerFactory.newLedgerManagerFactory(
+                bsConfs.get(0),
+                RegistrationManager.instantiateRegistrationManager(bsConfs.get(0)).getLayoutManager());
+        LedgerUnderreplicationManager underReplicationManager = mFactory.newLedgerUnderreplicationManager();
+        underReplicationManager.disableLedgerReplication();
+
+        LedgerHandle lh = bkc.createLedger(2, 2, 1, DigestType.CRC32, "passwd".getBytes());
+
+        // kill one of the bookies and replace it with one that rejects write;
+        // This way we get into the under replication state
+        BookieSocketAddress replacedBookie = replaceBookieWithWriteFailingBookie(lh);
+
+        // Write a few entries; this should cause under replication
+        byte[] data = "foobar".getBytes();
+        data = "foobar".getBytes();
+        lh.addEntry(data);
+        lh.addEntry(data);
+        lh.addEntry(data);
+
+        lh.close();
+
+        // enable under replication detection and wait for it to report
+        // under replicated ledger
+        underReplicationManager.enableLedgerReplication();
+        long underReplicatedLedger = -1;
+        for (int i = 0; i < 5; i++) {
+            underReplicatedLedger = underReplicationManager.pollLedgerToRereplicate();
+            if (underReplicatedLedger != -1) {
+                break;
+            }
+            Thread.sleep(CHECK_INTERVAL * 1000);
+        }
+        assertEquals("Ledger should be under replicated", lh.getId(), underReplicatedLedger);
+
+        // now start the replication workers
+        List<ReplicationWorker> l = new ArrayList<ReplicationWorker>();
+        for (int i = 0; i < numBookies; i++) {
+            ReplicationWorker rw = new ReplicationWorker(
+                    zkc, bsConfs.get(i), NullStatsLogger.INSTANCE);
+            rw.start();
+            l.add(rw);
+        }
+        underReplicationManager.close();
+
+        // Wait for ensemble to change after replication
+        Thread.sleep(3000);
 
 Review comment:
   Yeah, I haven't had the time to improve this one.  If it turns out to be a problem I'll take another look.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services