You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2012/08/18 02:36:45 UTC
Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
Review request for Flume and Brock Noland.
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10574
-----------------------------------------------------------
Ship it!
This improves the file channel under crash scenarios.
- Brock Noland
On Aug. 21, 2012, 3:01 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2012, 3:01 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
(Updated Aug. 21, 2012, 3:01 a.m.)
Review request for Flume and Brock Noland.
Changes
-------
Removing "queue.remove(FlumeEventPointer.fromLong(eventPointer));" from Replayhandler. Despite this fix, there is one more race condition: https://issues.apache.org/jira/browse/FLUME-1498
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
(Updated Aug. 20, 2012, 9:32 p.m.)
Review request for Flume and Brock Noland.
Changes
-------
Incorporated Brock's feedback.
Moved the log.commit calls and log.rollback calls.
I did not move the queue.* calls from Log.java back into transaction layer. A checkpoint in between log.put and queue.addWithoutCommit might be problematic, so decided to leave as is.
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
>
> Hari Shreedharan wrote:
> Thanks for the test. Do you mind if I added this to the unit test file?
>
> Thanks for such a detailed review!
>
> Regarding 1 and 2:
> * Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
>
> 3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
>
> Brock Noland wrote:
> Yes please add the unit test.
>
> Also, can you expand on #1? I got a log file like so:
>
> 1345405182865, 1345405184951, 2, 72894, Put, FlumeEventPointer [fileID=2, offset=72894]
> 1345405184952 -- checkpoint
> 1345405182865, 1345405184953, 2, 72974, Commit
>
> which was resolved by adding the:
> queue.remove(FlumeEventPointer.fromLong(eventPointer));
Brock:
A checkpoint would not get written unless and until at least one put gets committed(since elements.syncRequired() returns false). I just wrote a unit test that waits for a checkpoint after a put, and it just times out since the checkpoint never comes.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
>
> Hari Shreedharan wrote:
> Thanks for the test. Do you mind if I added this to the unit test file?
>
> Thanks for such a detailed review!
>
> Regarding 1 and 2:
> * Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
>
> 3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
>
> Brock Noland wrote:
> Yes please add the unit test.
>
> Also, can you expand on #1? I got a log file like so:
>
> 1345405182865, 1345405184951, 2, 72894, Put, FlumeEventPointer [fileID=2, offset=72894]
> 1345405184952 -- checkpoint
> 1345405182865, 1345405184953, 2, 72974, Commit
>
> which was resolved by adding the:
> queue.remove(FlumeEventPointer.fromLong(eventPointer));
>
> Hari Shreedharan wrote:
> Brock:
> A checkpoint would not get written unless and until at least one put gets committed(since elements.syncRequired() returns false). I just wrote a unit test that waits for a checkpoint after a put, and it just times out since the checkpoint never comes.
>
Right. But there is a race condition between:
synchronized (queue) {
while(!putList.isEmpty()) {
if(!queue.addTail(putList.removeFirst())) {
}
and the
queue.completeTransaction(transactionID);
which is in log.commit. So it's possible for a checkpoint to happen between the time we add the event to the queue (via addTail) and the time we complete the transaction. If a checkpoint happens during this time and we crash/restart, that event will be added to the queue twice, no? Once because it was added the queue and written to disk and once because it was written out to the in inflightputs file.
Seems like queue.completeTransaction(transactionID); should be done inside that queue lock.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
Thanks for the test. Do you mind if I added this to the unit test file?
Thanks for such a detailed review!
Regarding 1 and 2:
* Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 647
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line647>
> >
> > If we have no inflights this time, we will *not* overwrite the previous files which means they will get replayed later down the line.
No, we don't actually clear the inflights map on every write. So the previous set of values will get written out as is. That said, if there is a take commit, then syncRequired() returns false, and checkpoint will not be forced. So added a method to make sure take commits cause in flights to get written out.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 650
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line650>
> >
> > we never close, why are we rechecking to see if it's open? Also does fileChannel need to be volatile?
when we do future.cancel, that will cause the write to be interrupted throwing ClosedByInterruptedException, and thus closing the channel and the file.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 660
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line660>
> >
> > loooong line
I left it long so it is easy to understand what size maps to what.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 638
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line638>
> >
> > future.get is never called so any exception will not be propagated
Actually exceptions are not what we are looking for. Any exceptions will cause a bad checksum and we won't use the files. This is only meant to make sure there is only one write at a time.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 595
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line595>
> >
> > future gives generic warning
Thanks, fixed.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 590
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line590>
> >
> > Does this class have to be public and have public methods?
I don't mind it being public. Making it private will mean we need to add wrapper methods in FlumeEventQueue. Will do that.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 571
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line571>
> >
> > can you just print the contents of the inflight takes/puts as well?
Done.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 569
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line569>
> >
> > whats the FIXME?
I actually fixed it :-)
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 558
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line558>
> >
> > Can we move the main to the bottom?
ok.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 213
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line213>
> >
> > Do we have to expose this inner class since the only time this is used the deserialize method is immediately called
will do it through wrapper methods.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 391
> > <https://reviews.apache.org/r/6683/diff/3/?file=142789#file142789line391>
> >
> > Not a bad check, but I am not entirely sure this is needed since we use the semaphore to control capacity. Do you see a whole in that logic?
No, I just forgot about it when I added this. Removing.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
oops, ignore "with changes" and below.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
>
> Hari Shreedharan wrote:
> Thanks for the test. Do you mind if I added this to the unit test file?
>
> Thanks for such a detailed review!
>
> Regarding 1 and 2:
> * Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
>
> 3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
Yes please add the unit test.
Also, can you expand on #1? I got a log file like so:
1345405182865, 1345405184951, 2, 72894, Put, FlumeEventPointer [fileID=2, offset=72894]
1345405184952 -- checkpoint
1345405182865, 1345405184953, 2, 72974, Commit
which was resolved by adding the:
queue.remove(FlumeEventPointer.fromLong(eventPointer));
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 647
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line647>
> >
> > If we have no inflights this time, we will *not* overwrite the previous files which means they will get replayed later down the line.
>
> Hari Shreedharan wrote:
> No, we don't actually clear the inflights map on every write. So the previous set of values will get written out as is. That said, if there is a take commit, then syncRequired() returns false, and checkpoint will not be forced. So added a method to make sure take commits cause in flights to get written out.
So as long as we truncate the file it's fine. Reading the code, it actually truncates the file if you comment that out and it does work in practice as well.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 638
> > <https://reviews.apache.org/r/6683/diff/3/?file=142790#file142790line638>
> >
> > future.get is never called so any exception will not be propagated
>
> Hari Shreedharan wrote:
> Actually exceptions are not what we are looking for. Any exceptions will cause a bad checksum and we won't use the files. This is only meant to make sure there is only one write at a time.
Then I think we should log the exception in the runnable and not propagate.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
@Test
public void testTakeTransactionCrossingCheckpoint() throws Exception {
Map<String, String> overrides = Maps.newHashMap();
overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
String.valueOf(1000L));
channel = createFileChannel(overrides);
channel.start();
Assert.assertTrue(channel.isOpen());
List<String> in = Lists.newArrayList();
try {
while(true) {
in.addAll(putEvents(channel, "restart", 1, 1));
}
} catch (ChannelException e) {
Assert.assertEquals("Cannot acquire capacity. [channel="
+channel.getName()+"]", e.getMessage());
}
List<String> out = Lists.newArrayList();
// now take one item off the channel
Transaction tx = channel.getTransaction();
tx.begin();
Event e = channel.take();
Assert.assertNotNull(e);
String s = new String(e.getBody(), Charsets.UTF_8);
out.add(s);
LOG.info("Slow take got " + s);
Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
// and commit is after the checkpoint
tx.commit();
tx.close();
channel.stop();
channel = createFileChannel(overrides);
channel.start();
Assert.assertTrue(channel.isOpen());
// we should not geet the item we took of the queue above
out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
Collections.sort(in);
Collections.sort(out);
if(!out.equals(in)) {
List<String> difference = new ArrayList<String>();
if(in.size() > out.size()) {
LOG.info("The channel shorted us");
difference.addAll(in);
difference.removeAll(out);
} else {
LOG.info("We got more events than expected, perhaps dups");
difference.addAll(out);
difference.removeAll(in);
}
LOG.error("difference = " + difference +
", in.size = " + in.size() + ", out.size = " + out.size());
Assert.fail();
}
}
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
>
> Hari Shreedharan wrote:
> Thanks for the test. Do you mind if I added this to the unit test file?
>
> Thanks for such a detailed review!
>
> Regarding 1 and 2:
> * Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
>
> 3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
>
> Brock Noland wrote:
> Yes please add the unit test.
>
> Also, can you expand on #1? I got a log file like so:
>
> 1345405182865, 1345405184951, 2, 72894, Put, FlumeEventPointer [fileID=2, offset=72894]
> 1345405184952 -- checkpoint
> 1345405182865, 1345405184953, 2, 72974, Commit
>
> which was resolved by adding the:
> queue.remove(FlumeEventPointer.fromLong(eventPointer));
>
> Hari Shreedharan wrote:
> Brock:
> A checkpoint would not get written unless and until at least one put gets committed(since elements.syncRequired() returns false). I just wrote a unit test that waits for a checkpoint after a put, and it just times out since the checkpoint never comes.
>
>
> Brock Noland wrote:
> Right. But there is a race condition between:
>
> synchronized (queue) {
> while(!putList.isEmpty()) {
> if(!queue.addTail(putList.removeFirst())) {
> }
>
> and the
>
> queue.completeTransaction(transactionID);
>
> which is in log.commit. So it's possible for a checkpoint to happen between the time we add the event to the queue (via addTail) and the time we complete the transaction. If a checkpoint happens during this time and we crash/restart, that event will be added to the queue twice, no? Once because it was added the queue and written to disk and once because it was written out to the in inflightputs file.
>
> Seems like queue.completeTransaction(transactionID); should be done inside that queue lock.
>
> Hari Shreedharan wrote:
> You are right. There are 2 ways in whcih we could resolve this :
> 1. Move the queue.completeTxn() calls to the transaction level, and put them in synchronized(queue) blocks - so that the log and the queue are written atomically. This will reduce the concurrency of take commits and rollbacks from the current levels, though it is unlikely to affect put commits that much(considering the amount of work already being done).
> 2. Let the race condition be, and let replay handle the case where the events are both in pending and in the queue(by called queue.remove()).
>
> I prefer 2 over 1 because that will not affect the performance of the channel for take commits and rollbacks. Thoughts?
I can confirm both 1 and 2 worked in my testing. I prefer 1. The overhead should be trivial since it's an in memory operation on a very small data structure. It's also is more obvious, I had a good time finding that the queue methods were being called in the Log class and not the Transaction class as that is where the rest of the operations are at.
I'd also prefer we move any of the queue operations (like the on in the put) up the Transaction level. In the Log class other than calling checkpoint and a few getters we don't do any queue operations. They are all up in the Transaction class.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On Aug. 19, 2012, 10:51 p.m., Brock Noland wrote:
> > Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
> >
> > == Review items ==
> >
> > 1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
> >
> > put
> > checkpoint (put is written to in flights)
> > commit
> > replay
> >
> > the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
> >
> > transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
> > queue.remove(FlumeEventPointer.fromLong(eventPointer));
> >
> > That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
> >
> >
> > 2) The failure I got after 170 runs was caused by this scenerio:
> >
> > put
> > checkpoint (put is written to inflights)
> > commit
> > checkpoint (no in flights and as such inflight files are not updated, thus have old data)
> > replay
> >
> > After commenting out:
> >
> > if(values.isEmpty()){
> > return;
> > }
> >
> > in the serializeAndWrite method, the test ran 306 times in a row without failing.
> >
> > 3) I a little unsure of the inflight take logic.
> >
> > take
> > checkpoint
> > commit
> >
> > On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
> >
> >
> >
> > == Wishlist ==
> >
> > Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
> >
> > inflighttakes and inflighttakes.meta where the meta file would have the checksum
> >
> > This might be a .proto file which would work.
> >
> > message InFlightTransactions {
> > repeated InFlightTransaction transactions = 1;
> > }
> >
> > message InFlightTransaction {
> > required sfixed64 transactionID = 1;
> > repeated sfixed64 pointers = 3;
> > }
> >
> > message InFlightTransactionsMetaData {
> > required bytes checksum = 1;
> > }
> >
> >
> >
> > with changes
> > v1 9.5%
> > fail 20
> > success 211
> >
> > v2 7.5%
> > fail 7
> > success 93
> >
> > without
> > v1 3.9%
> > fail 5
> > success 127
> >
> > v2 5.2%
> > fail 5
> > success 95
> >
>
> Brock Noland wrote:
> oops, ignore "with changes" and below.
>
> Brock Noland wrote:
> Regarding the scenario in #3, I believe this test encapsulates the problem (with the patch applied)
>
> @Test
> public void testTakeTransactionCrossingCheckpoint() throws Exception {
> Map<String, String> overrides = Maps.newHashMap();
> overrides.put(FileChannelConfiguration.CHECKPOINT_INTERVAL,
> String.valueOf(1000L));
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> List<String> in = Lists.newArrayList();
> try {
> while(true) {
> in.addAll(putEvents(channel, "restart", 1, 1));
> }
> } catch (ChannelException e) {
> Assert.assertEquals("Cannot acquire capacity. [channel="
> +channel.getName()+"]", e.getMessage());
> }
> List<String> out = Lists.newArrayList();
> // now take one item off the channel
> Transaction tx = channel.getTransaction();
> tx.begin();
> Event e = channel.take();
> Assert.assertNotNull(e);
> String s = new String(e.getBody(), Charsets.UTF_8);
> out.add(s);
> LOG.info("Slow take got " + s);
> Thread.sleep(2000L); // sleep so a checkpoint occurs. take is before
> // and commit is after the checkpoint
> tx.commit();
> tx.close();
> channel.stop();
> channel = createFileChannel(overrides);
> channel.start();
> Assert.assertTrue(channel.isOpen());
> // we should not geet the item we took of the queue above
> out.addAll(takeEvents(channel, 1, Integer.MAX_VALUE));
> Collections.sort(in);
> Collections.sort(out);
> if(!out.equals(in)) {
> List<String> difference = new ArrayList<String>();
> if(in.size() > out.size()) {
> LOG.info("The channel shorted us");
> difference.addAll(in);
> difference.removeAll(out);
> } else {
> LOG.info("We got more events than expected, perhaps dups");
> difference.addAll(out);
> difference.removeAll(in);
> }
> LOG.error("difference = " + difference +
> ", in.size = " + in.size() + ", out.size = " + out.size());
> Assert.fail();
> }
> }
>
> Hari Shreedharan wrote:
> Thanks for the test. Do you mind if I added this to the unit test file?
>
> Thanks for such a detailed review!
>
> Regarding 1 and 2:
> * Only if commits happen, the puts are added to the queue. If the commit is not seen the event is never added to the queue. So the checkpoint will not have the puts if a commit did not happen(hence the need for inflight puts in the first place). And during replay only if a commit for a transaction is seen, do we actually put it into the queue. Also if a put commit happens between 2 checkpoints, then that causes the elements.syncRequired() method to return true, which will force a new inflights file to get written, if values has some content to be written. The problem is that when values.isEmpty() returns true, we still need to truncate the old file, so that we don't leave stale data as is. Nice catch!
>
> 3 is interesting. This was just me being lazy. I wanted to put the take replays also in the replay logic, but thought there is no need, but it turns out there is! Will do that.
>
> Brock Noland wrote:
> Yes please add the unit test.
>
> Also, can you expand on #1? I got a log file like so:
>
> 1345405182865, 1345405184951, 2, 72894, Put, FlumeEventPointer [fileID=2, offset=72894]
> 1345405184952 -- checkpoint
> 1345405182865, 1345405184953, 2, 72974, Commit
>
> which was resolved by adding the:
> queue.remove(FlumeEventPointer.fromLong(eventPointer));
>
> Hari Shreedharan wrote:
> Brock:
> A checkpoint would not get written unless and until at least one put gets committed(since elements.syncRequired() returns false). I just wrote a unit test that waits for a checkpoint after a put, and it just times out since the checkpoint never comes.
>
>
> Brock Noland wrote:
> Right. But there is a race condition between:
>
> synchronized (queue) {
> while(!putList.isEmpty()) {
> if(!queue.addTail(putList.removeFirst())) {
> }
>
> and the
>
> queue.completeTransaction(transactionID);
>
> which is in log.commit. So it's possible for a checkpoint to happen between the time we add the event to the queue (via addTail) and the time we complete the transaction. If a checkpoint happens during this time and we crash/restart, that event will be added to the queue twice, no? Once because it was added the queue and written to disk and once because it was written out to the in inflightputs file.
>
> Seems like queue.completeTransaction(transactionID); should be done inside that queue lock.
You are right. There are 2 ways in whcih we could resolve this :
1. Move the queue.completeTxn() calls to the transaction level, and put them in synchronized(queue) blocks - so that the log and the queue are written atomically. This will reduce the concurrency of take commits and rollbacks from the current levels, though it is unlikely to affect put commits that much(considering the amount of work already being done).
2. Let the race condition be, and let replay handle the case where the events are both in pending and in the queue(by called queue.remove()).
I prefer 2 over 1 because that will not affect the performance of the channel for take commits and rollbacks. Thoughts?
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10513
-----------------------------------------------------------
Nice patch! This looks to remove many of the probabilistic test failures of testRestartLogReplayV{1,2} which is exactly what I was hoping for! A couple of items to work on below but overall I think the approach is sound.
== Review items ==
1) OK, this fixes the big TestFileChannel.testRestartLogReplayV{1,2} failure mode, that is lost puts and takes. With the fix it still fails eventually to replay the logs. The reason I believe this is true is that we can have this scenario:
put
checkpoint (put is written to in flights)
commit
replay
the put is written out in the inflight puts file and then on replay it's added to the transaction map and put back into the queue, but it was also in the queue at checkpoint time. I was able to get the test to pass 170 times in a row by adding a queue remove in the replayLog method:
transactionMap.put(txnID, FlumeEventPointer.fromLong(eventPointer));
queue.remove(FlumeEventPointer.fromLong(eventPointer));
That is, if it's truly inflight, then a commit has not occurred and the record will be added to the queue when the commit has is replayed.
2) The failure I got after 170 runs was caused by this scenerio:
put
checkpoint (put is written to inflights)
commit
checkpoint (no in flights and as such inflight files are not updated, thus have old data)
replay
After commenting out:
if(values.isEmpty()){
return;
}
in the serializeAndWrite method, the test ran 306 times in a row without failing.
3) I a little unsure of the inflight take logic.
take
checkpoint
commit
On replay, put the take back in the queue and then skip ahead to the checkpoint. At that point we replay the commit the but the commit has no seen the takes so it will not remove them the queue?
== Wishlist ==
Since we are planning on making the rest of the file format more extensible, would you be opposed to using protocol buffers for these two files? That way we wouldn't have to upgrade when we integrate this with FLUME-1487. Basically you could copy the protocol buffers generation code from FLUME-1487. In that change we stop doing random writes to files so we'd have two files:
inflighttakes and inflighttakes.meta where the meta file would have the checksum
This might be a .proto file which would work.
message InFlightTransactions {
repeated InFlightTransaction transactions = 1;
}
message InFlightTransaction {
required sfixed64 transactionID = 1;
repeated sfixed64 pointers = 3;
}
message InFlightTransactionsMetaData {
required bytes checksum = 1;
}
with changes
v1 9.5%
fail 20
success 211
v2 7.5%
fail 7
success 93
without
v1 3.9%
fail 5
success 127
v2 5.2%
fail 5
success 95
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
<https://reviews.apache.org/r/6683/#comment22502>
Not a bad check, but I am not entirely sure this is needed since we use the semaphore to control capacity. Do you see a whole in that logic?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22501>
Do we have to expose this inner class since the only time this is used the deserialize method is immediately called
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22492>
Can we move the main to the bottom?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22493>
whats the FIXME?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22503>
can you just print the contents of the inflight takes/puts as well?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22494>
Does this class have to be public and have public methods?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22495>
future gives generic warning
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22496>
future.get is never called so any exception will not be propagated
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22497>
Exception catching is too wide
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22498>
If we have no inflights this time, we will *not* overwrite the previous files which means they will get replayed later down the line.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22500>
we never close, why are we rechecking to see if it's open? Also does fileChannel need to be volatile?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22499>
loooong line
- Brock Noland
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/#review10515
-----------------------------------------------------------
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
<https://reviews.apache.org/r/6683/#comment22504>
should we lock this file to ensure two threads don't do this write at the same time?
- Brock Noland
On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6683/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2012, 8:40 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
>
> I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
> With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
>
>
> This addresses bug FLUME-1437.
> https://issues.apache.org/jira/browse/FLUME-1437
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
>
> Diff: https://reviews.apache.org/r/6683/diff/
>
>
> Testing
> -------
>
> Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
(Updated Aug. 18, 2012, 8:40 a.m.)
Review request for Flume and Brock Noland.
Changes
-------
Seems like createTempDir does not return unique dirs each time. This was causing some flakiness in tests. Fixed that by sleeping test which spawned new threads and didn't call get() on the future.
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1437: FileChannel should write inflight events to
checkpoint.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6683/
-----------------------------------------------------------
(Updated Aug. 18, 2012, 1:58 a.m.)
Review request for Flume and Brock Noland.
Changes
-------
Updated the new tests. Also removed unused imports.
Description
-------
Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts).
I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean.
With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that.
This addresses bug FLUME-1437.
https://issues.apache.org/jira/browse/FLUME-1437
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f
Diff: https://reviews.apache.org/r/6683/diff/
Testing
-------
Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization).
Thanks,
Hari Shreedharan