You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/05/06 09:49:07 UTC
Review Request: FLUME-1586. File Channel should support verifying integrity
of individual events.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
Review request for Flume.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a7aa70c
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e61437d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopEventRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
pom.xml 686ecb6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated June 12, 2013, 6:51 p.m.)
Review request for Flume.
Changes
-------
Rebased on trunk
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
bin/flume-ng 22b95b8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-dist/pom.xml e266272
flume-ng-dist/src/main/assembly/bin.xml 74f0608
flume-ng-dist/src/main/assembly/src.xml 7fafab8
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
pom.xml 9982f5d
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review21777
-----------------------------------------------------------
Ship it!
Looks good! I tried applying the patch but it didn't apply. Can you rebase and then post the patch to the JIRA?
- Brock Noland
On May 24, 2013, 4:59 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 24, 2013, 4:59 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
> https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
> bin/flume-ng 22b95b8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
> flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
> flume-ng-dist/pom.xml 1a44626
> flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
> flume-ng-dist/src/main/assembly/src.xml 7fafab8
> flume-tools/pom.xml PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
> pom.xml a6992f6
>
> Diff: https://reviews.apache.org/r/10944/diff/
>
>
> Testing
> -------
>
> Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 24, 2013, 4:59 a.m.)
Review request for Flume.
Changes
-------
Check that files are backed up in the tests.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
bin/flume-ng 22b95b8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-dist/pom.xml 1a44626
flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
flume-ng-dist/src/main/assembly/src.xml 7fafab8
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 24, 2013, midnight)
Review request for Flume.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
bin/flume-ng 22b95b8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-dist/pom.xml 1a44626
flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
flume-ng-dist/src/main/assembly/src.xml 7fafab8
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 23, 2013, 11:14 p.m.)
Review request for Flume.
Changes
-------
Brock's review feedback incorporated.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
bin/flume-ng 22b95b8
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-dist/pom.xml 1a44626
flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
flume-ng-dist/src/main/assembly/src.xml 7fafab8
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On May 23, 2013, 10:01 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java, line 60
> > <https://reviews.apache.org/r/10944/diff/6/?file=295919#file295919line60>
> >
> > Curious as to what byte this ends up representing?
Oh, I guess I did not read this correctly earlier. The value of this byte is 0.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review20956
-----------------------------------------------------------
On May 23, 2013, 11:14 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 23, 2013, 11:14 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
> https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
> bin/flume-ng 22b95b8
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
> flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
> flume-ng-dist/pom.xml 1a44626
> flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
> flume-ng-dist/src/main/assembly/src.xml 7fafab8
> flume-tools/pom.xml PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
> pom.xml a6992f6
>
> Diff: https://reviews.apache.org/r/10944/diff/
>
>
> Testing
> -------
>
> Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review20956
-----------------------------------------------------------
Hari, this is great! I have a few questions/comments below. I just did an initial pass. Also since we are making so many classes public should we add visibility annotations to those classes?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43254>
Curious as to what byte this ends up representing?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43255>
this. is not consistent with the above
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43256>
Let's log the offset and file like the message above.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
<https://reviews.apache.org/r/10944/#comment43250>
Curious why we have "" + ".old"?
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
<https://reviews.apache.org/r/10944/#comment43253>
This should be typed IllegalStateException no?
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java
<https://reviews.apache.org/r/10944/#comment43262>
Curious as to what case would make eventPosition == -1?
- Brock Noland
On May 23, 2013, 8:54 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 23, 2013, 8:54 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
> https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
> flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
> flume-ng-dist/pom.xml 1a44626
> flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
> flume-ng-dist/src/main/assembly/src.xml 7fafab8
> flume-tools/pom.xml PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
> pom.xml a6992f6
>
> Diff: https://reviews.apache.org/r/10944/diff/
>
>
> Testing
> -------
>
> Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review20961
-----------------------------------------------------------
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43311>
This byte represents an event which was detected as corrupt, and then replaced by the tool. Basically the idea is that if this is seen the reads should simply ignore this event (log and ignore).
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43312>
Will drop the "this." in the next patch.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43313>
Will fix
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43316>
Will fix.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
<https://reviews.apache.org/r/10944/#comment43317>
Intellij craziness. Will fix.
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java
<https://reviews.apache.org/r/10944/#comment43318>
There should be none. WIll remove the if.
- Hari Shreedharan
On May 23, 2013, 8:54 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 23, 2013, 8:54 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
> https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
> flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
> flume-ng-dist/pom.xml 1a44626
> flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
> flume-ng-dist/src/main/assembly/src.xml 7fafab8
> flume-tools/pom.xml PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
> flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
> flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
> pom.xml a6992f6
>
> Diff: https://reviews.apache.org/r/10944/diff/
>
>
> Testing
> -------
>
> Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
>
>
> Thanks,
>
> Hari Shreedharan
>
>
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 23, 2013, 8:54 p.m.)
Review request for Flume.
Changes
-------
Maven changes. Make one class public, which is required for build. The jar still does not run, will fix that in a future patch - but the code is ready to review.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-dist/pom.xml 1a44626
flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
flume-ng-dist/src/main/assembly/src.xml 7fafab8
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 23, 2013, 7:59 p.m.)
Review request for Flume.
Changes
-------
Rebasing on trunk. Making several classes package private again. Removed unused imports
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 23, 2013, 1:52 a.m.)
Review request for Flume.
Changes
-------
Add tests + refactoring.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java c447335
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java 5f06ab7
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java dfcdd73
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
pom.xml a6992f6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 6, 2013, 9:28 p.m.)
Review request for Flume.
Changes
-------
Backup file before updating events
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a7aa70c
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e61437d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopEventRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
pom.xml 686ecb6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan
Re: Review Request: FLUME-1586. File Channel should support verifying
integrity of individual events.
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------
(Updated May 6, 2013, 8:12 p.m.)
Review request for Flume.
Changes
-------
Removed a bunch of sysouts.
Description
-------
Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a7aa70c
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e61437d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopEventRecordException.java PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
flume-tools/pom.xml PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
pom.xml 686ecb6
Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------
Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,
Hari Shreedharan