You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2012/08/15 06:50:37 UTC
[jira] [Created] (FLUME-1485) File Channel should support checksum
Hari Shreedharan created FLUME-1485:
---------------------------------------
Summary: File Channel should support checksum
Key: FLUME-1485
URL: https://issues.apache.org/jira/browse/FLUME-1485
Project: Flume
Issue Type: Bug
Reporter: Hari Shreedharan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434811#comment-13434811 ]
Hari Shreedharan commented on FLUME-1485:
-----------------------------------------
I'm ok with file level checksum. Transaction level might need considerable changes in the file format etc.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1485) File Channel should support checksum
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1485:
------------------------------------
Issue Type: New Feature (was: Bug)
Ideally, this will be implemented in such a way that the channel format would not change. So the checksum will either need to go at the end or in a different file. So the new code can check for the (optional) checksum and use it. If no checksum is found, alert of channel corruption.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435312#comment-13435312 ]
Mike Percy commented on FLUME-1485:
-----------------------------------
Doesn't this assist replay in identifying an incomplete event in the case of a system failure mid-write? Not sure what the value of it is at a file level.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1485) File Channel should support checksum
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1485:
--------------------------------
Fix Version/s: (was: v1.3.0)
v1.4.0
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.4.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434883#comment-13434883 ]
Brock Noland commented on FLUME-1485:
-------------------------------------
Yeah I am +1 on file level.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435405#comment-13435405 ]
Mike Percy commented on FLUME-1485:
-----------------------------------
It doesn't tell you where the error is though. So what can you do other than throw out an entire file with an invalid checksum?
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435362#comment-13435362 ]
Brock Noland commented on FLUME-1485:
-------------------------------------
We should know if we have an incomplete event because we first write the length and then the bytes. To me what this does is tell us where the file is corrupt (bits flipped) or not. Transaction or event level checksums gives us granular checks but at a higher cost. File level gives us less granular checks but at a lower cost.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1485) File Channel should support checksum
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1485:
------------------------------------
Component/s: Channel
Fix Version/s: v1.3.0
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Denny Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434809#comment-13434809 ]
Denny Ye commented on FLUME-1485:
---------------------------------
which level? each transaction or identified file?
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435447#comment-13435447 ]
Mike Percy commented on FLUME-1485:
-----------------------------------
By the way, thanks for the explanation Brock, totally see where you're coming from on the format.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435446#comment-13435446 ]
Mike Percy commented on FLUME-1485:
-----------------------------------
Makes sense.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435424#comment-13435424 ]
Brock Noland commented on FLUME-1485:
-------------------------------------
Yes, all you could do is notify the user and let them decide whether they want to replay that log. However, knowing that there was corruption is valuable.
Sure, ultimately, we could do this on a per event basis, but doing so would require changes to the file channel format. Until we resolve FLUME-1487 (just created) I am not for making the file channel checkpoint/log format more complex.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1485) File Channel should support
checksum
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442636#comment-13442636 ]
Brock Noland commented on FLUME-1485:
-------------------------------------
With FLUME-1487 Each event has a header with control information, the event itself, and a footer. We could use a simple input/output stream wrapper to calculate the checksum for each event and then store it in the footer. This was we could verify each event in whole.
> File Channel should support checksum
> ------------------------------------
>
> Key: FLUME-1485
> URL: https://issues.apache.org/jira/browse/FLUME-1485
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira