You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/09/07 05:24:08 UTC
[jira] [Created] (HDFS-3900) QJM: avoid validating log segments on
log rolls
Todd Lipcon created HDFS-3900:
---------------------------------
Summary: QJM: avoid validating log segments on log rolls
Key: HDFS-3900
URL: https://issues.apache.org/jira/browse/HDFS-3900
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Currently, we are paranoid and validate every log segment when it is finalized. For the a log segment that has been written entirely by one writer, with no recovery in between, this is overly paranoid (we don't do this for local journals). It also causes log rolls to be slow and take time linear in the size of the segment. Instead, we should optimize this path to simply trust that the segment is correct so long as the txids match up as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3900) QJM: avoid validating log segments on
log rolls
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-3900.
-------------------------------
Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
Hadoop Flags: Reviewed
Committed to branch, thanks
> QJM: avoid validating log segments on log rolls
> -----------------------------------------------
>
> Key: HDFS-3900
> URL: https://issues.apache.org/jira/browse/HDFS-3900
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: QuorumJournalManager (HDFS-3077)
>
> Attachments: hdfs-3900.txt
>
>
> Currently, we are paranoid and validate every log segment when it is finalized. For the a log segment that has been written entirely by one writer, with no recovery in between, this is overly paranoid (we don't do this for local journals). It also causes log rolls to be slow and take time linear in the size of the segment. Instead, we should optimize this path to simply trust that the segment is correct so long as the txids match up as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira