You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Zhe Zhang (JIRA)" <ji...@apache.org> on 2015/08/26 23:10:47 UTC
[jira] [Updated] (HDFS-8964) Provide max TxId when validating
in-progress edit log files
[ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhe Zhang updated HDFS-8964:
----------------------------
Description:
NN/JN validates in-progress edit log files in multiple scenarios, via {{EditLogFile#validateLog}}. The method scans through the edit log file to find the last transaction ID.
However, an in-progress edit log file could be actively written to, which creates a race condition and causes incorrect data to be read (and later we attempt to interpret the data as ops).
Currently {{validateLog}} is used in 3 places:
# NN {{getEditsFromTxid}}
# JN {{getEditLogManifest}}
# NN/JN {{recoverUnfinalizedSegments}}
In the first two scenarios we should provide a maximum TxId to validate in the in-progress file. The 3rd scenario won't cause a race condition because only non-current in-progress edit log files are validated.
{{validateLog}} is actually only used with in-progress files, and could use a better name and Javadoc.
was:
NN/JN validates in-progress edit log files in multiple scenarios, via {{EditLogFile#validateLog}}. The method scans through the edit log file to find the last transaction ID.
However, an in-progress edit log file could be actively written to, which creates a race condition and causes incorrect data to be read (and later we attempt to interpret the data as ops).
Currently {{validateLog}} is used in 3 places:
# NN {{getEditsFromTxid}}
# JN {{getEditLogManifest}}
# NN/JN {{recoverUnfinalizedSegments}}
In the first two scenarios we should provide a maximum TxId to validate in the in-progress file. The 3rd scenario won't cause a race condition because only non-current in-progress edit log files are validated.
> Provide max TxId when validating in-progress edit log files
> -----------------------------------------------------------
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namenode
> Affects Versions: 2.7.1
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
>
> NN/JN validates in-progress edit log files in multiple scenarios, via {{EditLogFile#validateLog}}. The method scans through the edit log file to find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which creates a race condition and causes incorrect data to be read (and later we attempt to interpret the data as ops).
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in the in-progress file. The 3rd scenario won't cause a race condition because only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a better name and Javadoc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)