You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Adam Antal (JIRA)" <ji...@apache.org> on 2018/08/09 09:56:00 UTC
[jira] [Commented] (HDFS-13031) To detect fsimage corruption on the
spot
[ https://issues.apache.org/jira/browse/HDFS-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574593#comment-16574593 ]
Adam Antal commented on HDFS-13031:
-----------------------------------
Looking through thoroughly the cases there are several aspects of the problems that need to be addressed carefully (plus recapping what's said so far):
1. There may be multiple bugs that can cause corruption in the FSImage. Besides knowing that snapshotting always plays a part, the exact cause is yet unknown.
2. Usually when the corruption turns out, the good fsimage is already deleted, and we couldn't replay previous actions, therefore couldn't get closer to the root cause.
3. If there were at least one case where we could obtain and examine an audit.log around the corruption event (provided the customer realize the corruption and shutdown the cluster in time), we would have an idea what causes the corruption ultimately leading to a solution.
4. We can provide a patch for HDFS to just ignore the NPE. It would cause loss of data for the customer, thus it should be avoided.
5. Apart from a NN startup throwing the error, there is nothing in the customer's hand that could reassure him/her that the FSImages is good or corrupted.
6. Solutions like put up a tertiary NN, and try running it to make sure of the correctness of the FSImage is a little bit of overkill. The goal is to detect the corruption on the spot, possibly without starting a tertiary NN.
7. Patching HDFS with "if you detect something nasty while writing out the FSImage, stop" is basically equivalent with finding and solving the bug itself, so the FSImage must be loaded again after written out.
8. An additional NN would be an overkill so another program is advisable. The OIV would be a handy choice, while building an independent tool for this sole purpose is may be too much effort.
9. The OIV already has functionality like loading the fsimage and constructing the folder structure, we just have to add the option of detecting the null INodes.
10. For e.g. the Delimited OIV processor can already use in disk MetadataMap, which reduces memory consumption. Also there may be a window for parallelizing: iterating through INodes for e.g. could be done distributed, increasing efficiency, and we wouldn't need a high mem-high CPU setup for just checking the FSImage.
For the above mentioned reasons I suggest a modified OIV processor as a solution for detecting fsimage corruption on the spot.
As a sidemark: it is another approach to HDFS-13314. The OIV can be run as a guarantee: if corruption is detected, the NN exits.
What do you think about?
> To detect fsimage corruption on the spot
> ----------------------------------------
>
> Key: HDFS-13031
> URL: https://issues.apache.org/jira/browse/HDFS-13031
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Environment:
> Reporter: Yongjun Zhang
> Assignee: Adam Antal
> Priority: Major
>
> Since we fixed HDFS-9406, there are new cases reported from the field that similar fsimage corruption happens. We need good fsimage + editlogs to replay to reproduce the corruption. However, usually when the corruption is detected (at later NN restart), the good fsimage is already deleted.
> We need to have a way to detect fsimage corruption on the spot. Currently what I think we could do is:
> # after SNN creates a new fsimage, it spawn a new modified NN process (NN with some new command line args) to just load the fsimage and do nothing else.
> # If the process failed, the currently running SNN will do either a) backup the fsimage + editlogs or b) no longer do checkpointing. And it need to somehow raise a flag to user that the fsimage is corrupt.
> In step 2, if we do a, we need to introduce new NN->JN API to backup editlogs; if we do b, it changes SNN's behavior, and kind of not compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org