You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Weiwei Yang (JIRA)" <ji...@apache.org> on 2017/05/31 09:40:04 UTC
[jira] [Comment Edited] (HDFS-11886) Ozone : improve error handling
for putkey operation
[ https://issues.apache.org/jira/browse/HDFS-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030899#comment-16030899 ]
Weiwei Yang edited comment on HDFS-11886 at 5/31/17 9:39 AM:
-------------------------------------------------------------
Hi [~anu]
Thank you for the quick response. Keep uncommitted keys in memory is less reliable but faster and easier. If you are going to store state in DB, then we need to deal with rollback problems and more error handling code (update that DB has chance to fail too). And we need to think how to recover state from that when KSM restarted, is it necessary or not to recover uncommitted keys in this DB? It will be hard because we don't know which stage this key has been processed to.
was (Author: cheersyang):
Hi [~anu]
Thank you for the quick response. Keep uncommitted keys in memory is less reliable but faster and easier. If you are going to store state in DB, then we need to deal with rollback problems and more error handling code (update that DB has chance to fail too). And we need to think how to recover state from that when KSM restarted, is it necessary or not to recover uncommitted keys in this DB? It will be hard because we don't know which stage this key has been processed to.
> Ozone : improve error handling for putkey operation
> ---------------------------------------------------
>
> Key: HDFS-11886
> URL: https://issues.apache.org/jira/browse/HDFS-11886
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Reporter: Chen Liang
> Attachments: design-notes-putkey.pdf
>
>
> Ozone's putKey operations involve a couple steps:
> 1. KSM calls allocateBlock to SCM, writes this info to KSM's local metastore
> 2. allocatedBlock gets returned to client, client checks to see if container needs to be created on datanode, if yes, create the container
> 3. writes the data to container.
> it is possible that 1 succeeded, but 2 or 3 failed, in this case there will be an entry in KSM's local metastore, but the key is actually nowhere to be found. We need to revert 1 is 2 or 3 failed in this case.
> To resolve this, we need at least two things to be implemented first.
> 1. We need deleteKey() to be added KSM first.
> 2. We also need container reports to be implemented first such that SCM can track whether the container is actually added.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org