You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2009/04/24 23:06:30 UTC
[jira] Created: (HADOOP-5744) Revisit append
Revisit append
--------------
Key: HADOOP-5744
URL: https://issues.apache.org/jira/browse/HADOOP-5744
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.20.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.21.0
HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5744:
----------------------------------
Attachment: AppendSpec.pdf
Attached is the first draft of the append specification. Please comment if this specification meets your requirements.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710530#action_12710530 ]
Jim Kellerman commented on HADOOP-5744:
---------------------------------------
> Hairong Kuang added a comment - 18/May/09 03:53 PM
> Note that the current implementation in 0.20 does not provide the second guarantee described above.
> A byte saw by a reader become invisible when the replica it read from fails. Even when the replica that
> a reader is reading from is alive, the recovery of an error caused by another replica in the pipeline may
> remove a byte saw by a reader temporary from dfs but recover the byte later. A removed byte may never
> get recovered if the writer dies or the writer does not have this byte any more.
Yes, this is what HADOOP-4379 is trying to address with Doug Cutting's comment:
> Doug Cutting added a comment - 26/Jan/09 11:20 AM
>
>I think it would be better to add a SequenceFile.Writer method that flushes and calls
> FSDataOutputStream.sync. The problem is that SequenceFile.Writer#sync() should
> probably be called flush(), not sync(), but that'd be hard to change. So perhaps we
> should add a SequenceFile.Writer#syncFs() method?
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709624#action_12709624 ]
Hairong Kuang commented on HADOOP-5744:
---------------------------------------
Flush needs to make data visible before a block is completed. This has changed a lot of assumptions in HDFS. Previous append work has put tremendous amount of great work and set up a foundation for improvement. However there are issues that seemed trivial initially but it turned out that they needed a thorough design. HADOOP-4379, 4663, 5027, 5133, and 4692 etc. were filed as bugs but are indeed caused by a lack of design. This issue aims at a design which solves them all. It's targeted for 0.21.
If no hflush is called, dfs client pushes data to datanodes when a packet is filled up. The packet size is configurable with a default size of 64K.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709961#action_12709961 ]
stack commented on HADOOP-5744:
-------------------------------
Thanks Hairong. Any comment on the unqualified 'delay' made mention of in the spec? What kind of time scale should we be thinking in? If it is an 'hour' as I believe is the current default for lease timeout, hbase-like applications will have given up the wait long before. If its in the tens of seconds -- ~30 seconds -- as I'm currently seeing in HBASE-4379 patches I'm helping Dhruba test, then that is still too long but tolerable in a first release, as long as it can be improved on over time ("Instantaneous" would be the ideal). Thanks.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710542#action_12710542 ]
Hairong Kuang commented on HADOOP-5744:
---------------------------------------
Note this jira no longer uses "sync", instead we use hflush.
The spec posted in this jira aims at API 3.
> another process knows that the writer has crashed and needs to be able to read all the data up to the last sync()
As I said, this spec guarantees that another process read all the data up to the last flush without the need to close this file. We also guarantee that the flushed data won't be removed as a result of lease recovery when the file is finally closed.
> recover the lease (immediately)
If another process can read all the flushed data without the file being closed, do you still need to recover the lease?
> this is what HADOOP-4379 is trying to address with Doug Cutting's comment.
I might be wrong. But I do not think HADOOP-4379 addresses the problem that I raised. HADOOP-4379 tries to make flushed data visible to a new reader.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709612#action_12709612 ]
stack commented on HADOOP-5744:
-------------------------------
A few comments and questions:
Document is without context. It is undated and has no author. In particular, I'd think there'd be some positioning of this document in relation to previous append work (Is hadoop-4379, etc., dead? Are we starting over as the name of this issue portends. If starting over, is 0.21 fix version optimistic?).
What kind of 'delay' are we talking about here do you think in the section Read, item #4 ? "If a writer/appender does not call hflush, a reader can still progressively see previously written data with a delay"
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-5744:
----------------------------------
Attachment: appendDesign.pdf
A draft of the design document is attached. Please comment. Nicholas will post a test plan.
I will be on vacation in the next few weeks. Please be not surprised if my response is delayed.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: appendDesign.pdf, AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710527#action_12710527 ]
Jim Kellerman commented on HADOOP-5744:
---------------------------------------
The problem we need to solve is:
- process writing a file crashes after doing some sync() operations
- another process knows that the writer has crashed and needs to
-- recover the lease (immediately)
-- be able to read all the data up to the last sync()
Of the APIs described in the following email, from Sanjay below, APIs 1-2 are are inadequate, API3 is ok provided HDFS does not fail. API4 works if the datanode(s) fail but not if the machine crashes. Only API5 will guarantee that we can read the data that has been sync'd.
> From: Sanjay Radia [mailto:sradia@yahoo-inc.com]
> Sent: Thursday, May 14, 2009 1:46 PM
> To: Jim Kellerman (POWERSET)
> Cc: Michael Stack; Chad Walters; Dhruba Borthakur; Sameer Paranjpye; Hairong Kuang; Robert Chansler
> Subject: Re: Append, flush, sync write and HBase
>
>> On May 13, 2009, at 1:20 PM, Jim Kellerman (POWERSET) wrote:
>>
>> What we need are two things:
>>
>> 1. When we call sync() we want to be assured that any buffered data can be read by another process
>
> Actually I wanted to have a larger discussion to understand your
> current and future requirements on append/flush/sync also on latency
> of HDFS. I am trying to document current and future
> requirements. Which is why I wanted to do a quick chat on the phone. I
> will try this via email for now.
>
> BTW Hairong @ Y! is driving the re-implementation of append. See
> HADOOP-5744. She has defined the semantics she is considering. Please
> comment whether you agree or disagree.
>
> We are also looking at variation on semantics that may have lower
> latencies and lesser guarantees. We would like to get your initial
> feedback. Eventually we will update the Jira when we have semantics
> and apis better formulated.
>
> Below is a list of APIs/semantics variations we are considering.
> Which ones do you absolutely needed for HBase in the short term and
> which ones may be useful to HBase in the longer term.
>
> API1: flushes out from the address space of client into the socket to the data nodes.
>
> On the return of the call there is no guarantee that that data is
> out of the underlying node and no guarantee of having reached a
> DN. Readers will see this data soon if there are no failures.
>
> For example, I suspect Scribe and chukwa will like the lower
> latency of this API and are prepared to loose some records
> occasionally in case of failures. Clearly a journal will not find
> this api acceptable.
>
> API2: flushes out to at lease one data node and receives an ack.
>
> New readers will eventually see the data
>
> API3: flushes out to all replicas of the block. The data is in the buffers of the DNs but not on the DN's OS buffers
>
> New readers will see the data after the call has returned. (Hadoop
> 5744 calls API3 hflush for now).
>
> API4: flushes out to all replicas and all replicas DNs have done a posix fflush equivalent - ie data is out the under lying OS file system of the DNs
>
> API5: flushes out to all replicas and all repliacs have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache).
>
> Does the HBase edits journal require API 3, 4 or 5?
>
> What are your latency requirements for the write operation. For
> example can you tolerate occasional larger latency for the
> fflush/fsycn operation?
>
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710608#action_12710608 ]
stack commented on HADOOP-5744:
-------------------------------
@ Hairong
.bq ..do you still need to recover the lease?
No. If readers will see data near immediately after its hflushed, there is no need to do the append, wait-on-lease, close, and then new open sequence.
.bq I might be wrong. But I do not think HADOOP-4379 addresses the problem that I raised. HADOOP-4379 tries to make flushed data visible to a new reader.
To be clear, this is our use case. When we notice a process has died, a different process opens a new reader on the file the dead process had been writing (Later, in your '18/May/09 05:03 PM' comment, you say new readers will see up to the last hflush - - just making sure you understand our usage).
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710547#action_12710547 ]
Hairong Kuang commented on HADOOP-5744:
---------------------------------------
Jim/Stack, thanks for your comments. This makes us to understand our users' requirements better.
> It doesn't have to be closed as long the reader is able to go all the ways up to the last sync made by the writer before crash.
Yes, all flushed data are guaranteed to be read by a new reader. I also want you to be aware that the reader may read beyond the flushed data.
> The file may have more bytes than when it was previously read. This is a norm case. Will this be an issue to hbase? How would this circumstance arise?
Our idea is that not all the data received by a datanode are visible to a user. For example. if a dfs application calls flush, the data left the client machine and are being pushed to all datanodes, but the application dies before it receives the flush ack. This flush is considered as a failure. The flushed data may have reached none, one, or all datanodes. So the data is not guaranteed to be visible to a reader. But the lease recovery MAY still keep the data in the file if the data have reached one or all datanodes.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710541#action_12710541 ]
stack commented on HADOOP-5744:
-------------------------------
@ Hairong
.bq My question is that why the file needs to be closed before it is read.
It doesn't have to be closed as long the reader is able to go all the ways up to the last sync made by the writer before crash.
.bq Is it OK for your client to trigger the close of the file but does not wait for it to close?
Yes, as long as the reader is able to go all the ways up to the last sync made....
.bq (1) may have more bytes than when it was previously read. This is a norm case. Will this be an issue to hbase?
How would this circumstance arise?
.bq Note that the current implementation in 0.20 does not provide the second guarantee described above.
In my testing of hadoop-4379, i've only been killing the writer application. I should play with killing the writer application AND the local datanode.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5744) Revisit append
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703448#action_12703448 ]
Konstantin Boudnik edited comment on HADOOP-5744 at 4/27/09 5:00 PM:
---------------------------------------------------------------------
Hairong,
I've a couple of comments about semantic of clause of the document:
1. DFS provides "best effort durability" to data in an unclosed file:
- ... If no replicas of a block being written is available, write fails
While it really sounds like a best effort it has some implications from the stand point of deterministic testing of the feature. In this test case, shall we always expect the negative result of the test and dismiss occasional positive results as non-significant? Shall we apply some kind of heuristic to judge the test case's execution results?
2. DFS provides "strong durability" to data in a closed file:
- ... However, file close doesn't guarantee that data has hit the disk, If data does not hit the disk, restarting DataNode may cause the loss of the data.
While I totally understand the somewhat non-deterministic behavior of this feature (e.g. a hardware failure might happen in the very instant between a file's close invocation and actual data storage), I still have a concern or two about the testing perspectives of it. As I understand, synchronous close() call returns when all DNs have reported the successful flash for their respective blocks of the file. However, and this is where non-determinism kicks in, a particular DN's OS might be slow in flushing its own IO buffers and writing the data to disk, and where the actual data loss could happen in case of sudden restart of the DN. However, from the testing perspective it creates a certain issues like the follows:
- let's assume that a test client appends a file in DFS
- at some point it calls close() and waits for its return
- let's suppose the test client runs in a distributed harness where another piece of the test can simulate a failure on a DN, where a replica of the file is located. And the timing of the failure is happen to be right after this very DN reported the success of the flush() call. E.g. our assumed test harness can guarantee that data hasn't hit the disk. However, our test client will see a successful return from the call() method and report the test success respectively. So, I'm not sure how to deal with this situation too.
was (Author: cos):
Hairong,
I've a couple of comments about semantic of clause of the document:
1. DFS provides "best effort durability" to data in an unclosed file:
- ... If no replicas of a block being written is available, write fails
While it really sounds like a best effort it has some implications from the stand point of deterministic testing of the feature. In this test case, shall we always expect the negative result of the test and dismiss occasional positive results as non-significant? Shall we apply some kind of heuristic to judge the test case's execution results?
2. DFS provides "strong durability" to data in a closed file:
- ... However, file close doesn't guarantee that data has hit the disk, If data does not hit the disk, restarting DataNode may cause the loss of the data.
While I totally understand the somewhat non-deterministic behavior of this feature (e.g. a hardware failure might happen in the very instant between a file's close invocation and actual data storage), I still have a concern or two about the testing perspectives of it. As I understand, synchronous close() call returns when all DNs have reported the successful flash for their respective blocks of the file. However, and this is where non-determinism kicks in, a particular DN's OS might be slow in flushing its own IO buffers and writing the data to disk, and where the actual data loss could happen in case of sudden restart of the DN. However, from the testing perspective it creates a certain issues like the follows:
- let's assume that a test client appends a file in DFS
- at some point it calls close() and waits for its return
- let's suppose the test client runs in a distributed harness where another piece of the test can simulate a failure on a DN, where a replica of the file is located. And the timing of the failure is happen to be right after this very DN reported the success of the flush() call. E.g. our assumed test harness can guarantee that data hasn't hit the disk. However, our test client will see a successful return from the call() method and report the test success respective. So, I'm not sure how to deal with this situation too.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703448#action_12703448 ]
Konstantin Boudnik commented on HADOOP-5744:
--------------------------------------------
Hairong,
I've a couple of comments about semantic of clause of the document:
1. DFS provides "best effort durability" to data in an unclosed file:
- ... If no replicas of a block being written is available, write fails
While it really sounds like a best effort it has some implications from the stand point of deterministic testing of the feature. In this test case, shall we always expect the negative result of the test and dismiss occasional positive results as non-significant? Shall we apply some kind of heuristic to judge the test case's execution results?
2. DFS provides "strong durability" to data in a closed file:
- ... However, file close doesn't guarantee that data has hit the disk, If data does not hit the disk, restarting DataNode may cause the loss of the data.
While I totally understand the somewhat non-deterministic behavior of this feature (e.g. a hardware failure might happen in the very instant between a file's close invocation and actual data storage), I still have a concern or two about the testing perspectives of it. As I understand, synchronous close() call returns when all DNs have reported the successful flash for their respective blocks of the file. However, and this is where non-determinism kicks in, a particular DN's OS might be slow in flushing its own IO buffers and writing the data to disk, and where the actual data loss could happen in case of sudden restart of the DN. However, from the testing perspective it creates a certain issues like the follows:
- let's assume that a test client appends a file in DFS
- at some point it calls close() and waits for its return
- let's suppose the test client runs in a distributed harness where another piece of the test can simulate a failure on a DN, where a replica of the file is located. And the timing of the failure is happen to be right after this very DN reported the success of the flush() call. E.g. our assumed test harness can guarantee that data hasn't hit the disk. However, our test client will see a successful return from the call() method and report the test success respective. So, I'm not sure how to deal with this situation too.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710524#action_12710524 ]
Hairong Kuang commented on HADOOP-5744:
---------------------------------------
Note that the current implementation in 0.20 does not provide the second guarantee described above. A byte saw by a reader become invisible when the replica it read from fails. Even when the replica that a reader is reading from is alive, the recovery of an error caused by another replica in the pipeline may remove a byte saw by a reader temporary from dfs but recover the byte later. A removed byte may never get recovered if the writer dies or the writer does not have this byte any more.
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5744) Revisit append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710523#action_12710523 ]
Hairong Kuang commented on HADOOP-5744:
---------------------------------------
Dhruba told me that hbase depended on a client calling of append to trigger the close of a file that lost its writer. Once the file is closed, the client reads the file and starts to work from the state defined in the closed file.
My question is that why the file needs to be closed before it is read. The read semantics defined in this jira guarantees that
(1) any hflushed data become visible to any new readers;
(2) Once a byte becomes visible to a reader, it continues to be visible to the reader except when all replicas containing the byte fail. This implies that a reader continues to see a byte it saw before even when the replica that it read from fails, during any error recovery, and after any error recovery as long as one replica containing the byte is available.
Is it OK for your client to trigger the close of the file but does not wait for it to close? The idea is to read the file and resume working before the file gets closed. When the file finally gets closed, the file
(1) may have more bytes than when it was previously read. This is a norm case. Will this be an issue to hbase?
(2) If all replicas went down during the period the file is triggered to close and the time it is closed, the file may end up with less bytes. This is a rare case. The default time for this period is 10 minutes so the chance of losing the visible bytes is very slim. Can hbase tolerate this?
> Revisit append
> --------------
>
> Key: HADOOP-5744
> URL: https://issues.apache.org/jira/browse/HADOOP-5744
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.20.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: AppendSpec.pdf
>
>
> HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.