You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2012/05/08 03:05:48 UTC
[jira] [Created] (HBASE-5954) Allow proper fsync support for HBase
Lars Hofhansl created HBASE-5954:
------------------------------------
Summary: Allow proper fsync support for HBase
Key: HBASE-5954
URL: https://issues.apache.org/jira/browse/HBASE-5954
Project: HBase
Issue Type: Improvement
Reporter: Lars Hofhansl
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk.txt
Here's a patch against HBase-trunk matching the v2-trunk patch in HDFS-744.
Again, this is not yet configurable and just for testing.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk-v2.txt
Patch that makes durable sync configurable (separately for WAL and HFiles).
Should probably allow all file creation to be durable (losing tableinfo or regioninfo files would be bad too).
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503321#comment-13503321 ]
Lars Hofhansl edited comment on HBASE-5954 at 11/24/12 5:59 AM:
----------------------------------------------------------------
While I'm at it, I'd like to add deferred log flush as a per operation option as well.
So we'd have:
# No WAL update (for the existing option to disable writing to the WAL)
# deferred log flush
# flush WAL (default)
# sync WAL
If there are multiple mutations in a batch the highest option will be used for the entire batch.
Since these options cannot be combined in a sensible way. This is best represented by an enum.
In 0.96, we can break wire compatibility. I'll just add a protobuf enum, and remove the current writeToWal bit. The actual logic will be put in the hadoop-2 shim module.
In 0.94 this is a bit more tricky. Both in terms of doing this in a wire compatible way and in terms of being forced to use reflection to detect and use Hadoop-2 vs Hadoop-1. Leaning towards only doing this in 0.96, even though I really wanted this in 0.94.
Comments?
was (Author: lhofhansl):
While I'm add it, I'd like to add deferred log flush as a per operation option as well.
So we'd have:
# No WAL update (for the existing option to disable writing to the WAL)
# deferred log flush
# flush WAL (default)
# sync WAL
If there are multiple mutations in a batch the highest option will be used for the entire batch.
Since these options cannot be combined in a sensible way. This is best represented by an enum.
In 0.96, we can break wire compatibility. I'll just add a protobuf enum, and remove the current writeToWal bit. The actual logic will be put in the hadoop-2 shim module.
In 0.94 this is a bit more tricky. Both in terms of doing this in a wire compatible way and in terms of being forced to use reflection to detect and use Hadoop-2 vs Hadoop-1. Leaning towards only doing this in 0.96, even though I really wanted this in 0.94.
Comments?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419738#comment-13419738 ]
Lars Hofhansl edited comment on HBASE-5954 at 7/21/12 4:01 AM:
---------------------------------------------------------------
I think the API going multiple ways (these are not mutually exclusive):
# hsync for HFiles (would guard compactions, etc, very lightweight), enabled with a config option (default on I think)
# hsync all WAL edits (very expensive, but would not require client changes), enabled with a config option (default off)
# hsync for tables or column families for HFiles (configured in the table/column descriptor)
# hsync for tables or column families for the WAL (configured in the table/column descriptor)
# WAL hsync per Put. Gives control to the application. A batch put would hsync the WAL if at least one Put in the batch was market with hsync. What about deletes? In 0.94 they are not batched; could it at the end of operation there.
# WAL hsync per RPC. Could send flag with the RPC from the client. I.e. HTable would have a Put(List<Put> puts, boolean hsync) method
# HTable.hsync. Client calls this when WAL must be sync'ed. Most flexible, but incurs an extra RPC to the RegionServer just to force the hsync.
Comments welcome.
Edit: Forgot some options.
was (Author: lhofhansl):
I think the API going multiple ways (these are not mutually exclusive):
# hsync for HFiles (would guard compactions, etc, very lightweight), enabled with a config option (default on I think)
# hsync all WAL edits (very expensive, but would not require client changes), enabled with a config option (default off)
# sync per Put. Gives control to the application. A batch put would hsync the WAL if at least one Put in the batch was market with hsync. What about deletes? In 0.94 they are not batched; could it at the end of operation there.
# Per RPC. Could send flag with the RPC from the client. I.e. HTable would have a Put(List<Put> puts, boolean hsync) method
# HTable.hsync. Client calls this when data must be sync'ed. Most flexible, but incurs an extra RPC to the RegionServer just to force the hsync.
Comments welcome.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423272#comment-13423272 ]
Andrew Purtell commented on HBASE-5954:
---------------------------------------
bq. a flag per Put/Delete/Increment/Append/etc, would be best option
Makes sense, since both you and Todd got here after giving it consideration.
bq. On the other hand I do not want to have completely diverging implementations for this for 0.94 and 0.96
I think we have to take this pain, a reflection based strategy in 0.94 and a module based strategy in 0.96. I this case I'd judge it worth it. But that's going to be a high bar for other things.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503326#comment-13503326 ]
liang xie commented on HBASE-5954:
----------------------------------
sound good for me. hope more guys have an eye on it
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk-v3.txt
Patch that works correctly for WAL and HFile sync. (This is the one I used for above performance tests)
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475338#comment-13475338 ]
Kan Zhang commented on HBASE-5954:
----------------------------------
It's true hsync is better than hflush in terms of persisting to disk. However, IMO, what's important for apps is whether it is safe to discard data from their buffers when acknowledged (without having to worry about retrying the writes in case of cluster failures). The current hsync doesn't give you that assurance, while both pre and after HDFS-265 hflush implementations do with respect to the semantics they support.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk-v6.txt
Previous patch accidentally always enable WAL sync if HDFS-744 is present.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503321#comment-13503321 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
While I'm add it, I'd like to add deferred log flush as a per operation option as well.
So we'd have:
# No WAL update (for the existing option to disable writing to the WAL)
# deferred log flush
# flush WAL (default)
# sync WAL
If there are multiple mutations in a batch the highest option will be used for the entire batch.
Since these options cannot be combined in a sensible way. This is best represented by an enum.
In 0.96, we can break wire compatibility. I'll just add a protobuf enum, and remove the current writeToWal bit. The actual logic will be put in the hadoop-2 shim module.
In 0.94 this is a bit more tricky. Both in terms of doing this in a wire compatible way and in terms of being forced to use reflection to detect and use Hadoop-2 vs Hadoop-1. Leaning towards only doing this in 0.96, even though I really wanted this in 0.94.
Comments?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286056#comment-13286056 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
HDFS-744 was committed. Time to finalize this patch.
Todd suggested at the hack-a-thon to have a sync option per column family rather than per HBase cluster.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503461#comment-13503461 ]
Lars Hofhansl edited comment on HBASE-5954 at 11/25/12 5:41 AM:
----------------------------------------------------------------
In 0.96 Put.java, Delete.java, and Increment.java still have readFields() and write() methods from writable.
Were they left over by accident? I assume those can be removed now?
was (Author: lhofhansl):
In 0.96 Put.java, Delete.java, and Increment.java still have readFields() and write() methods from writable.
Were the left over by accident? I assume those can be removed now?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Description: Unscheduling from both 0.94 and 0.96 until HDFS-3979 is committed.
Fix Version/s: (was: 0.94.3)
(was: 0.96.0)
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
> Unscheduling from both 0.94 and 0.96 until HDFS-3979 is committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276059#comment-13276059 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
This is only partially true. It is true for the sync packet stuff that I added, because this does not require closing the block.
If the block needs to be closed (which causes it to be fsync'ed) it is done after the ack from the downstream DN and before the ack to the upstream DN. Here in that case the fsyncs are serial. Looking at the code, that part seems hard to change.
Good news is: HLog files are smaller than a DFS block, so for HBase we never run into the 2nd issue.
Semi bad news: HFiles also need to fsync'ed at least on block close, so here we'd see the issue. But since HFiles are written asynchronously it should be OK.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423201#comment-13423201 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Thinking more about HTable#hsync, I think that would hard to make useful to an application. The application would need to do know which RegionServers to hsync the WAL on (unless we want to flush for a Table, which means all RegionServer hosting regions for this table need to hsync their WAL, and that does not seem to be useful).
A similar argument goes for the RPCs (which will be split to multiple RegionServers).
So #6 and #7 are out I think.
I think Todd was right after all (just took me a long time to come around to it), a flag per Put/Delete/Increment/Append/etc, would be best option.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426966#comment-13426966 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Sigh... The WAL files would still need to be sync'ed upon blockclose. Since they mix data from different stores, there's no telling ahead of time. Which leads to:
# global hsync option upon close block for HFiles and Hlogs (makes no sense to sync HLogs but not HFiles or vice versa)
# global WAL edit hsync option
# hsync CF's WAL edits
# WAL hsync per Put/Delete/Append/Increment/etc
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276348#comment-13276348 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Ok... Now with HFile sync.
With HFile sync: ~20s
Both WAL and HFile sync: ~35s
With HFile sync enabled, I seek occasional IO spikes on top of the constant WAL IO.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503613#comment-13503613 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Filed HBASE-7215.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419738#comment-13419738 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I think the API going multiple ways (these are not mutually exclusive):
# hsync for HFiles (would guard compactions, etc, very lightweight), enabled with a config option (default on I think)
# hsync all WAL edits (very expensive, but would not require client changes), enabled with a config option (default off)
# sync per Put. Gives control to the application. A batch put would hsync the WAL if at least one Put in the batch was market with hsync. What about deletes? In 0.94 they are not batched; could it at the end of operation there.
# Per RPC. Could send flag with the RPC from the client. I.e. HTable would have a Put(List<Put> puts, boolean hsync) method
# HTable.hsync. Client calls this when data must be sync'ed. Most flexible, but incurs an extra RPC to the RegionServer just to force the hsync.
Comments welcome.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270116#comment-13270116 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Over I HDFS-744 I propose changes to HDFS to allow fsync on files.
Without that both WAL and HFiles (resulting from compactions/flushes) are not guaranteed to be on disk at the DFS replicas.
The HDFS-744 is becoming big, so I'm not sure how the hadoop folks will receive it. I any case, this issues tracks the necessary changes on the HBase side.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474827#comment-13474827 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Hi Luke,
you mean hflush in hadoop 2 less durable than hflush in hadoop 1. hsync is still better (even when it is not on the synchronous path, so there's a little gap where a client was told that everything is on disk when in fact it isn't).
Filed HDFS-3979 (you know that :), just for the benefit of others reading here), which needs some testing to be committed.
Since the hsync code is only in hadoop 2.1.0+ we'd need a new shim here for that version (or reflect the sh*t out of it).
I'm still happy to commit this to 0.94.x.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276950#comment-13276950 ]
Luke Lu commented on HBASE-5954:
--------------------------------
Thanks for the numbers, Lars! Are you using ext3? I wonder what the numbers would look like if you enable barrier=1 in the mount options or just use ext4 (with barrier turned on by default). If the underlying fs doesn't do barrier, the result is somewhat meaningless (you might as well use hflush).
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503557#comment-13503557 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I'd vote for removing them. If we keep these we have failed with wire compatibility and all the protofuf stuff was for nothing.
Put/Delete is still used as writable at least in these cases:
* IdentityTableReduce.java
* MultiPut.java
* HRegionServer.checkAndMutate
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-5954:
------------------------------------
Assignee: Lars Hofhansl
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503528#comment-13503528 ]
Jimmy Xiang commented on HBASE-5954:
------------------------------------
Those methods should be removed. But old applications may not compile any more if someone happens to use them. Are we ok with that?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475475#comment-13475475 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Wait, in all cases the data was flushed to the DN. hsync and hflush give exactly the same guarantees here (it's the same code path).
If hsync is broken here, so it hflush.
What can currently happen (past HDFS-265) for both hflush and hsync is that the data is still in the DN buffers (not in the OS buffers - in case of hflush, or on disk - in case of hsync)... Unless I seriously misunderstand.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475491#comment-13475491 ]
Kan Zhang commented on HBASE-5954:
----------------------------------
Hi Lars, I think you did misunderstand what I said, esp. on the part "... with respect to the semantics they support." :) As you know, post HDFS-265 hflush implements API3 and when hflush returns data is guaranteed to be in the DN buffers on all replica nodes, which is what API3 promises and client can't expect anything more than that by calling hflush. But client would expect data to hit disk by calling hsync, which is not guaranteed to happen in the current implementation. The expected semantics are simply different for hflush and hsync.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401923#comment-13401923 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I almost feel like the HFile sync should be enabled by default. This ensures that after an HFile is written and closed all of its blocks are sync'ed to disk. The overhead of this is "fairly" minimal.
The edit-by-edit sync for the WAL is definitely optional.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5954) Allow proper fsync
support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270116#comment-13270116 ]
Lars Hofhansl edited comment on HBASE-5954 at 5/8/12 1:06 AM:
--------------------------------------------------------------
Over in HDFS-744 I propose changes to HDFS to allow fsync on files.
Without that both WAL and HFiles (resulting from compactions/flushes) are not guaranteed to be on disk at the DFS replicas.
The HDFS-744 is becoming big, so I'm not sure how the hadoop folks will receive it. I any case, this issues tracks the necessary changes on the HBase side.
was (Author: lhofhansl):
Over I HDFS-744 I propose changes to HDFS to allow fsync on files.
Without that both WAL and HFiles (resulting from compactions/flushes) are not guaranteed to be on disk at the DFS replicas.
The HDFS-744 is becoming big, so I'm not sure how the hadoop folks will receive it. I any case, this issues tracks the necessary changes on the HBase side.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Fix Version/s: 0.94.2
0.96.0
Would be nice to be able to enable this (when using Hadoop-2).
Not sure about the 0.94 target, but since the change is not disruptive to HBase and purely optional it would be a good addition.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Description: (was: Unscheduling from both 0.94 and 0.96 until HDFS-3979 is committed.)
Unscheduling from both 0.94 and 0.96 until HDFS-3979 is committed.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk-v4.txt
Patch matching latest patch on HDFS-744. Will probably change again soon.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475511#comment-13475511 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I see, you're right, the expectations are different. Let's fix HDFS-3979 already :)
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503558#comment-13503558 ]
Andrew Purtell commented on HBASE-5954:
---------------------------------------
bq. I'd vote for removing them. If we keep these we have failed with wire compatibility and all the protofuf stuff was for nothing.
+1
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475114#comment-13475114 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I disagree, having the signal out for the DNs to sync durably to disk now (even if they only do as soon as they get to it) is better than having an indefinite amount in which the data on only in volatile memory (which is as good as we can get with only hflush).
But let's not harp this point... We all agree that it needs to be fixed.
I could use some help testing HDFS-3979. I believe the pipeline tests cover the failure scenarios (which are the same whether we flush/sync or not), the test needed there is verifying that the fsync is on the synchronous path of the client.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426949#comment-13426949 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Created HBASE-6492.
Since I am interested in having this in 0.94 I'll start with the reflection based approach (but still in trunk for HadoopQA).
This is what I am going to do:
# global HFiles hsync option upon close block (this will also apply sync'ing the WAL on close)
# global WAL edit hsync opion
# hsync CF's HFiles
# hsync CF's WAL edits
# WAL hsync per Put/Delete/Append/Increment/etc
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273840#comment-13273840 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Btw. the last attached patch also includes a changed pom.xml file. -Dhadoop.profile=30 can be used to build HBase against hadoop 3.0.0 snapshots.
In order to build hadoop-trunk and hbase-trunk just do:
# mvn -Pnative -Pdist -Dtar -DskipTests clean install
# mvn clean install -Dhadoop.profile=30 -DskipTests
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286923#comment-13286923 ]
Todd Lipcon commented on HBASE-5954:
------------------------------------
Even per-put would be pretty nice, actually -- I can imagine some applications where most updates are "unimportant" but the occasional one should be hard-persisted.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474886#comment-13474886 ]
Luke Lu commented on HBASE-5954:
--------------------------------
bq. hsync is still better (even when it is not on the synchronous path, so there's a little gap where a client was told that everything is on disk when in fact it isn't).
No. It (hsync) is worse in the sense that there will be data loss (i.e., inconsistency: acked writes missing) if people bounce hdfs, while hbase is still writing. Bouncing hdfs happens much more often in practice than actual machine/pdu failures. To put it more strongly, hsync in the current hadoop 2 is semantically wrong.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494390#comment-13494390 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I'm planning to pick this up again, soon.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423275#comment-13423275 ]
Andrew Purtell commented on HBASE-5954:
---------------------------------------
More on my comment above. We have two options: we can start breaking out reflection into modules in 0.94 too, or save all of that for 0.96. I don't have a strong opinion but if I had to make a choice the refactoring should be in next major version / trunk / currently unstable.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277206#comment-13277206 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
I'm using ext4 with default mount options (indeed the numbers would be useless otherwise)
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420348#comment-13420348 ]
stack commented on HBASE-5954:
------------------------------
bq. Would be nice to be able to enable this (when using Hadoop-2).
Can we do it in the hadoop2 compatibility module?
I think its fine its optionally off in 0.94 and that in 0.96 its on by default.
I like your list. Would think a CF/Table option and the HTable#hsync the more important options to offer (though on the latter, perhaps a Put+hsync would be better given extra rpc).
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276333#comment-13276333 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Finally some numbers for "PerformanceEvaluation --nomapred --rows=100000 --presplit=3 randomWrite 10"
Without WAL/HFile sync: ~18s
With WAL sync: ~34s
With WAL sync on, I see a constant ~70mb/s write load. Without WAL sync I see a few spikes of far higher IO load.
This is all on a single machine with HBase in fully distributed mode on top of a pseudo distributed HDFS.
Note that my patch does not yet do HFile sync'ing correctly.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423241#comment-13423241 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
Another question: HBase 0.96 now has the hadoop{1|2}-compat projects.
On the other hand I do not want to have completely diverging implementations for this for 0.94 and 0.96 (which would mean to use reflection in both branches). Any thoughts on that?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503461#comment-13503461 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
In 0.96 Put.java, Delete.java, and Increment.java still have readFields() and write() methods from writable.
Were the left over by accident? I assume those can be removed now?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288916#comment-13288916 ]
Lars Hofhansl commented on HBASE-5954:
--------------------------------------
HDFS-744 is in Hadoop-2.0 now.
I'd like to make a clean patch with only the cluster wide option first. That way, this can be enabled and allow HBase to still be a nice HDFS citizen.
Then we can add other options (per CF, per Put, etc).
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: 5954-trunk-hdfs-trunk-v5.txt
Minor change, matching latest HDFS-744 patch.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5954:
---------------------------------
Attachment: hbase-hdfs-744.txt
Here's what I have. Not configurable.
With this and HDFS-744, it should be possible to test this out.
I will be doing that.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Attachments: hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462584#comment-13462584 ]
Luke Lu commented on HBASE-5954:
--------------------------------
Hi Lars,
We just noticed that HDFS-744 did not implement the correct hsync semantics (mostly due to HDFS-265) so that the hsync is slower AND (arguably) less durable than hflush in Hadoop 1.x.
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for
HBase
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493707#comment-13493707 ]
liang xie commented on HBASE-5954:
----------------------------------
Hi [~lhofhansl],the HDFS-3979 has been committed, so maybe we can have a more clear target/fix version plan on HBASE-5954 now, right?
> Allow proper fsync support for HBase
> ------------------------------------
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira