You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/10/30 01:02:59 UTC
[jira] Created: (HBASE-1944) Add a "is synced" attribute to HTD
Add a "is synced" attribute to HTD
----------------------------------
Key: HBASE-1944
URL: https://issues.apache.org/jira/browse/HBASE-1944
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.21.0
At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771725#action_12771725 ]
ryan rawson commented on HBASE-1944:
------------------------------------
this is absolutely necessary to achieve high performance and data safety under the presence of hdfs-265, since the best case scenario will be about 1000 fsync/second.
I am not super fond of the name, since synced suggests perhaps that the data isnt written to disk or something, which is NOT true.
Perhaps PER_TX_FLUSH :-)
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771756#action_12771756 ]
stack commented on HBASE-1944:
------------------------------
Yeah, any name is better than IS_SYNC (ain't that the name of a boyband?).... And yeah, it should have FLUSH rather than SYNC since sync don't work yet?
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772041#action_12772041 ]
Jean-Daniel Cryans commented on HBASE-1944:
-------------------------------------------
Ryan told me that the name may be good but it should be false by default then and the check in HRS should change.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1944) Add a "deferred log flush" attribute
to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1944.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed with Stack's suggestion.
> Add a "deferred log flush" attribute to HTD
> -------------------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Attachment: HBASE-1944.patch
This patch adds the new HTD.IS_SYNCED attribute and in HRS I refactored the code that handles the syncs in a method that also checks if the table has to be synced.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772067#action_12772067 ]
ryan rawson commented on HBASE-1944:
------------------------------------
looking good, +1
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "deferred log flush" attribute
to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Summary: Add a "deferred log flush" attribute to HTD (was: Add a "deferrend log flush" attribute to HTD)
> Add a "deferred log flush" attribute to HTD
> -------------------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Attachment: HBASE-1944-v3.patch
Fixed logic.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "deferred log flush" attribute
to HTD
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772731#action_12772731 ]
stack commented on HBASE-1944:
------------------------------
Patch looks good. Should cache the boolean though rather than look it up each time; those cost if you look in profiler.
> Add a "deferred log flush" attribute to HTD
> -------------------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Attachment: HBASE-1944-v4.patch
Better yet, some refactoring to get rid of checking region's null.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771766#action_12771766 ]
Jean-Daniel Cryans commented on HBASE-1944:
-------------------------------------------
It's nsync Stack! I guess you're too old ;)
Sorry I confused sync and flush, i guess that variable isn't volatile in my mind and I still see an old value.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771729#action_12771729 ]
Jean-Daniel Cryans commented on HBASE-1944:
-------------------------------------------
Yeah IS_SYNCED seems to imply that you don't write in the WAL but you do it's just that you don't make sure it's synced. Current makes sense for a dev but probably not for user.
So PER_TX_SYNC? DELAYED_SYNC? DEFERRED_SYNC?
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Attachment: HBASE-1944-v2.patch
New patch with new name, now it's called DEFERRED_LOG_FLUSH.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1944) Add a "deferrend log flush" attribute
to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1944:
--------------------------------------
Summary: Add a "deferrend log flush" attribute to HTD (was: Add a "is synced" attribute to HTD)
> Add a "deferrend log flush" attribute to HTD
> --------------------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944-v2.patch, HBASE-1944-v3.patch, HBASE-1944-v4.patch, HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771735#action_12771735 ]
Jean-Daniel Cryans commented on HBASE-1944:
-------------------------------------------
I just tested this patch on a cluster, I get 1300 incrCol/sec with a single threaded client on a table that doesn't sync. I killed -9 the RS a couple of times and even with optional sync every 100ms I lose only 1-2 edits.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1944) Add a "is synced" attribute to HTD
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771758#action_12771758 ]
stack commented on HBASE-1944:
------------------------------
Otherwise the patch is good.
> Add a "is synced" attribute to HTD
> ----------------------------------
>
> Key: HBASE-1944
> URL: https://issues.apache.org/jira/browse/HBASE-1944
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1944.patch
>
>
> At SU we have a case where we'd like to be able to, by default, sync every appended edit but for some tables have the control to disable their sync. You could then rely on other tables like .META. to do syncs, the optional log syncer timer down in LogSyncer and user tables that are always synced. For example, set a 10ms timer on the log syncer so that in the worse case you lose 10ms of non-catalog edits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.