You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/10/28 01:25:59 UTC
[jira] Created: (HBASE-1939) HLog group commit
HLog group commit
-----------------
Key: HBASE-1939
URL: https://issues.apache.org/jira/browse/HBASE-1939
Project: Hadoop HBase
Issue Type: New Feature
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.21.0
Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771542#action_12771542 ]
Raghu Angadi commented on HBASE-1939:
-------------------------------------
Thanks J-D.
I should have seen the update earlier... not sure how important 'force' flag is. Currently it might not work properly.
Say sync thread starts fs_sync() at time t0. At t1 a client appends and invokes a force-sync... I am assuming the contract is it should return after next fs_sync. But in the current implementation, it may not : when fs_sync() returns at t2, it sets this flag to false.. client's force-sync could return without a fs_sync.
I might implement it something like :
{code}
addToSyncQ(boolean force) {
lock()
if (force) {
this.forceSync = true;
}
//...
} {code}
Rest of the implementation remains same.. with forceSync being a simple boolean rather than an atomic.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770786#action_12770786 ]
Raghu Angadi commented on HBASE-1939:
-------------------------------------
>+ // Set force sync if force is true and forceSync is false
>+ forceSync.compareAndSet(!forceSync.get() && force, true);
Wouldn't this leave forceSync to 'true' all the time?
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771617#action_12771617 ]
stack commented on HBASE-1939:
------------------------------
Should the forceSync be volatile? Otherwise patch looks good (make it volatile on commit?)
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770771#action_12770771 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Doh missed that part.
BTW, I cannot have the class static because it relies on the instance of that HLog as it calls hsync and uses its close boolean.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1939) HLog group commit
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-1939:
---------------------------------------
Attachment: HBASE-1939-20.4.patch
Ported this patch back to version 0.20. Note that this refactoring incidentally fixed a bug on the tip with HLog.sync(). To summarize, the code under sync() should read
sync(); syncFs()
instead of
(syncFs) ? syncFs() : sync();
sync() flushes the SequenceFile.Writer buffer to DFSOutputStream. syncFs() flushes the output stream across the network to the datanodes. So the current implementation isn't flushing the buffers properly and still gets stuck on local storage until the SequenceFile.Writer buffer is full.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-20.4.patch, HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771700#action_12771700 ]
Raghu Angadi commented on HBASE-1939:
-------------------------------------
yes. should be same as accessing inside a synchronized section. We could consider atomic or volatile if findbugs does not see that.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771173#action_12771173 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Committed patch to trunk.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770769#action_12770769 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
I'm good with the comments.
There's already a method called sync in HLog that's used from HRS and such so I was searching for another name. I'll rename it to syncAppends maybe? Or syncOutputStream?
Have you tried it on a heavy write job?
Thx!
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770726#action_12770726 ]
Andrew Purtell commented on HBASE-1939:
---------------------------------------
How does this relate to HBASE-1416? I was looking at that. I think work on that issue and this one would clobber each other.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771710#action_12771710 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
I committed without the volatile.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770789#action_12770789 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Raghu,
Thanks for looking at this patch, I guess I am wrong but in the way you described.
!(true) && false => false == false then set it to true. That's ok you want to let it true
!(false) && false => false == false then set it to true. That's not ok we want it to stay false
!(false) && true => true != false then let it be false. That's not ok we want it to be true
!(true) && true => false != true then let it be true. That's ok you want it true
Also that value is reset in hsync to false.
So I indeed need to fix this thx!
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770736#action_12770736 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Damn forgot to mention that according to my many tests, single thread writing is ~1% slower but using 9 threads you get a 15-20% improvement.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1939.
---------------------------------------
Resolution: Fixed
Resolving since it wasn't the source of Hudson's instability.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771641#action_12771641 ]
stack commented on HBASE-1939:
------------------------------
@Raghu Will that ensure all threads see most recent change to forceSync?
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771634#action_12771634 ]
Raghu Angadi commented on HBASE-1939:
-------------------------------------
+1. It does not need to be volatile since it is always accessed under the same lock.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770765#action_12770765 ]
stack commented on HBASE-1939:
------------------------------
This can be final:
+ private LogSyncer logSyncerThread;
Can this be static inner class? Paass in flushinterval? And the atomicboolean close? Make it private?
+ public class LogSyncer extends Thread {
Why syncfs and not just sync? syncfs was name of the method in hdfs-200... its now called hflush?
Do the above on commit I'd say. They are small changes. Looks good to me J-D.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770770#action_12770770 ]
stack commented on HBASE-1939:
------------------------------
call it hflush? its the new style!
I did not try your patch. Will do after you commit.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771563#action_12771563 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Raghu,
Again thanks a lot for spending some time looking at my work.
So the background is that the catalog tables like .META. must never lose any edits so, even if we sync every 100 edits for some reason, when this catalog edit comes in we want to make sure it's safely registered.
And you are right, this is not the right place to set force as you pointed out.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1939:
--------------------------------------
Attachment: HBASE-1939-v2.patch
Second version of the patch against current trunk with Raghu's comments.
So I changed the condition so that it really sets to force if force == true and forceSync == false. In all the other cases it should stay to current value and hsync resets to false if the value was set to true. I also cleaned a LOG which has an unnecessary stack trace and removed now un-thrown exceptions.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1939:
--------------------------------------
Attachment: HBASE-1939.patch
Patch that implements the group commit but that also cleans up HDFS-200 era code. Passes all tests
- I remove LogFlusher in favor of the LogSyncer thread. This also fixes the fact that the optional log flush value was never use.
- I use an AtomicBoolean to force sync. This part was tricky because there is no message passing between the client thread and the LogSyncer.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1939.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed to trunk, please test your massive inserts and see how better (or worse, could happen) it is when syncing every edit.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770795#action_12770795 ]
Raghu Angadi commented on HBASE-1939:
-------------------------------------
> Wouldn't this leave forceSync to 'true' all the time?
not all the time. forceSync will remain unchanged when 'force' is true.
syncDone needs to be signaled in finally? Otherwise might leave the waiters hanging.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reopened HBASE-1939:
---------------------------------------
Reopening since it seems I was too fast at committing.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770731#action_12770731 ]
Andrew Purtell commented on HBASE-1939:
---------------------------------------
Sounds good J-D.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1939:
--------------------------------------
Attachment: HBASE-1939-v3.patch
v3 with Raghu's comments.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939-v3.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770729#action_12770729 ]
Jean-Daniel Cryans commented on HBASE-1939:
-------------------------------------------
Well if HBASE-1416 is really just about the first issue then I think this jira plays well with it, we can have a commit thread per WAL for example. Also my patch is already done, I'm currently running the unit tests to make sure I didn't forget something.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1939) HLog group commit
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771124#action_12771124 ]
stack commented on HBASE-1939:
------------------------------
+1 Looks good to me.
> HLog group commit
> -----------------
>
> Key: HBASE-1939
> URL: https://issues.apache.org/jira/browse/HBASE-1939
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-1939-v2.patch, HBASE-1939.patch
>
>
> Currently multiple clients writing to a RS are serialized when sync'ing their appends. Implementing group commit can help this by being aware of all the clients in that queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.