You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/09/26 07:43:44 UTC
[jira] Created: (HBASE-902) Add force major compaction and force
split to UI
Add force major compaction and force split to UI
------------------------------------------------
Key: HBASE-902
URL: https://issues.apache.org/jira/browse/HBASE-902
Project: Hadoop HBase
Issue Type: Improvement
Reporter: stack
Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
I thought it would be just a little messing in jsp but its a bit more than that.
Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641571#action_12641571 ]
Jean-Daniel Cryans commented on HBASE-902:
------------------------------------------
All tests in trunk should be passing now.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639307#action_12639307 ]
stack commented on HBASE-902:
-----------------------------
Looking at HbaseObjectWritable, looks like they should make it across the great RPC divide. Not the end of the world if you don't convert them A. Worry about your meta issue (smile).
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635885#action_12635885 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
Edited my earlier comment to s/ServerManager/RegionManager/ for clarity.
My take is that RegionManager in the master is already orchestrating assignment, and is a central place to e.g. determine if a requested split has already happened.
Also it's clean and easy then to have a big 'Split' button in the master UI this way. :-)
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Summary: Add force compaction and force split operations to UI and Admin (was: Add force major compaction and force split to UI)
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.19.0
Status: Resolved (was: Patch Available)
Committed.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636482#action_12636482 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
HStore has a reference to its HRI but not to the HRegion. Instead of using HRI I could just convert the reference that HStore has to the region to HRegion and fix up the affected code.
Don't know about metautils.
If a HStore still holds references, the split request will be ignored. The only thing the "force split' request does differently is ask that the HStore max file size be effectively considered to be 1.
If the regionserver is already part way into the split, it should be ignored also but I'll double check this. compactSplitThread and children should handle the case already. If not they should.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640754#action_12640754 ]
stack commented on HBASE-902:
-----------------------------
One other thing: TRUNK unit tests are broke at the moment so don't be surprised when hudson complains after commit.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Status: Patch Available (was: Reopened)
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, 902.fix.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635055#action_12635055 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
If this is going to happen, can we/I put in a client side method for forcing a split? An importer process can be smart about splitting a region whenever the row key distribution is good, and anyway pressing a button on the Master UI repeatedly seems a coarse way to do it (not that I'm arguing against having such a button...)
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635068#action_12635068 ]
stack commented on HBASE-902:
-----------------------------
+1
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639296#action_12639296 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
Thanks for the review Stack. I'll hit all of your points in the next rev.
About the SET_HTD, SPLIT, and COMPACT constants, are Java enums an issue for HRPC? The thought occurred to me so I just punted and used ints.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651476#action_12651476 ]
stack commented on HBASE-902:
-----------------------------
+1 on patch. FYI Andrew for next time, the general hadoop style adds '{' parenthesis even if only one line inside the curly-braces. Good stuff.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, 902.fix.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-902:
------------------------------------
Assignee: Andrew Purtell
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639178#action_12639178 ]
stack commented on HBASE-902:
-----------------------------
Andrew: I'm down w/ your explaination of explicit row rather than explicit region and +1 on your last png.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: split.png
Please let me know if the proposed UI change (attached as split.png) is not acceptable in some way.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635058#action_12635058 ]
stack commented on HBASE-902:
-----------------------------
Go for it.
Its easy in UI since you have the HRS context just there. Running split from client is a bit harder in that you have to add support for new admin message. Rather than an a message per, should we just add support once for a Writable version of http://java.sun.com/j2se/1.4.2/docs/api/java/util/EventObject.html? Split would be the first message type. Flush, Major Compact, Merge could be others. UI and Client could send same eventing mechanism.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: 902-2.patch
New patch with UI bits included. Compaction requests work ok. Split requests break META. Debugging that now.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635802#action_12635802 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
I've got something preliminary mostly worked out.
admin.modifyTable as described is ioctl-like (which I personally like, for this type of swiss army knife type interface) and so in the client API uses Java varargs and at the HRPC level uses Writable[] to box them up and ship them to the master.
For example:
{code}
// replace HTD
admin.modifyTable(tableName, SET_META, htd);
// split all regions of the table
admin.modifyTable(tableName, SPLIT);
// split one region of the table only, closest to the given key
admin.modifyTable(tableName, SPLIT, rowKey);
{code}
Objections?
Also, in order to communicate administrative actions from master to regionservers, the master will keep a list of pending actions in ServerManager. I'll need to add new HMsg ops for forcing splits etc. that the master will send back to the appropriate region servers when they check in.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638789#action_12638789 ]
Jean-Daniel Cryans commented on HBASE-902:
------------------------------------------
I like what I see. A bit of nitpicking tho would be that you refer to region "keys" yet the table lists "names". One or the other.
Would it be easy to have the same functionalities in the shell?
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640879#action_12640879 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
Thanks Stack for the review. I'll make the necessary fixups and commit assuming all tests succeed (except for TestMigrate).
Thanks for reminding me about the shell access to the split facility. Really that should be part of HBASE-800 I think. Meantime the modifyTableMeta (now modifyTable after this patch is committed) workaround will let you split a table from the shell:
To replace HTD, for example:
{code}
table = HTable.new(@configuration, 'test')
htd = table.getTableDescriptor()
# need a read-write descriptor
htd = HTableDescriptor.new(htd)
htd.setMaxFileSize(67108864)
admin = HBaseAdmin.new(@configuration)
admin.modifyTable(Bytes.toBytes('test'), HConstants.MODIFY_TABLE_SET_HTD, htd)
{code}
To split all regions of a table:
{code}
admin = HBaseAdmin.new(@configuration)
admin.modifyTable(Bytes.toBytes('test'), HConstants.MODIFY_TABLE_SPLIT)
{code}
To split one region of a table:
{code}
admin = HBaseAdmin.new(@configuration)
admin.modifyTable(Bytes.toBytes('test'), HConstants.MODIFY_TABLE_SPLIT, Bytes.toBytes('keyInRegion'))
{code}
To force a compaction of all regions of a table:
{code}
admin = HBaseAdmin.new(@configuration)
admin.modifyTable(Bytes.toBytes('test'), HConstants.MODIFY_TABLE_COMPACT)
{code}
To force a compaction of one region of a table:
{code}
admin = HBaseAdmin.new(@configuration)
admin.modifyTable(Bytes.toBytes('test'), HConstants.MODIFY_TABLE_COMPACT, Bytes.toBytes('keyInRegion'))
{code}
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638792#action_12638792 ]
Jean-Daniel Cryans commented on HBASE-902:
------------------------------------------
{quote}
Hmm... Well at the interface/HRPC level the split and compact requests use a key to locate a specific region, not a region name or encoded name, so I think key is appropriate. Using keys allows a client to split regions considering the distribution of the keys if it knows anything about that, which I assume it will hence the desire to control splitting. We can keep it this way or move to using region names instead of a key contained within to target specific regions. Either way works for me, but the latter means more work for the client for what I believe would be a common use case.
{quote}
Oh yeah I see. You should keep it that way.
bq. As for putting this functionality in the shell, indeed I intend to add it. The shell can use the client side HBaseAdmin.modifyTable interface. See earlier comments on this.
Ok great.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-902) Add force major compaction
and force split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635802#action_12635802 ]
apurtell edited comment on HBASE-902 at 9/30/08 3:39 PM:
---------------------------------------------------------------
I've got something preliminary mostly worked out.
admin.modifyTable as described is ioctl-like (which I personally like, for this type of swiss army knife type interface) and so in the client API uses Java varargs and at the HRPC level uses Writable[] to box them up and ship them to the master.
For example:
{code}
// replace HTD
admin.modifyTable(tableName, SET_META, htd);
// split all regions of the table
admin.modifyTable(tableName, SPLIT);
// split one region of the table only, closest to the given key
admin.modifyTable(tableName, SPLIT, rowKey);
{code}
Objections?
Also, in order to communicate administrative actions from master to regionservers, the master will keep a list of pending actions in RegionManager. I'll need to add new HMsg ops for forcing splits etc. that the master will send back to the appropriate region servers when they check in.
was (Author: apurtell):
I've got something preliminary mostly worked out.
admin.modifyTable as described is ioctl-like (which I personally like, for this type of swiss army knife type interface) and so in the client API uses Java varargs and at the HRPC level uses Writable[] to box them up and ship them to the master.
For example:
{code}
// replace HTD
admin.modifyTable(tableName, SET_META, htd);
// split all regions of the table
admin.modifyTable(tableName, SPLIT);
// split one region of the table only, closest to the given key
admin.modifyTable(tableName, SPLIT, rowKey);
{code}
Objections?
Also, in order to communicate administrative actions from master to regionservers, the master will keep a list of pending actions in ServerManager. I'll need to add new HMsg ops for forcing splits etc. that the master will send back to the appropriate region servers when they check in.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-902:
------------------------
Fix Version/s: (was: 0.20.0)
0.19.0
Moved wrong issue.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638790#action_12638790 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
Hmm... Well at the interface/HRPC level the split and compact requests use a key to locate a specific region, not a region name or encoded name, so I think key is appropriate. Using keys allows a client to split regions considering the distribution of the keys if it knows anything about that, which I assume it will hence the desire to control splitting. We can keep it this way or move to using region names instead of a key contained within to target specific regions. Either way works for me, but the latter means more work for the client for what I believe would be a common use case.
As for putting this functionality in the shell, indeed I intend to add it. The shell can use the client side HBaseAdmin.modifyTable interface. See earlier comments on this.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: 902-1.patch
Patch attached that implements force split functions. Provided at this time for feedback on implementation choices (if any). No master UI tweaks yet. Passes all tests but util.TestMigrate. Working on a testcase now to confirm this works as expected and also a "split" button for the master UI.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reopened HBASE-902:
----------------------------------
Forced splits do not seem to work any more on trunk. Find out why.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-902) Add force compaction and
force split operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641572#action_12641572 ]
apurtell edited comment on HBASE-902 at 10/21/08 1:24 PM:
----------------------------------------------------------------
Committed. Passed all tests in trunk.
was (Author: apurtell):
Committed.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-902) Add force major compaction
and force split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639630#action_12639630 ]
apurtell edited comment on HBASE-902 at 10/14/08 3:37 PM:
----------------------------------------------------------------
Patch 902-3 is much better. It does everything it is supposed to. This time around I had substantially more clue.
Passes all tests but TestMigrate, a FileSystemVersionException from FSUtils.checkVersion. I didn't change the file system version so I don't know what's going on with that.
was (Author: apurtell):
This patch is much better. It does everything it is supposed to. This time around I had substantially more clue.
Passes all tests but TestMigrate, a FileSystemVersionException from FSUtils.checkVersion. I didn't change the file system version so I don't know what's going on with that.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635065#action_12635065 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
modifyTableMeta was originally going to be a catch-all like this for meta ops -- of which HTD modification would be one -- but I scaled back because all I wanted to do at the time was modify HTD. So instead of modifyTableMeta() it would be like modifyTable(). Maybe that should get a fresh look?
{code}
admin.modifyTable(table, SET_META, htd)
admin.modifyTable(table, FLUSH)
admin.modifyTable(table, MERGE, ...)
{code}
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640734#action_12640734 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
I would like to commit this but do not want to commit something (of mine) without review by another pair of eyes. Will wait.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed. Added braces as suggested.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, 902.fix.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638709#action_12638709 ]
stack commented on HBASE-902:
-----------------------------
Yeah on the bit of doc.
What if table has many regions? Where does the button go then? Does the midkey determine which region to squash? Maybe squash the optional midkey and let the system figure it out?
On major compactions, lads here were asking if it'd be possible to run a major compaction on a table, not just a region?
Good stuff Andrew.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: 902-3.patch
This patch is much better. It does everything it is supposed to. This time around I had substantially more clue.
Passes all tests but TestMigrate, a FileSystemVersionException from FSUtils.checkVersion. I didn't change the file system version so I don't know what's going on with that.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635073#action_12635073 ]
Jonathan Gray commented on HBASE-902:
-------------------------------------
+1
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636481#action_12636481 ]
stack commented on HBASE-902:
-----------------------------
Oh, if regionserver is already part ways into a split, what happens? Is request to split ignored and dropped?
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638705#action_12638705 ]
Jean-Daniel Cryans commented on HBASE-902:
------------------------------------------
Big button is nice! I would add a bit of doc (maybe to the right in italic) to briefly explain what it does.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639291#action_12639291 ]
stack commented on HBASE-902:
-----------------------------
Patch looks good to me Andrew.
Below doesn't need to be in an else:
{code}
+ } else {
+ synchronized (onlineMetaRegions) {
+ return onlineMetaRegions.get(
+ onlineMetaRegions.tailMap(key).firstKey());
+ }
{code}
...given as the clause above it ends in a return.
Only concern is what if a split is ongoing or if master gets a SPLIT message when it has one pending?
Run 'ant javadoc'. I think you should see warnings going by what you've added. See if you can clean them up.
Is this intended:
{code}
- writeClassCode(out, c);
+ out.writeByte(code);
{code}
Should these be enums? No biggie, just a suggestion:
{code}
+ /** modifyTable op for replacing the table descriptor */
+ public static final int SET_HTD = 1;
+ /** modifyTable op for forcing a split */
+ public static final int SPLIT = 2;
+ /** modifyTable op for forcing a compaction */
+ public static final int COMPACT = 3;
{code}
Otherwise, looks great.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636488#action_12636488 ]
stack commented on HBASE-902:
-----------------------------
I'd say go with what you have for now. I was going to suggest that we pass to HStore its hosting HRegion but our general impetus should be going the opposite direction making it so HStore doesn't even take an HRI so can be standalone (Things are a little perverse at the moment; HRI is needed supporting binary keys until we redo HSK).
Don't worry about metautils either.
OK on rest of your comment.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Status: Patch Available (was: Open)
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force major compaction and force
split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: ui.1.png
The idea is to have a menu of table actions at the bottom of table.jsp output, separated by a horizontal rule from the region listing. The forms post back via GET to table.jsp which performs the appropriate action as indicated by query parameters. This also allows table actions to be easily automated.
See attached 'ui.1.png' which incorporates suggestions by jdcryans and stack.
If a key is not given that locates a specific region, all regions are split or compacted or whatever action is under consideration. The key does not need to be a midkey, only <= the start key for the particular region. (This is why getMetaRegionForKeyClosest().)
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635882#action_12635882 ]
stack commented on HBASE-902:
-----------------------------
Only objection I would have had was making varargs cross the RPC but you seem to have figured that so, go for it.
Does the master have to tell the regionserver split? Can the client go direct to the regionserver and tell it since normally regionservers initiate and orchestrate splits? (There is probably a reason if I took a look at code).
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-902) Add force major compaction
and force split to UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638783#action_12638783 ]
apurtell edited comment on HBASE-902 at 10/11/08 11:55 AM:
-----------------------------------------------------------------
The idea is to have a menu of table actions at the bottom of table.jsp output, separated by a horizontal rule from the region listing. The forms post back via GET to table.jsp which performs the appropriate action as indicated by query parameters. This also allows table actions to be easily automated.
See attached 'ui.1.png' which incorporates suggestions by jdcryans and stack.
If a key is not given that locates a specific region, all regions are split or compacted or whatever action is under consideration. The key does not need to be a midkey, only >= the start key for the particular region. (This is why getMetaRegionForKeyClosest().)
was (Author: apurtell):
The idea is to have a menu of table actions at the bottom of table.jsp output, separated by a horizontal rule from the region listing. The forms post back via GET to table.jsp which performs the appropriate action as indicated by query parameters. This also allows table actions to be easily automated.
See attached 'ui.1.png' which incorporates suggestions by jdcryans and stack.
If a key is not given that locates a specific region, all regions are split or compacted or whatever action is under consideration. The key does not need to be a midkey, only <= the start key for the particular region. (This is why getMetaRegionForKeyClosest().)
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-902:
------------------------
Fix Version/s: (was: 0.19.0)
0.20.0
Moving out of 0.19 (unless Clint shows up soon with new patch addressing Andrew's commentary).
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.20.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636480#action_12636480 ]
stack commented on HBASE-902:
-----------------------------
Its interesting how you carry then internal split message in HRI. Thats fine I suppose least I cannot come up with anything more elegant (better to have HStore go to HRI than to its hosting HRegion. There is a presumption that HRegion and HStore both share HRI instances. Maybe add a comment to this effect on the HStore HRI data member?
Perhaps name the HRI method 'split' rather than shouldSplit?
Should RegionManager.startSplit and endSplit instead be scheduleSplit and removeSplit or some such? start and end don't seem to describe whats going on here.
Does getMetaRegionForTableClosest belong in metautils? It looks generally useful?
If an HStore still holds references, will we split (We shouldn't -- or rather, we can't).
This is really great functionality. I like the way you've made it easily extensible too.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650696#action_12650696 ]
Andrew Purtell commented on HBASE-902:
--------------------------------------
In private communication Krzysztof Szlapinski asks for the ability to split empty regions. Will look into this also.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-902) Add force compaction and force split
operations to UI and Admin
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-902:
---------------------------------
Attachment: 902.fix.patch
Was having problems with splitting due to not flushing memcache first. Patch "902.fix.patch" also adds a test case for force splitting.
> Add force compaction and force split operations to UI and Admin
> ---------------------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Fix For: 0.19.0
>
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, 902.fix.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-902) Add force major compaction and force
split to UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640753#action_12640753 ]
stack commented on HBASE-902:
-----------------------------
Reviewed patch and did basic testing. Looks great to me. Go commit it.
Be warned that the patch fails going into HRegionServer, IIRC, because HRS has changed since this patch. Fix is easy though. Also, in your webapp changes, no such thing as isEmpty in String. Change the !isEmpty to key != null && key.length() > 0.
Its excellent hitting the split key, waiting a while, and then seeing your regions doubled before your eyes.
How do I get at this new facility from the shell? Its ok if I have to write a bit of code to do it. We just need to document it in FAQ or some place.
Good stuff Andrew.
> Add force major compaction and force split to UI
> ------------------------------------------------
>
> Key: HBASE-902
> URL: https://issues.apache.org/jira/browse/HBASE-902
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Andrew Purtell
> Attachments: 902-1.patch, 902-2.patch, 902-3.patch, split.png, ui.1.png
>
>
> Would help debugging or at start of big upload or if we have a hot region we want to distribute over the cluster if we could just manually force the splitting of regions.
> I thought it would be just a little messing in jsp but its a bit more than that.
> Actual decision to split is made down in guts of the Store. Won't split if any references still around and a file must be > maxsize.
> Chatting w/ Jon Gray, could set a flag on the region that we want to split. Could do this from UI (Can get to HRegion instance from HRS instance in HRS jsp). Stores have references to their hosting regions (I think). When figuring if we're to split, check the force split flag up in the hosting region.
> To get the flag checked, safest way is to just force a flush (Otherwise, have to mess w/ locks in the compact/splitter thread). Registering a flush request on a region is easy enough to do. Its a public method on HRS (or HR).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.