You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Lars George (JIRA)" <ji...@apache.org> on 2009/12/01 19:20:21 UTC
[jira] Created: (HBASE-2021) Add compaction details to master UI
Add compaction details to master UI
-----------------------------------
Key: HBASE-2021
URL: https://issues.apache.org/jira/browse/HBASE-2021
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Lars George
Assignee: Lars George
Priority: Minor
There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-2021) Add compaction details to master
UI
Posted by Lars George <la...@gmail.com>.
Hi,
I tested this now on a larger cluster and the initial wait will
definitely cause frustration. I will go with plan be and load the
fragmentation asynchronously. Any preferred way do do that? Small
thread class like the other workers or is there a better way? Is that
OK? One more thread in the master which is idle most of the time
compared to the RS's.
Lars
On Tue, Jan 5, 2010 at 4:14 PM, Lars George (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796693#action_12796693 ]
>
> Lars George commented on HBASE-2021:
> ------------------------------------
>
> I added those "title" attributes to the headers with some "soothing" explanation. Do you think that is not good enough?
>
>> Add compaction details to master UI
>> -----------------------------------
>>
>> Key: HBASE-2021
>> URL: https://issues.apache.org/jira/browse/HBASE-2021
>> Project: Hadoop HBase
>> Issue Type: Improvement
>> Reporter: Lars George
>> Assignee: Lars George
>> Priority: Minor
>> Fix For: 0.20.3, 0.21.0
>>
>> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>>
>>
>> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
>> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
>> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
>> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Fix Version/s: 0.20.3
Added 0.20.3 again in an attempt to get that metric also included. Punt if you do not like it. Will add a new patch for 0.21 based on the above discussion, so we leave this issue open once the 0.20 patch is applied by moving it then to 0.21 only?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796693#action_12796693 ]
Lars George commented on HBASE-2021:
------------------------------------
I added those "title" attributes to the headers with some "soothing" explanation. Do you think that is not good enough?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791663#action_12791663 ]
Andrew Purtell commented on HBASE-2021:
---------------------------------------
bq. Should we have an additional patch that adds the HServerInfo stuff on top of the above patch so we can apply the minor change to 0.20.3 and the additional RPC changes to 0.21.0?
+1
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Fix Version/s: 0.21.0
Status: Patch Available (was: Open)
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796135#action_12796135 ]
Lars George commented on HBASE-2021:
------------------------------------
Good idea with the hidden config! Didn't think of that. Will add today.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785459#action_12785459 ]
Lars George commented on HBASE-2021:
------------------------------------
Not yet (have no cluster yet to test this on) but will asap. Also, please note that this is cached and reloads every 2 minutes only. So the slowest will be the filesystem scan which is meta data. Just when the DFS and master is heavily taxed I could see slower response times (every 2 minutes).
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793035#action_12793035 ]
Lars George commented on HBASE-2021:
------------------------------------
Bugger! Patch was made on trunk. Did you try 0.20 branch or trunk? Will redo a clean 0.20 branch based patch and then a combined patch for trunk with the HServerLoad additions.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793028#action_12793028 ]
Andrew Purtell commented on HBASE-2021:
---------------------------------------
bq. Would you agree to put this into HServerLoad? Seems a natural fit to add it there.
+1
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791316#action_12791316 ]
Andrew Purtell commented on HBASE-2021:
---------------------------------------
Actually I would encourage you to consider adding information about regions being compacted to HServerInfo. For 0.21 we can change RPC formats and bump the versions. I would like to be able to see a list of regions pending compaction (or currently being compacted) when I type status 'detailed' in the shell.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Attachment: HBASE-2021.patch
HBASE-2021.patch adds
- compaction queue size to region server metrics, can be seen on regionserver.jsp
|Metrics | request=0.0, regions=3, stores=6, storefiles=3, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0, usedHeap=35, maxHeap=994, blockCacheSize=1713760, blockCacheFree=206913552, blockCacheCount=1, blockCacheHitRatio=95 | RegionServer Metrics; file and heap sizes are in megabytes |
The "compactionQueueSize" reflects the number of regions this regionserver has in the queue to be processed and allows to "determine" if compactions are in progress from the UI.
I would have preferred to have a global indicator in the master UI but that value cannot be bubbled up unless I for example add it to HServerInfo - which I have been told is not a good idea to change. Right?
- Added fragmentation details to UI
Added a total fragmentation precentage to master.jsp in top table, for example:
| Regions On FS | 3 | Number of regions on FileSystem. Rough count. |
| Fragmentation | 17% | Overall fragmentation of all tables, including .META. and -ROOT-.|
|Zookeeper Quorum | localhost:2181 | Addresses of all registered ZK servers. For more, see zk dump.|
Then in the tables below for .META., -ROOT- and user table I added it as a column with an abbreviated title but added a pop-up explanation when you hover over it.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785445#action_12785445 ]
Jean-Daniel Cryans commented on HBASE-2021:
-------------------------------------------
That functionality looks very good to me, did you try it on a cluster with tons of regions?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796169#action_12796169 ]
Lars George commented on HBASE-2021:
------------------------------------
BTW, the idea is that if this too much of a tool on the UI to simply move this into a thread and update every X msecs as per the config. That also would have the advantage of a delayed start where the UI loads as usual and shows "n/a" in the fragmentation spots and when the thread completes the first time it would show those values on a UI reload and from then on each set of last determined values.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Attachment: HBASE-2021-0.20-v2.patch
2021-0.20-v2 patch is done again after the previous 0.20 patch failed on the master.jsp again.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796053#action_12796053 ]
stack commented on HBASE-2021:
------------------------------
Patch looks great Lars. It applies cleanly and all builds. I like that you only check FS every two minutes (Doing it on every metrics interval would be too much methinks). I'll commit this tomorrow, but if you have a second between here and then, perhaps change the two minutes to be a configuration... it doesn't have to have an entry in hbase-default.xml.. just be something someone could discover configurable if they took the trouble to read the source. Good stuff.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792893#action_12792893 ]
Lars George commented on HBASE-2021:
------------------------------------
Would you agree to put this into HServerLoad? Seems a natural fit to add it there.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784323#action_12784323 ]
Lars George commented on HBASE-2021:
------------------------------------
Exactly, I would build in a performance gauge to reload the /hbase to compute the fragmentation to run only every x seconds (I would default it to 60) so that a UI reload is quick. This also depends where this goes, into the master.jsp or table.jsp.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Attachment: HBASE-2021-0.20-v3.patch
Patch v3 adds
{code}
// only check every two minutes by default
int check = this.conf.getInt("hbase.master.fragmentation.check.frequency", 2 * 60 * 1000);
{code}
so it can be configured if so desired.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797268#action_12797268 ]
stack commented on HBASE-2021:
------------------------------
The UI just says "fragmentation". It doesn't explain what it means.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791656#action_12791656 ]
Lars George commented on HBASE-2021:
------------------------------------
I am sure all for it. Should we have an additional patch that adds the HServerInfo stuff on top of the above patch so we can apply the minor change to 0.20.3 and the additional RPC changes to 0.21.0?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George updated HBASE-2021:
-------------------------------
Attachment: HBASE-2021-0.20.patch
Patch for 0.20 branch.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2021) Add compaction details to master UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2021.
--------------------------
Resolution: Fixed
Applied branch and trunk. Thanks for the patch Lars.
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784320#action_12784320 ]
Jean-Daniel Cryans commented on HBASE-2021:
-------------------------------------------
I like the idea of the percentage, then when we would ask for a major compaction we would see that number going down?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2021) Add compaction details to master UI
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2021:
----------------------------------
Status: Open (was: Patch Available)
The attached patch is corrupt. Cancelling patch. Resubmit when ready.
{noformat}
[...]
patching file src/webapps/master/master.jsp
Hunk #1 FAILED at 3.
1 out of 1 hunk FAILED -- saving rejects to file src/webapps/master/master.jsp.rej
missing header for unified diff at line 263 of patch
can't find file to patch at input line 263
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|
| import="org.apache.hadoop.hbase.master.HMaster"
|
| import="org.apache.hadoop.hbase.HConstants"
|
| import="org.apache.hadoop.hbase.master.MetaRegion"
|
--------------------------
[...]
{noformat}
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2021) Add compaction details to master UI
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796541#action_12796541 ]
stack commented on HBASE-2021:
------------------------------
... though I wonder if we don't need better explaination of what fragmentation means and how its addressed?
> Add compaction details to master UI
> -----------------------------------
>
> Key: HBASE-2021
> URL: https://issues.apache.org/jira/browse/HBASE-2021
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Lars George
> Assignee: Lars George
> Priority: Minor
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2021-0.20-v2.patch, HBASE-2021-0.20-v3.patch, HBASE-2021-0.20.patch, HBASE-2021.patch
>
>
> There are two issues with this, first to detect that there is a compaction needed. You can currently use the little helper util that checks if a table has at least one colfam with more than one store file. I though about scanning all tables and all colfams in each and then compute the "fragmentation" ratio as a percentage of colfams with more than one store to the total number of colfams. That gives a "Table xyz is 33% fragmented" output. While minor percentage are normal under insert operations it is still important to know how bad the fragmentation is overall.
> Another idea is to weigh the number of files per store too, so that if you have two per colfam it is considered "low" and if you have more, for example 6-8 it is considered "high". Not sure how that can be done yet but noting the idea down here.
> Of course seeing the .META. fragmentation is useful to quickly debug performance issues (as JD told me on IRC).
> The other issue is that when you have started a compaction you have no idea how far it is and if it is still in progress. One indication of course is the above value. If it is 0% then all is done. But if you are at say 23%, is it still compacting? We could have a simple status that compactions are still in progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.