You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Liyin Tang (Created) (JIRA)" <ji...@apache.org> on 2012/01/14 00:42:40 UTC
[jira] [Created] (HBASE-5199) Delete out of TTL store files before
compaction selection
Delete out of TTL store files before compaction selection
---------------------------------------------------------
Key: HBASE-5199
URL: https://issues.apache.org/jira/browse/HBASE-5199
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Currently, HBase deletes the out of TTL store files after major compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
In this way, HBase can keep deleting the old invalid store files without major compaction, and also prevent from unnecessary major compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Liyin Tang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HBASE-5199:
------------------------------
Description:
Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
was:
Currently, HBase deletes the out of TTL store files after major compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
In this way, HBase can keep deleting the old invalid store files without major compaction, and also prevent from unnecessary major compactions since the out of TTL store files will be deleted before the compaction selection.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188840#comment-13188840 ]
Phabricator commented on HBASE-5199:
------------------------------------
khemani has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 find out the set of files to be deleted in a read lock. If that set is non-empty, then acquire the write lock and remove the to-be-deleted set of files (if they still exist) from the storefiles set.
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5199:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.1.patch
Liyin requested code review of "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA
Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
TEST PLAN
TestStore
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209167#comment-13209167 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has committed the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
REVISION DETAIL
https://reviews.facebook.net/D1311
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5199:
------------------------------
Fix Version/s: 0.94.0
Hadoop Flags: Reviewed
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Liyin Tang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204926#comment-13204926 ]
Liyin Tang commented on HBASE-5199:
-----------------------------------
Ping committers !
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, HBASE-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209168#comment-13209168 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has committed the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
REVISION DETAIL
https://reviews.facebook.net/D1311
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.5.patch
Liyin added you to the CC list for the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik
Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
TEST PLAN
TestStore
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205171#comment-13205171 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204934#comment-13204934 ]
Zhihong Yu commented on HBASE-5199:
-----------------------------------
HBASE-5199.patch doesn't apply cleanly:
{code}
3 out of 6 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java.rej
patching file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java.rej
{code}
Can you rebase patch ?
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, HBASE-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188891#comment-13188891 ]
Zhihong Yu commented on HBASE-5199:
-----------------------------------
This becomes interesting, Liyin.
Currently we have:
{code}
override = region.getCoprocessorHost().preCompactSelection(
this, candidates);
{code}
If I understand your idea correctly, a sub-list of candidates would be extracted for deletion.
Do you think that we should expose selection of expired store files through coprocessor ?
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192352#comment-13192352 ]
Phabricator commented on HBASE-5199:
------------------------------------
Liyin has requested a review of the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Hi @Karthik,
The new diff have changed the logic a little bit. There is no need to get the write lock to do the deletion job for these expired store files. It would be much easier to put these expired store files directly to the compaction, which will do the deletion job for free.
Would you mind reviewing it again? Thanks
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205172#comment-13205172 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205175#comment-13205175 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Liyin Tang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HBASE-5199:
------------------------------
Attachment: HBASE-5199.patch
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, HBASE-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Liyin Tang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HBASE-5199:
------------------------------
Release Note: set hbase.store.delete.expired.storefile as true to enable the expired store file deletion
Status: Patch Available (was: Open)
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, HBASE-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.5.patch
Liyin updated the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik
Address @Karthik's comments.
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192378#comment-13192378 ]
Phabricator commented on HBASE-5199:
------------------------------------
Karthik has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1118 Could we return a new CompactSelection object instead of modifying the existing one and returning a boolean? We can return null or an empty compact selection if there are no such files...
CompactSelection expiredFiles = compactSelection.selectExpiredStoreFiles(...)
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205173#comment-13205173 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188819#comment-13188819 ]
Phabricator commented on HBASE-5199:
------------------------------------
Liyin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Since this function is to change <this.storeFiles> by deleting the expired storeFiles, writeLock is needed here.
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Good Point ! I will update this.
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188633#comment-13188633 ]
Phabricator commented on HBASE-5199:
------------------------------------
Kannan has added reviewers to the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Added Reviewers: khemani, aaiyer, Karthik
adding a few more reviewers in case someone else can get to this earlier.
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192414#comment-13192414 ]
Phabricator commented on HBASE-5199:
------------------------------------
Karthik has accepted the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Looks good!
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.3.patch
Liyin updated the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik
Fix a typo.
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.2.patch
Liyin updated the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik
Address Karthik's comments.
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188823#comment-13188823 ]
Phabricator commented on HBASE-5199:
------------------------------------
tedyu has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Currently notifyChangedReadersObservers() is called outside lock.writeLock().
Please add the call after line 1044.
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188715#comment-13188715 ]
Phabricator commented on HBASE-5199:
------------------------------------
Karthik has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Havent reviewed the test code yet...
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1039 Do we need this assignment? Can we make the condition:
if (nonExpiredFiles.size() < currentStoreFiles.size()) {
//...
}
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1017 Should we exclude files contained in filesCompacting? Otherwise we could end up deleting a file which is currently being compacted...
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Liyin Tang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186059#comment-13186059 ]
Liyin Tang commented on HBASE-5199:
-----------------------------------
Thanks Lars and Kannan. I will double check this.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
>
> Currently, HBase deletes the out of TTL store files after major compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without major compaction, and also prevent from unnecessary major compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188681#comment-13188681 ]
Phabricator commented on HBASE-5199:
------------------------------------
tedyu has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1053 Should this check be moved to line 1016 ?
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1024 Changing 'because' to 'where' would make the sentence more readable
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Kannan Muthukkaruppan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186058#comment-13186058 ]
Kannan Muthukkaruppan commented on HBASE-5199:
----------------------------------------------
I think Lars is referring to HBASE-4721 by Prakash. Yes, worth making sure it plays well with that -- though I suspect it should work out just fine.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
>
> Currently, HBase deletes the out of TTL store files after major compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without major compaction, and also prevent from unnecessary major compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188780#comment-13188780 ]
Phabricator commented on HBASE-5199:
------------------------------------
khemani has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 should Readers be notified via notifyChangeReadersObservers()?
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Can this be done inside a readLock() (for most of the compactions no storefiles will actually get removed)
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209173#comment-13209173 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has committed the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
REVISION DETAIL
https://reviews.facebook.net/D1311
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205174#comment-13205174 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5199:
-------------------------------
Attachment: D1311.4.patch
Liyin updated the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik
During the minor compaction selection time, if there is any expired store files, these expired store files will be selected to compact directly.
Since these files have already expired, there would be no-ops during the compaction time and these files will be deleted after the compaction.
REVISION DETAIL
https://reviews.facebook.net/D1311
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206171#comment-13206171 ]
Hudson commented on HBASE-5199:
-------------------------------
Integrated in HBase-TRUNK #2659 (See [https://builds.apache.org/job/HBase-TRUNK/2659/])
HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin)
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192518#comment-13192518 ]
Zhihong Yu commented on HBASE-5199:
-----------------------------------
@Liyin:
Can you try adding JIRA to the CCs ?
I have to deal with 6 copies of each review email in my InBox.
Thanks
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Liyin Tang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HBASE-5199:
------------------------------
Attachment: hbase-5199.patch
The new patch is rebased on the latest trunk and all the unit tests are passed.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209163#comment-13209163 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has committed the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
REVISION DETAIL
https://reviews.facebook.net/D1311
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206694#comment-13206694 ]
Hudson commented on HBASE-5199:
-------------------------------
Integrated in HBase-TRUNK-security #109 (See [https://builds.apache.org/job/HBase-TRUNK-security/109/])
HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin) (Revision 1243086)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209160#comment-13209160 ]
Phabricator commented on HBASE-5199:
------------------------------------
mbautin has committed the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
REVISION DETAIL
https://reviews.facebook.net/D1311
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186029#comment-13186029 ]
Lars Hofhansl commented on HBASE-5199:
--------------------------------------
Need to pay some special attention to specific delete-marker TTL that FB added (i.e. a delete marker could outlive the TTL of a store). Can't find the jira for that right, but it was committed relatively recently.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
>
> Currently, HBase deletes the out of TTL store files after major compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without major compaction, and also prevent from unnecessary major compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188748#comment-13188748 ]
Phabricator commented on HBASE-5199:
------------------------------------
Karthik has accepted the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
LGTM!
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206163#comment-13206163 ]
Zhihong Yu commented on HBASE-5199:
-----------------------------------
The test failures were not related to this patch.
Integrated to TRUNK.
Thanks for the patch Liyin.
Thanks for the review Karthick, Prakash and Mikhail.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205163#comment-13205163 ]
Hadoop QA commented on HBASE-5199:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514057/hbase-5199.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -136 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationPeer
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/934//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/934//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/934//console
This message is automatically generated.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195135#comment-13195135 ]
Hadoop QA commented on HBASE-5199:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512236/HBASE-5199.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/860//console
This message is automatically generated.
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, HBASE-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files
before compaction selection
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188868#comment-13188868 ]
Phabricator commented on HBASE-5199:
------------------------------------
Liyin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@khemani: I see your point, which inspire me to a new idea: When holding a read lock for selection compaction candidates, it should also select the expired deletion store file candidates. After when processing completeCompaction logic within the write lock, it will delete the expired store files.
So this approach should NOT introduce any potential dead lock risk since there is no read or write lock introduced by this feature at all.
@tedyu: Thanks for pointing it out. And no additional notifyChangedReadersObservers() is needed if implementing in this way.
REVISION DETAIL
https://reviews.facebook.net/D1311
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5199:
------------------------------
Comment: was deleted
(was: mbautin has commented on the revision "[jira][HBASE-5199] Delete out of TTL store files before compaction selection ".
@Liyin: could you please post the rebased version of the patch so I can commit?
REVISION DETAIL
https://reviews.facebook.net/D1311
)
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch, hbase-5199.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before
compaction selection
Posted by "Liyin Tang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HBASE-5199:
------------------------------
Attachment: (was: HBASE-5199.patch)
> Delete out of TTL store files before compaction selection
> ---------------------------------------------------------
>
> Key: HBASE-5199
> URL: https://issues.apache.org/jira/browse/HBASE-5199
> Project: HBase
> Issue Type: Improvement
> Reporter: Liyin Tang
> Assignee: Liyin Tang
> Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch, D1311.4.patch, D1311.5.patch, D1311.5.patch
>
>
> Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
> In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira