You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/05/09 19:28:55 UTC
[jira] Created: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
---------------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-621
URL: https://issues.apache.org/jira/browse/HBASE-621
Project: Hadoop HBase
Issue Type: Improvement
Reporter: stack
HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-621.
-------------------------
Resolution: Fixed
Committed after review and run of unit tests locally (TestMigrate is hanging sometimes... probably scanner issue I saw a few times up in eclipse). Thanks for the patch Jean-Daniel.
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
> Attachments: hbase-621-v1.patch
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598322#action_12598322 ]
Jean-Daniel Cryans commented on HBASE-621:
------------------------------------------
Another problem with versions. Take a look at :
{code}
/**
* Get the value for the indicated HStoreKey. Grab the target value and the
* previous 'numVersions-1' values, as well.
*
* If 'numVersions' is negative, the method returns all available versions.
* @param key
* @param numVersions Number of versions to fetch. Must be > 0.
* @return values for the specified versions
* @throws IOException
*/
Cell[] get(HStoreKey key, int numVersions) throws IOException {
if (numVersions <= 0) {
throw new IllegalArgumentException("Number of versions must be > 0");
}
{code}
If 'numVersions', it will throw a IllegalArgumentException and we will not get all versions. As recommended by Bryan, I will add a new constant like HConstants.ALL_VERSIONS = Integer.MAX_VALUE and change the comments.
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-621:
----------------------------------------
Assignee: Jean-Daniel Cryans
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598362#action_12598362 ]
stack commented on HBASE-621:
-----------------------------
Good on you for digging in JDC and figuring no need of same mod up in MemCache.
Remove the assertion. It depended on a broken supposition which you are fixing here.
Regards where to slot in your test, if you can slot in your test in a place that makes sense, go for it -- especially if you can put it in in a place where the setup price has already been paid. Otherwise, make a new test.
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-621:
------------------------
Fix Version/s: 0.2.0
Adding it in for now. Can bump it later if we have to.
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.2.0
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598346#action_12598346 ]
Jean-Daniel Cryans commented on HBASE-621:
------------------------------------------
I have been digging around in HStore and here is what I found that causes me problem or that I think should get someone's attention :
- stack mentionned that this code around line 1250 will maybe need to change :
{code}
if (results.size() == numVersions) {
return results.toArray(new Cell[results.size()]);
}
{code}
I thought the same but by looking in Memcache we can see that the TTL problem is already handled :
{code}
if (ttl == HConstants.FOREVER ||
now < itKey.getTimestamp() + ttl) {
result.add(new Cell(tailMap.get(itKey), itKey.getTimestamp()));
} else {
victims.add(itKey);
...
// Remove expired victims from the map.
for (HStoreKey v: victims)
map.remove(v);
{code}
So I guess that it's ok to leave it like that.
- I was wondering where to put tests for this modification and I found that regionserver.TestCompaction tests conflicts with my modification. Have a look at :
{code}
// Add more content. Now there are about 5 versions of each column.
// Default is that there only 3 (MAXVERSIONS) versions allowed per column.
// Assert > 3 and then after compaction, assert that only 3 versions
// available.
addContent(new HRegionIncommon(r), Bytes.toString(COLUMN_FAMILY));
Cell[] cellValues =
r.get(STARTROW, COLUMN_FAMILY_TEXT, 100 /*Too many*/);
// Assert that I can get > 5 versions (Should be at least 5 in there).
assertTrue(cellValues.length >= 5);
{code}
This now fails because it returns MAXVERSIONS versions. Any advices on how to handle the max version problem? Is there an easy way of changing it just for this test?
Also, is TestCompaction a good place to test this jira? (since it seems it is already handled)
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-621) Make MAX_VERSIONS work like TTL: In
scans and gets, check MAX_VERSIONs setting and return that many only rather
than wait on compaction
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-621:
-------------------------------------
Attachment: hbase-621-v1.patch
First working version.
> Make MAX_VERSIONS work like TTL: In scans and gets, check MAX_VERSIONs setting and return that many only rather than wait on compaction
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-621
> URL: https://issues.apache.org/jira/browse/HBASE-621
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
> Attachments: hbase-621-v1.patch
>
>
> HBASE-47 added specification of TTL on cells. The implementation checks cell timestamp against configured TTL before returning results scanning or getting. You can also set the maximum versions of a cell to keep. The maximum versions is not checked scanning or getting, only when we compact (We'll drop cells that are beyond the maximum version at compaction time). This issue is about adding check for maximum versions to gets and scans so that if you ask for all versions but have configured the store to only keep 3 versions, though 4 may have been inserted, you'll currently get 4 returned (if compactions have not had a chance to run).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.