You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2011/01/14 00:20:45 UTC
[jira] Created: (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
----------------------------------------------------------------------------------------------------------------------
Key: HBASE-3443
URL: https://issues.apache.org/jira/browse/HBASE-3443
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
Sample test code outline:
{code}
admin.createTable(desc)
table = HTable.new(conf, tableName)
table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
admin.flush(tableName)
sleep(2)
del = Delete.new(Bytes.toBytes("row"))
table.delete(del)
table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
get = Get.new(Bytes.toBytes("row"))
keyValues = table.get(get).raw()
keyValues.each do |keyValue|
puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
end
{code}
The above prints:
{code}
Expect 5; Got Value=10
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252730#comment-13252730 ]
Jean-Daniel Cryans commented on HBASE-3443:
-------------------------------------------
The release notes should mention this tho.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253629#comment-13253629 ]
Hudson commented on HBASE-3443:
-------------------------------
Integrated in HBase-0.94-security #9 (See [https://builds.apache.org/job/HBase-0.94-security/9/])
HBASE-3443 ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix (Revision 1325453)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Benoit Sigoure (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250447#comment-13250447 ]
Benoit Sigoure commented on HBASE-3443:
---------------------------------------
This seems like a pretty bad bug that causes data corruption. I ran into it this weekend. Which release do you guys think will have a fix for this issue?
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252050#comment-13252050 ]
Lars Hofhansl commented on HBASE-3443:
--------------------------------------
I also verified that the test included in the patch fails without the other changes.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Priority: Critical
> Labels: corruption
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252743#comment-13252743 ]
stack commented on HBASE-3443:
------------------------------
What J-D said.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-3443.
----------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to trunk
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250968#comment-13250968 ]
Lars Hofhansl commented on HBASE-3443:
--------------------------------------
For 0.96, though.
I can take this. Can't promise to work on it today, though.
Had checked out the code a while ago to make ICV correct w.r.t. to WAL updates, this involves mostly removing some crufty code.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Priority: Critical
> Labels: corruption
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-3443:
---------------------------------
Fix Version/s: 0.94.0
a'right, committed to 0.94 aswell, and added release notes.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253143#comment-13253143 ]
Hudson commented on HBASE-3443:
-------------------------------
Integrated in HBase-TRUNK-security #169 (See [https://builds.apache.org/job/HBase-TRUNK-security/169/])
HBASE-3443 ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix (Revision 1325406)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3443:
-------------------------
Priority: Critical (was: Major)
Marking critical
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Priority: Critical
> Labels: corruption
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252813#comment-13252813 ]
Hudson commented on HBASE-3443:
-------------------------------
Integrated in HBase-0.94 #107 (See [https://builds.apache.org/job/HBase-0.94/107/])
HBASE-3443 ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix (Revision 1325453)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Kannan Muthukkaruppan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128529#comment-13128529 ]
Kannan Muthukkaruppan commented on HBASE-3443:
----------------------------------------------
Now that we have lazy seeks, i.e. HBASE-4465, we should be able to revert the work/optimization done HBASE-3082, and avoid this bug. What do you folks think?
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252729#comment-13252729 ]
Jean-Daniel Cryans commented on HBASE-3443:
-------------------------------------------
Correctness should always come first IMO.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-3443:
---------------------------------
Attachment: 3443.txt
Here's a patch. Quite trivial, just removing some code and using the existing get without invoking coprocessors
BUT... To test the performance I jacked up the number of increments per thread in TestAtomicOperation.testIncrementMultiThreads to 10000 and ran it with and without this change.
On my machine I find that the test runs around 23-24s with the existing code and around 27-28s with this patch.
Not sure that is a hit we want take.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Priority: Critical
> Labels: corruption
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-3443:
------------------------------------
Assignee: Lars Hofhansl
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252711#comment-13252711 ]
Hudson commented on HBASE-3443:
-------------------------------
Integrated in HBase-TRUNK #2746 (See [https://builds.apache.org/job/HBase-TRUNK/2746/])
HBASE-3443 ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix (Revision 1325406)
Result = SUCCESS
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252190#comment-13252190 ]
stack commented on HBASE-3443:
------------------------------
+1 on patch.
It makes us correct in the face of deletes. That is more important than a possible 10% slow down.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250920#comment-13250920 ]
stack commented on HBASE-3443:
------------------------------
Anyone got a patch for this?
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Labels: corruption
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128530#comment-13128530 ]
Ted Yu commented on HBASE-3443:
-------------------------------
+1 on the proposal.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252726#comment-13252726 ]
Lars Hofhansl commented on HBASE-3443:
--------------------------------------
@Stack: Wasn't sure about this one. Didn't feel right to add this to a "performance release" :)
You think this should go into 0.94? Might be good to have this correctness fix early.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252744#comment-13252744 ]
stack commented on HBASE-3443:
------------------------------
Oh, and if you don't fix it, you'll have to explain why you didn't to Benoît.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128564#comment-13128564 ]
Lars Hofhansl commented on HBASE-3443:
--------------------------------------
I agree. I think delete handling is generally a bit "funky" in HBase (see also HBASE-4536).
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-3443:
---------------------------------
Release Note:
This is a correctness fix and will incur a 10-20% performance penalty for ICV and Increment operations. Other operations are not affected.
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3443) ICV optimization to look in
memstore first and then store files (HBASE-3082) does not work when deletes
are in the mix
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252709#comment-13252709 ]
stack commented on HBASE-3443:
------------------------------
0.94?
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-3443:
---------------------------------
Fix Version/s: 0.96.0
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Assignee: Lars Hofhansl
> Priority: Critical
> Labels: corruption
> Fix For: 0.96.0
>
> Attachments: 3443.txt
>
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3443) ICV optimization to look in memstore
first and then store files (HBASE-3082) does not work when deletes are in
the mix
Posted by "Benoit Sigoure (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benoit Sigoure updated HBASE-3443:
----------------------------------
Component/s: regionserver
Affects Version/s: 0.90.0
0.90.1
0.90.2
0.90.3
0.90.4
0.90.5
0.90.6
0.92.0
0.92.1
Labels: corruption (was: )
> ICV optimization to look in memstore first and then store files (HBASE-3082) does not work when deletes are in the mix
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3443
> URL: https://issues.apache.org/jira/browse/HBASE-3443
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0, 0.90.1, 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6, 0.92.0, 0.92.1
> Reporter: Kannan Muthukkaruppan
> Labels: corruption
>
> For incrementColumnValue() HBASE-3082 adds an optimization to check memstores first, and only if not present in the memstore then check the store files. In the presence of deletes, the above optimization is not reliable.
> If the column is marked as deleted in the memstore, one should not look further into the store files. But currently, the code does so.
> Sample test code outline:
> {code}
> admin.createTable(desc)
> table = HTable.new(conf, tableName)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> admin.flush(tableName)
> sleep(2)
> del = Delete.new(Bytes.toBytes("row"))
> table.delete(del)
> table.incrementColumnValue(Bytes.toBytes("row"), cf1name, Bytes.toBytes("column"), 5);
> get = Get.new(Bytes.toBytes("row"))
> keyValues = table.get(get).raw()
> keyValues.each do |keyValue|
> puts "Expect 5; Got Value=#{Bytes.toLong(keyValue.getValue())}";
> end
> {code}
> The above prints:
> {code}
> Expect 5; Got Value=10
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira