You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ShiXing (JIRA)" <ji...@apache.org> on 2012/06/26 11:23:42 UTC
[jira] [Created] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
ShiXing created HBASE-6269:
------------------------------
Summary: Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-v1.patch
When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = "testPutWithMemStoreFlush";
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes("family");;
byte[] qualifier = Bytes.toBytes("qualifier");
byte[] row = Bytes.toBytes("putRow");
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
List<KeyValue> kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes("value0");
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print("get value before flush after put value0 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print("get value after flush after put value0 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes("value1");
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print("get value before flush after put value1 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print("get value after flush after put value1 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes("value2");
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print("get value before flush after put value2 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print("get value after flush after put value2 : ");
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i < kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
}
{code}
and the result print as followed:
{code}
get value before flush after put value0 : value0
get value after flush after put value0 : value0
get value before flush after put value1 : value1
get value after flush after put value1 : value0
get value before flush after put value2 : value2
get value after flush after put value2 : value0
{code}
I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Anoop Sam John (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401930#comment-13401930 ]
Anoop Sam John commented on HBASE-6269:
---------------------------------------
@ShiXing
Thanks for ur detailed explanation. Yes I had gone through the code in more detail when the Lazy seeks happens.
Both the files are getting lazy seeked with same fake KV. And later in pollReadlKV() the 1st file's scanner is selected as scanner as both the KVs in the files are same.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401294#comment-13401294 ]
Ted Yu edited comment on HBASE-6269 at 6/26/12 11:00 AM:
---------------------------------------------------------
Can you generate patch for trunk for Hadoop QA ?
was (Author: yuzhihong@gmail.com):
Can you generate patch for trunk for Hadoop aa ?
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Status: Open (was: Patch Available)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Attachment: 6269.94
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401958#comment-13401958 ]
Zhihong Ted Yu commented on HBASE-6269:
---------------------------------------
Hadoop QA hasn't run tests for the patch.
@Xing:
Can you run test suite and post back the result ?
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402839#comment-13402839 ]
Zhihong Ted Yu commented on HBASE-6269:
---------------------------------------
Integrated to 0.94 as well.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403276#comment-13403276 ]
Hudson commented on HBASE-6269:
-------------------------------
Integrated in HBase-0.94-security #38 (See [https://builds.apache.org/job/HBase-0.94-security/38/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue (Xing Shi) (Revision 1354815)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "ShiXing (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401383#comment-13401383 ]
ShiXing commented on HBASE-6269:
--------------------------------
@anoop
There are 2 StoreFiles after flush two times, the sf2's sequenceId > sf1's sequenceId.
When get:
step1. the sf2 is the highest StoreFileScanner, and it enforceSeek() in KeyValueHeap.pollRealKV(), so the KeyValue2 is read out from StoreFile by real seek. And it compares to the fake KeyValue(called FakeKeyValue) that generated by KeyValue.createFirstOnRow() in StoreScanner.next(), and the FakeKeyValue's row, family, qualifier, timestamp, memstoreTS(always 0 for StoreFileScanner) are the same as KeyValue2 excepts Key type is Maximum, and Key type in KeyValue2 is Put. And the {code}comparator.compare(curKV=KeyValue2, nextKV=FakeKeyValue) = 251 > 0{code}. It means that the highest StoreFileScanner's highest KeyValue is not higher than the second. Followed is the value for example
{code}
KeyValue2 : putRow/family:qualifier/1234567/Put/vlen=6/ts=0
FakeKeyValue : putRow/family:qualifier/1234567/Maximum/vlen=0/ts=0
{code}
And then the second highest StoreFileScanner becomes the highest, and the highest is added to the heap.
Step2. The sf1's highest KeyValue is read out , we call it KeyValue1, the real value is the same as KeyValue2 fetched again by heap.peek():
{code}
KeyValue1 : putRow/family:qualifier/1234567/Put/vlen=6/ts=0
{code}
Step3. KeyValue1 compares KeyValue2, and the {code}comparator.compare(curKV=KeyValue1, nextKV=KeyValue2) = 0{code}, and return the sf1's scanner as the highest StoreFileScanner.
My solution is that:
If all the highest KeyValue read out from the StoreFileScanners are the same(compare return 0), then we should keep the Scanners orig order by sequenceId.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401294#comment-13401294 ]
Ted Yu commented on HBASE-6269:
-------------------------------
Can you generate patch for trunk for Hadoop aa ?
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402580#comment-13402580 ]
Hudson commented on HBASE-6269:
-------------------------------
Integrated in HBase-TRUNK #3083 (See [https://builds.apache.org/job/HBase-TRUNK/3083/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue (Xing Shi) (Revision 1354703)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "ShiXing (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ShiXing updated HBASE-6269:
---------------------------
Attachment: HBASE-6269-trunk-V1.patch
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-6269.
--------------------------------
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Fix For: 0.94.1, 0.96.0
>
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402523#comment-13402523 ]
Zhihong Ted Yu commented on HBASE-6269:
---------------------------------------
Integrated to trunk.
Thanks for the patch, Xing.
Thanks for the review, Ramkrishna and Annop.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402684#comment-13402684 ]
Hudson commented on HBASE-6269:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #72 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/72/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue (Xing Shi) (Revision 1354703)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402880#comment-13402880 ]
Hudson commented on HBASE-6269:
-------------------------------
Integrated in HBase-0.94 #286 (See [https://builds.apache.org/job/HBase-0.94/286/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue (Xing Shi) (Revision 1354815)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402833#comment-13402833 ]
ramkrishna.s.vasudevan commented on HBASE-6269:
-----------------------------------------------
@Ted
For 0.94? We need to apply this right?
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "ShiXing (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ShiXing updated HBASE-6269:
---------------------------
Attachment: HBASE-6269-v1.patch
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Status: Patch Available (was: Open)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402023#comment-13402023 ]
Hadoop QA commented on HBASE-6269:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533611/runAllTests.out.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2271//console
This message is automatically generated.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-6269:
------------------------------------------
Status: Patch Available (was: Open)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "Anoop Sam John (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401311#comment-13401311 ]
Anoop Sam John commented on HBASE-6269:
---------------------------------------
@ShiXing
In KeyValueHeap we use KVScannerComparator for the heap.
You can see the comparator checks for the 1st entries coming out of the scanners and in case of a tie, it checks the seq ids of the scanners.
This seqId of the StoreFileScanner will be the meta data "MAX_SEQ_ID_KEY" stored in that file. This value will be greater for the newest file resulting in this scanner coming on top. In the real functional scenarios this MAX_SEQ_ID will be keep on changing. Only the memstoreTS associated with each KV will be reset to 0.
So in the real functional scenario there wont be such a problem I think.
The newest file's scanner comes on top of the older ones.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Status: Open (was: Patch Available)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "ShiXing (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ShiXing updated HBASE-6269:
---------------------------
Attachment: runAllTests.out.txt
I run
mvn test -P runAllTests
and the output is in the runAllTests.out.txt
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan resolved HBASE-6269.
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.94.1
0.96.0
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 6269.94, HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch, runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401296#comment-13401296 ]
ramkrishna.s.vasudevan commented on HBASE-6269:
-----------------------------------------------
@ShiXing
If the data is not flushed we are able to get value1 which is latest. But when we flush we have this behavourial change like the StoreFileScanner(KeyValueHeap) gives us the older value?
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId
StoreFile's KeyValue as the latest KeyValue
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Ted Yu updated HBASE-6269:
----------------------------------
Status: Open (was: Patch Available)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the
maxSequenseId StoreFile's KeyValue as the latest KeyValue
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401325#comment-13401325 ]
ramkrishna.s.vasudevan commented on HBASE-6269:
-----------------------------------------------
@ShiXing
Yes, I agree that instead of getting the highest store file's scanner we get the second highest. And since in this case comparing for '<0' should be fine i feel.
Its better we fix this tho it may be very rare to get this problem.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: ShiXing
> Assignee: ShiXing
> Attachments: HBASE-6269-v1.patch
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp and different memstoreTS, after each Put/Increment, we do a memstore flush. So there will be two StoreFile with same KeyValue(except memstoreTS and SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List<KeyValue> kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap, and the KeyValue for latest StoreFile will comapre to the second latest StoreFile, but the second latest StoreFile generated the fake row for same row, family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira