You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Eugene Morozov (JIRA)" <ji...@apache.org> on 2012/09/22 00:36:08 UTC
[jira] [Created] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Eugene Morozov created HBASE-6861:
-------------------------------------
Summary: HFileOutputFormat set TIMERANGE wrongly
Key: HBASE-6861
URL: https://issues.apache.org/jira/browse/HBASE-6861
Project: HBase
Issue Type: Bug
Reporter: Eugene Morozov
Priority: Minor
In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
Example (in pseudo code): my reducer has a condition
if (condition ) {
keyValue = new KeyValue(.., CF1, .., timestamp, ..);
} else {
keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
}
context.write( keyValue );
These two keyValues would be written into two different HFiles.
But the code, which is actually write do the following:
// we now have the proper HLog writer. full steam ahead
kv.updateLatestStamp(this.now);
trt.includeTimestamp(kv);
wl.writer.append(kv);
Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "Eugene Morozov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465475#comment-13465475 ]
Eugene Morozov commented on HBASE-6861:
---------------------------------------
Basically yes, but it looks more like hack, than patch. I believe it's not correct to commit it to trunk. Do you still want me to share it?
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "Eugene Morozov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugene Morozov updated HBASE-6861:
----------------------------------
Attachment: diff
This is how I've fixed this issue for myself.
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
> Attachments: diff
>
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "Eugene Morozov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugene Morozov updated HBASE-6861:
----------------------------------
Description:
In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
Example (in pseudo code): my reducer has a condition
if ( condition ) {
keyValue = new KeyValue(.., CF1, .., timestamp, ..);
} else {
keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
}
context.write( keyValue );
These two keyValues would be written into two different HFiles.
But the code, which is actually write do the following:
// we now have the proper HLog writer. full steam ahead
kv.updateLatestStamp(this.now);
trt.includeTimestamp(kv);
wl.writer.append(kv);
Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
was:
In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
Example (in pseudo code): my reducer has a condition
if (condition ) {
keyValue = new KeyValue(.., CF1, .., timestamp, ..);
} else {
keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
}
context.write( keyValue );
These two keyValues would be written into two different HFiles.
But the code, which is actually write do the following:
// we now have the proper HLog writer. full steam ahead
kv.updateLatestStamp(this.now);
trt.includeTimestamp(kv);
wl.writer.append(kv);
Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465032#comment-13465032 ]
stack commented on HBASE-6861:
------------------------------
You have a patch Eugene? Thanks.
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466897#comment-13466897 ]
Ted Yu commented on HBASE-6861:
-------------------------------
@Eugene:
Consider using the formatter from HBASE-5961.
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
> Attachments: diff
>
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6861) HFileOutputFormat set TIMERANGE
wrongly
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465751#comment-13465751 ]
stack commented on HBASE-6861:
------------------------------
I'd suggest adding the patch here anyways. It could inspire a patch that would get committed to trunk. Thanks.
> HFileOutputFormat set TIMERANGE wrongly
> ---------------------------------------
>
> Key: HBASE-6861
> URL: https://issues.apache.org/jira/browse/HBASE-6861
> Project: HBase
> Issue Type: Bug
> Reporter: Eugene Morozov
> Priority: Minor
>
> In case if timestamps for KeyValues specified differently for different column families, then TIMERANGEs of both HFiles would be wrong.
> Example (in pseudo code): my reducer has a condition
> if ( condition ) {
> keyValue = new KeyValue(.., CF1, .., timestamp, ..);
> } else {
> keyValue = new KeyValue(.., CF2, .., ..); // <- no timestamp
> }
> context.write( keyValue );
> These two keyValues would be written into two different HFiles.
> But the code, which is actually write do the following:
> // we now have the proper HLog writer. full steam ahead
> kv.updateLatestStamp(this.now);
> trt.includeTimestamp(kv);
> wl.writer.append(kv);
> Basically, two HFiles shares the same instance of trt (TimeRangeTracker), which leads to the same TIMERANGEs of both of them. Which is definitely incorrect, because first HFile must have TIMERANGE=timestamp...timestamp, cause we do not write any other timestamps there. And another HFile must have TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira