You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Menno Luiten (JIRA)" <ji...@apache.org> on 2010/01/22 19:12:21 UTC
[jira] Created: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
--------------------------------------------------------------
Key: HBASE-2157
URL: https://issues.apache.org/jira/browse/HBASE-2157
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.2
Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Reporter: Menno Luiten
I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "Menno Luiten (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803799#action_12803799 ]
Menno Luiten commented on HBASE-2157:
-------------------------------------
BTW: Solved the issue by using the constructor including timestamp and passing System.currentTimeMillis() manually. Works fine.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "Menno Luiten (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803802#action_12803802 ]
Menno Luiten commented on HBASE-2157:
-------------------------------------
That explains it :) Cost me hours figuring it out :D Could this be solved by adding the check/replacement to HFileOutputFormat as well?
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2157:
-------------------------
Attachment: 2157.patch
I added note to the bulk load documentation that warns against creating KVs w/o explicit timestamp so others don't trip over Menno's issue.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
> Attachments: 2157.patch
>
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2157.
--------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
0.20.3
Assignee: stack
Committed branch and trunk.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
> Assignee: stack
> Fix For: 0.20.3, 0.21.0
>
> Attachments: 2157.patch
>
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803825#action_12803825 ]
stack commented on HBASE-2157:
------------------------------
I don't think adding a check/replacement to HFOF will work since this is after the sort done up in the KeyValueSortReducer. Rather, we could have it throw an exception if HFOF is passed a KV with LATEST_TIMESTAMP? I'll open a new issue to do that.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
> Assignee: stack
> Fix For: 0.20.3, 0.21.0
>
> Attachments: 2157.patch
>
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803809#action_12803809 ]
Jean-Daniel Cryans commented on HBASE-2157:
-------------------------------------------
Yes, I guess it would make sense.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2157) LATEST_TIMESTAMP not replaced by
current timestamp in KeyValue
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803800#action_12803800 ]
Jean-Daniel Cryans commented on HBASE-2157:
-------------------------------------------
Right the replacement is done in the region server and in this case it gets bypassed.
> LATEST_TIMESTAMP not replaced by current timestamp in KeyValue
> --------------------------------------------------------------
>
> Key: HBASE-2157
> URL: https://issues.apache.org/jira/browse/HBASE-2157
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0 - Hbase 0.20.2 - Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
> Reporter: Menno Luiten
>
> I was trying to bulk load using the new HFileOutputFormat. When using a MapReduce in which map generates {{KeyValue}}s and reduce is equal to KeyValueSortReducer, and using the constructor using (byte[] row, byte[] family, byte[] qualifier, byte[] value), the (undefined) timestamp was inserted as HConstants.LATEST_TIMESTAMP/Long.MAX_VALUE into HBase. This causes all kinds of troubles, but most importantly, while the records were in the table, other MapReduces (using TableInputFormat) and Hbase shell's 'get'-command did not fetch them. Guess there is some sort of filtering of future dates.
> As I understood from St.Ack, the LASTEST_TIMESTAMP is supposed to be replaced by System.currentTimeMillis(), but I don't see this reflected in the code of KeyValue, and apparently it did not happen elsewhere; perhaps because there is no actual HBase connection?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.