You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Amitanand Aiyer (Created) (JIRA)" <ji...@apache.org> on 2011/10/21 16:46:32 UTC
[jira] [Created] (HBASE-4645) Edits Log recovery losing data across
column families
Edits Log recovery losing data across column families
-----------------------------------------------------
Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132903#comment-13132903 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2755
-----------------------------------------------------------
Wow... Losing data?! And that went unnoticed for so long.
I guess I don't understand when the store's maxid go out of sync and why this does not happen all the time.
Nice find!!
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/2524/#comment6196>
Good comment. Makes sense.
Does that mean that up to now we could have failed to replay some of the logs and actually lose data?!
(I guess, yes, that's how you describe the problem)
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
<https://reviews.apache.org/r/2524/#comment6198>
Is the because before your change it would not need to replay any logs, but now it does?
- Lars
On 2011-10-21 18:06:16, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 18:06:16)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4645) Edits Log recovery losing data across
column families
Posted by "Nicolas Spiegelberg (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-4645:
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
0.92.0
Status: Resolved (was: Patch Available)
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4645-v9.diff
>
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133091#comment-13133091 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 21:45:49.683712)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
fix whitespace errors.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133138#comment-13133138 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2765
-----------------------------------------------------------
Ship it!
update test looks good!
- Kannan
On 2011-10-21 22:21:58, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 22:21:58)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133122#comment-13133122 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 22:14:22.830521)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
delete the middle cf.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132894#comment-13132894 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2756
-----------------------------------------------------------
Amit-- code changes look solid.
It is unfortunate that there was a nice test TestWALReplay.java to test this exact case-- but was broken in catching this issue.
Furthermore, the test writes to multiple CFs in separate puts (transactions). It will be good to enhance it to do cross-CF puts (in a single txn).
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
<https://reviews.apache.org/r/2524/#comment6197>
looks like the original test is busted. It expects to flush only one region (in line 271); but here region.close() will cause it to flush all stores :)
- Kannan
On 2011-10-21 18:06:16, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 18:06:16)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132875#comment-13132875 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 18:01:32.703554)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
(i) Address kannan's comments.
(ii) Update the TestWALReplay.
Turns out that it already had a test case to check for what
we want. We were looking for the wrong behavior. Fixed it.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132877#comment-13132877 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 18:03:54.336799)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
fix white space issue.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133133#comment-13133133 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 22:21:58.484290)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing (updated)
-------
Initial patch. v1.
mvn test (running).
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132919#comment-13132919 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 18:49:45.916474)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
ensure that when we are testing region.close does not flush the stores.
pass abort=true.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133095#comment-13133095 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 21:46:51.829806)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
get rid of some unintended comments.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133105#comment-13133105 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
bq. On 2011-10-21 18:31:16, Lars Hofhansl wrote:
bq. > Wow... Losing data?! And that went unnoticed for so long.
bq. > I guess I don't understand when the store's maxid go out of sync and why this does not happen all the time.
bq. >
bq. > Nice find!!
currently, we seem to flush all the column families together.
For this failure scenario to kick in, there has to be a failure after *some* stores have flushed. But, not
all of them.
Unclean shutdowns are rare. That too in the middle of the flushing wasn't too common.
But, important to fix, given that we now know about it.
bq. On 2011-10-21 18:31:16, Lars Hofhansl wrote:
bq. > src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java, line 289
bq. > <https://reviews.apache.org/r/2524/diff/4/?file=52155#file52155line289>
bq. >
bq. > Is the because before your change it would not need to replay any logs, but now it does?
yes. earlier we would not have replayed any logs. The test was ensuring that we don't.
Now that we do our decision based on the minimum across different Stores, we do end
up replaying some edits.
- Amitanand
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2755
-----------------------------------------------------------
On 2011-10-21 21:46:51, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 21:46:51)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4645) Edits Log recovery losing data across
column families
Posted by "Amitanand Aiyer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amitanand Aiyer updated HBASE-4645:
-----------------------------------
Attachment: 4645-v9.diff
v9 of the revision from https://reviews.apache.org/r/2524/diff
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
> Attachments: 4645-v9.diff
>
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133151#comment-13133151 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2767
-----------------------------------------------------------
Ship it!
Thanks for the explanation Amit.
- Lars
On 2011-10-21 22:21:58, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 22:21:58)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132726#comment-13132726 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132879#comment-13132879 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 18:06:16.651615)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Changes
-------
missed the white space errors in TestWALReplay. fixed.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132745#comment-13132745 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
bq. On 2011-10-21 15:20:37, Kannan Muthukkaruppan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 424
bq. > <https://reviews.apache.org/r/2524/diff/1/?file=52115#file52115line424>
bq. >
bq. > as Madhu pointed out in a later email, if now edits were were there for this region, then this function will simply return minSeqId back as its return value.
bq. >
bq. > Like before, we should also track the maxSeqId for all stores in the above loop.
bq. >
bq. > And, then change this line to something like:
bq. >
bq. > maxSeqId = Math.max(maxSeqId, replayRecoveredEditsIfAny(...));
bq. >
bq. >
typo: s/if now edits were/if no edits were
- Kannan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2745
-----------------------------------------------------------
On 2011-10-21 15:01:11, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 15:01:11)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133086#comment-13133086 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
-----------------------------------------------------------
(Updated 2011-10-21 21:43:17.815733)
Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
Summary
-------
There is a data loss happening (for some of the column families) when we do the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the column
families were flushed), then we lose data for the column families that have not yet
caught up.
The correct logic for replay should begin the replay from the minimum across the
maximum in each store.
This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 966262b
Diff: https://reviews.apache.org/r/2524/diff
Testing
-------
Initial patch. v1.
mvn test (running).
TBD: add a test case to repro the issue and make sure it fixes.
Thanks,
Amitanand
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136316#comment-13136316 ]
Hudson commented on HBASE-4645:
-------------------------------
Integrated in HBase-0.92 #82 (See [https://builds.apache.org/job/HBase-0.92/82/])
HBASE-4645 Edits Log recovery losing data across column families
nspiegelberg :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
> Attachments: 4645-v9.diff
>
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136175#comment-13136175 ]
Hudson commented on HBASE-4645:
-------------------------------
Integrated in HBase-TRUNK #2370 (See [https://builds.apache.org/job/HBase-TRUNK/2370/])
HBASE-4645 Edits Log recovery losing data across column families
nspiegelberg :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
> Attachments: 4645-v9.diff
>
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132794#comment-13132794 ]
Ted Yu commented on HBASE-4645:
-------------------------------
The TODO should refer to HBASE-3149
I will add a cross reference in HBASE-3149 to this JIRA.
Nice job, Amit.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4645) Edits Log recovery losing data across
column families
Posted by "Amitanand Aiyer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amitanand Aiyer updated HBASE-4645:
-----------------------------------
Status: Patch Available (was: Open)
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
> Attachments: 4645-v9.diff
>
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data
across column families
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132741#comment-13132741 ]
jiraposter@reviews.apache.org commented on HBASE-4645:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2745
-----------------------------------------------------------
Amit-- thanks for getting a fix out soon. Comments inline. As you mentioned, I am assuming you are planning on adding a test as part of the commit itself.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/2524/#comment6180>
Comment:
Get minimum seqid across all store
should read:
Get minimum of the max seq id for each store.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/2524/#comment6181>
trailing white space
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/2524/#comment6183>
yes, makes sense as a future optimization.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/2524/#comment6184>
as Madhu pointed out in a later email, if now edits were were there for this region, then this function will simply return minSeqId back as its return value.
Like before, we should also track the maxSeqId for all stores in the above loop.
And, then change this line to something like:
maxSeqId = Math.max(maxSeqId, replayRecoveredEditsIfAny(...));
- Kannan
On 2011-10-21 15:01:11, Amitanand Aiyer wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-10-21 15:01:11)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. There is a data loss happening (for some of the column families) when we do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
bq. column family is ahead of others (because the crash happened before all the column
bq. families were flushed), then we lose data for the column families that have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.
> Edits Log recovery losing data across column families
> -----------------------------------------------------
>
> Key: HBASE-4645
> URL: https://issues.apache.org/jira/browse/HBASE-4645
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.89.20100924, 0.92.0
> Reporter: Amitanand Aiyer
> Assignee: Amitanand Aiyer
>
> There is a data loss happening (for some of the column families) when we do the replay logs.
> The bug seems to be from the fact that during replay-logs we only choose to replay
> the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
> column family is ahead of others (because the crash happened before all the column
> families were flushed), then we lose data for the column families that have not yet
> caught up.
> The correct logic for replay should begin the replay from the minimum across the
> maximum in each store.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira