You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "chunhui shen (Created) (JIRA)" <ji...@apache.org> on 2011/11/27 07:54:41 UTC
[jira] [Created] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Master crash when spliting hlog may cause data loss
---------------------------------------------------
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus("Finishing writing output logs and closing down.");
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when splitting hlog may
cause data loss
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4878:
--------------------------
Summary: Master crash when splitting hlog may cause data loss (was: Master crash when spliting hlog may cause data loss)
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "chunhui shen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-4878:
--------------------------------
Attachment: hbase-4878.diff
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Reporter: chunhui shen
> Attachments: hbase-4878.diff
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when splitting hlog
may cause data loss
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158993#comment-13158993 ]
Hudson commented on HBASE-4878:
-------------------------------
Integrated in HBase-0.92-security #22 (See [https://builds.apache.org/job/HBase-0.92-security/22/])
HBASE-4878 Master crash when splitting hlog may cause data loss (Chunhui Shen)
tedyu :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when splitting hlog may
cause data loss
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4878:
--------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when splitting hlog
may cause data loss
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159785#comment-13159785 ]
Hudson commented on HBASE-4878:
-------------------------------
Integrated in HBase-0.92 #163 (See [https://builds.apache.org/job/HBase-0.92/163/])
HBASE-4878 Master crash when splitting hlog may cause data loss (Chunhui Shen)
tedyu :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158208#comment-13158208 ]
ramkrishna.s.vasudevan commented on HBASE-4878:
-----------------------------------------------
+1 on patch.
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Reporter: chunhui shen
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when splitting hlog
may cause data loss
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159129#comment-13159129 ]
Hudson commented on HBASE-4878:
-------------------------------
Integrated in HBase-TRUNK-security #13 (See [https://builds.apache.org/job/HBase-TRUNK-security/13/])
HBASE-4878 Master crash when splitting hlog may cause data loss (Chunhui Shen)
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "chunhui shen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157683#comment-13157683 ]
chunhui shen commented on HBASE-4878:
-------------------------------------
I think the correct correct is
putting the line archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
in the end
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Reporter: chunhui shen
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158286#comment-13158286 ]
Hadoop QA commented on HBASE-4878:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505294/hbase-4878v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -162 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestInstantSchemaChange
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/390//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/390//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/390//console
This message is automatically generated.
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when splitting hlog
may cause data loss
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158879#comment-13158879 ]
Hudson commented on HBASE-4878:
-------------------------------
Integrated in HBase-TRUNK #2494 (See [https://builds.apache.org/job/HBase-TRUNK/2494/])
HBASE-4878 Master crash when splitting hlog may cause data loss (Chunhui Shen)
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158685#comment-13158685 ]
Ted Yu commented on HBASE-4878:
-------------------------------
Looks like asf000.sp2.ygridcore.net was able to perform test suite validation.
The failed tests were due to 'Too many open files'
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "chunhui shen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-4878:
--------------------------------
Attachment: hbase-4878v2.patch
@ramkrishna
Have done in patchv2
Thanks.
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Reporter: chunhui shen
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when splitting hlog
may cause data loss
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158711#comment-13158711 ]
Ted Yu commented on HBASE-4878:
-------------------------------
I ran TestHLogSplit and TestHLog which passed.
Integrated to 0.92 and TRUNK.
Thanks for the patch Chunhui.
Thanks for the review Ramkrishna.
> Master crash when splitting hlog may cause data loss
> ----------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158199#comment-13158199 ]
ramkrishna.s.vasudevan commented on HBASE-4878:
-----------------------------------------------
@Chunhui
{code}
status.setStatus("Archiving logs after completed split");
{code}
also needs to be moved before
{code}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
{code}
Otherwise the status will be interleaved with
"Finishing writing output logs and closing down."
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Reporter: chunhui shen
> Attachments: hbase-4878.diff
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4878:
------------------------------------------
Assignee: chunhui shen
Affects Version/s: 0.92.0
Status: Patch Available (was: Open)
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may
cause data loss
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4878:
--------------------------
Fix Version/s: 0.94.0
0.92.0
> Master crash when spliting hlog may cause data loss
> ---------------------------------------------------
>
> Key: HBASE-4878
> URL: https://issues.apache.org/jira/browse/HBASE-4878
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.92.0, 0.94.0
>
> Attachments: hbase-4878.diff, hbase-4878v2.patch
>
>
> Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
> {code}
> private List<Path> splitLog(final FileStatus[] logfiles) throws IOException {
> try {
> for (FileStatus log : logfiles) {
> parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
> }
> archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
> } finally {
> status.setStatus("Finishing writing output logs and closing down.");
> splits = outputSink.finishWritingAndClose();
> }
> }
> {code}
> If master is killed, after finishing archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf),
> but before finishing splits = outputSink.finishWritingAndClose();
> Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira