You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gregory Chanan (JIRA)" <ji...@apache.org> on 2012/10/04 23:23:47 UTC
[jira] [Created] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Gregory Chanan created HBASE-6950:
-------------------------------------
Summary: TestAcidGuarantees system test now flushes too aggressively
Key: HBASE-6950
URL: https://issues.apache.org/jira/browse/HBASE-6950
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.2, 0.94.2, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
Fix For: 0.96.0
HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
Here is my comment in the (upcoming) patch:
{code}
// Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
// we would flush as often as possible. On a running cluster, this isn't practical:
// (1) we will cause a lot of load due to all the flushing and compacting
// (2) we cannot change the flushing/compacting related Configuration options to try to
// alleviate this
// (3) it is an unrealistic workload, since no one would actually flush that often.
// Therefore, let's flush every minute to have more flushes than usual, but not overload
// the running cluster.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469788#comment-13469788 ]
Hadoop QA commented on HBASE-6950:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12547827/HBASE-6950.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 81 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3008//console
This message is automatically generated.
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469854#comment-13469854 ]
Gregory Chanan commented on HBASE-6950:
---------------------------------------
Thanks for the review, Ted.
Committed to trunk.
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6950:
----------------------------------
Status: Patch Available (was: Open)
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6950:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6950:
----------------------------------
Attachment: HBASE-6950.patch
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469961#comment-13469961 ]
Hudson commented on HBASE-6950:
-------------------------------
Integrated in HBase-TRUNK #3427 (See [https://builds.apache.org/job/HBase-TRUNK/3427/])
HBASE-6950 TestAcidGuarantees system test now flushes too aggressively (Revision 1394335)
Result = FAILURE
gchanan :
Files :
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470243#comment-13470243 ]
Hudson commented on HBASE-6950:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #210 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/210/])
HBASE-6950 TestAcidGuarantees system test now flushes too aggressively (Revision 1394335)
Result = FAILURE
gchanan :
Files :
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6950) TestAcidGuarantees system test now
flushes too aggressively
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469790#comment-13469790 ]
Ted Yu commented on HBASE-6950:
-------------------------------
{code}
Running org.apache.hadoop.hbase.TestAcidGuarantees
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.275 sec
{code}
+1 on patch.
Test failure is unrelated.
> TestAcidGuarantees system test now flushes too aggressively
> -----------------------------------------------------------
>
> Key: HBASE-6950
> URL: https://issues.apache.org/jira/browse/HBASE-6950
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.2, 0.94.2, 0.96.0
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6950.patch
>
>
> HBASE-6552 caused the TestAcidGuarantees system test to flush more aggressively, because flushes are where ACID problems have occurred in the past.
> After some more cluster testing, it seems like this too aggressive; my clusters eventually can't keep up with the number of flushes/compactions and start getting SocketTimeoutExceptions. We could try to optimize the flushes/compactions, but since this workload would never occur in practice, I don't think it is worth the effort. Instead, let's just only flush once a minute. This is arbitrary, but seems to work.
> Here is my comment in the (upcoming) patch:
> {code}
> // Flushing has been a source of ACID violations previously (see HBASE-2856), so ideally,
> // we would flush as often as possible. On a running cluster, this isn't practical:
> // (1) we will cause a lot of load due to all the flushing and compacting
> // (2) we cannot change the flushing/compacting related Configuration options to try to
> // alleviate this
> // (3) it is an unrealistic workload, since no one would actually flush that often.
> // Therefore, let's flush every minute to have more flushes than usual, but not overload
> // the running cluster.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira