You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2017/06/16 03:38:00 UTC
[jira] [Commented] (HBASE-18133) Low-latency space quota size
reports
[ https://issues.apache.org/jira/browse/HBASE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051330#comment-16051330 ]
Hadoop QA commented on HBASE-18133:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 12 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s {color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s {color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 29m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s {color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s {color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 48s {color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s {color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 23s {color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestKeepDeletes |
| | hadoop.hbase.io.encoding.TestSeekBeforeWithReverseScan |
| | hadoop.hbase.regionserver.TestScanWithBloomError |
| | hadoop.hbase.filter.TestDependentColumnFilter |
| | hadoop.hbase.regionserver.TestMinVersions |
| | hadoop.hbase.regionserver.TestScanner |
| | hadoop.hbase.regionserver.TestStoreFileRefresherChore |
| | hadoop.hbase.coprocessor.TestRegionObserverStacking |
| | hadoop.hbase.regionserver.TestBlocksScanned |
| | hadoop.hbase.regionserver.TestResettingCounters |
| | hadoop.hbase.coprocessor.TestCoprocessorInterface |
| | hadoop.hbase.filter.TestFilter |
| | hadoop.hbase.filter.TestFilterFromRegionSide |
| | hadoop.hbase.io.encoding.TestPrefixTree |
| | hadoop.hbase.client.TestIntraRowPagination |
| | hadoop.hbase.filter.TestInvocationRecordFilter |
| | hadoop.hbase.filter.TestColumnPrefixFilter |
| | hadoop.hbase.regionserver.TestWideScanner |
| | hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange |
| | hadoop.hbase.filter.TestMultipleColumnPrefixFilter |
| | hadoop.hbase.regionserver.TestWALMonotonicallyIncreasingSeqId |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12873207/HBASE-18133.001.patch |
| JIRA Issue | HBASE-18133 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c77929819f30 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
| git revision | master / dd1d81e |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/7199/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/7199/artifact/patchprocess/patch-unit-hbase-server.txt |
| unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/7199/artifact/patchprocess/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/7199/testReport/ |
| modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/7199/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Low-latency space quota size reports
> ------------------------------------
>
> Key: HBASE-18133
> URL: https://issues.apache.org/jira/browse/HBASE-18133
> Project: HBase
> Issue Type: Improvement
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: 3.0.0
>
> Attachments: HBASE-18133.001.patch
>
>
> Presently space quota enforcement relies on RegionServers sending reports to the master about each Region that they host. This is done by periodically, reading the cached size of each HFile in each Region (which was ultimately computed from HDFS).
> This means that the Master is unaware of Region size growth until the the next time this chore in a RegionServer fires which is a fair amount of latency (a few minutes, by default). Operations like flushes, compactions, and bulk-loads are delayed even though the RegionServer is running those operations locally.
> Instead, we can create an API which these operations could invoke that would automatically update the size of the Region being operated on. For example, a successful flush can report that the size of a Region increased by the size of the flush. A compaction can subtract the size of the input files of the compaction and add in the size of the resulting file.
> This de-couples the computation of a Region's size from sending the Region sizes to the Master, allowing us to send reports more frequently, increasing the responsiveness of the cluster to size changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)