You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2018/03/20 12:10:00 UTC
[jira] [Commented] (HBASE-20226) Performance Improvement Taking
Large Snapshots In Remote Filesystems
[ https://issues.apache.org/jira/browse/HBASE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406238#comment-16406238 ]
Hadoop QA commented on HBASE-20226:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 4s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 58s{color} | {color:red} hbase-server: The patch generated 13 new + 5 unchanged - 0 fixed = 18 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 5s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 16m 41s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s{color} | {color:red} hbase-server generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 58s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 9s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 16s{color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | v2Regions could be null and is guaranteed to be dereferenced in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() Dereferenced at SnapshotManifest.java:is guaranteed to be dereferenced in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() Dereferenced at SnapshotManifest.java:[line 549] |
| | Possible null pointer dereference of v1Regions in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() Dereferenced at SnapshotManifest.java:v1Regions in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() Dereferenced at SnapshotManifest.java:[line 516] |
| | Nullcheck of v1Regions at line 516 of value previously dereferenced in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() At SnapshotManifest.java:516 of value previously dereferenced in org.apache.hadoop.hbase.snapshot.SnapshotManifest.convertToV2SingleManifest() At SnapshotManifest.java:[line 516] |
| Failed junit tests | hadoop.hbase.snapshot.TestSnapshotManifest |
| | hadoop.hbase.master.snapshot.TestSnapshotHFileCleaner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20226 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12915302/HBASE-20226..01.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux e3c7548678b0 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/12036/artifact/patchprocess/diff-checkstyle-hbase-server.txt |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/12036/artifact/patchprocess/new-findbugs-hbase-server.html |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/12036/artifact/patchprocess/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12036/testReport/ |
| Max. process+thread count | 705 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12036/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Performance Improvement Taking Large Snapshots In Remote Filesystems
> --------------------------------------------------------------------
>
> Key: HBASE-20226
> URL: https://issues.apache.org/jira/browse/HBASE-20226
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
> Affects Versions: 1.4.0
> Environment: HBase 1.4.0 running on an AWS EMR cluster with the hbase.rootdir set to point to a folder in S3
> Reporter: Saad Mufti
> Priority: Minor
> Attachments: HBASE-20226..01.patch
>
>
> When taking a snapshot of any table, one of the last steps is to delete the region manifests, which have already been rolled up into a larger overall manifest and thus have redundant information.
> This proposal is to do the deletion in a thread pool bounded by hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the current single threaded deletion is taking longer than all the rest of the snapshot tasks when the Hbase data and the snapshot folder are both in a remote filesystem like S3.
> I have a patch for this proposal almost ready and will submit it tomorrow for feedback, although I haven't had a chance to write any tests yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)