You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/02/28 12:45:38 UTC
[GitHub] [hbase] infraio opened a new pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
infraio opened a new pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223
…scriptor directly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r385989245
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/ReadOnlyFSTableDescriptors.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class ReadOnlyFSTableDescriptors extends FSTableDescriptors {
Review comment:
Because FSTableDescriptor's constructer method throws IOException now. Let me open a new issue to refactor FSTableDescriptor firstly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-595118207
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 32s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | checkstyle | 1m 11s | master passed |
| +1 :green_heart: | shadedjars | 4m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
| +0 :ok: | spotbugs | 5m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 14s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 4s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 50s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 47s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 39s | the patch passed |
| +1 :green_heart: | findbugs | 5m 31s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 69m 3s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 130m 25s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1223 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 20664c0b1fdb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1223/out/precommit/personality/provided.sh |
| git revision | master / 9bd39786df |
| Default Java | 1.8.0_181 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/6/testReport/ |
| Max. process+thread count | 8541 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/6/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594280531
I think cache is default on at master side? No?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-593800761
I fixed this issue in my local branch, do you mind reviewing that? @infraio @Apache9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594321870
> > Reading table descriptor always from FS wont be good actually, it will cause many RPC to Namenode for same operation from all RS. (Please correct me if wrong)
>
> Read from master still need rpc to master?
Pankaj: Yes, but it wont overload NN. In a large cluster scenario, there will be huge NN RPC if each RS read the table descriptor from FS indivdually. IMHO, we should reduce redundant RPC call to NN.
>
> > Currently HMaster already load the table decriptors during startup and RS retrieve it. IMO RS should read table descriptor from FS directly only when it fails to retrieve from HM due to incomplete initialization.
>
> Master use FSTableDesctriptors but not use cache now. It need to retrieve from NN too.
Pankaj: Master enable the cache during init.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r387445203
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
##########
@@ -114,20 +118,24 @@
@VisibleForTesting
WALSplitter(final WALFactory factory, Configuration conf, Path walDir, FileSystem walFS,
- LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination) {
+ Path rootDir, FileSystem rootFS, LastSequenceId idChecker,
+ SplitLogWorkerCoordination splitLogWorkerCoordination) {
this.conf = HBaseConfiguration.create(conf);
String codecClassName =
conf.get(WALCellCodec.WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class.getName());
this.conf.set(HConstants.RPC_CODEC_CONF_KEY, codecClassName);
this.walDir = walDir;
this.walFS = walFS;
+ this.rootDir = rootDir;
+ this.rootFS = rootFS;
this.sequenceIdChecker = idChecker;
this.splitLogWorkerCoordination = splitLogWorkerCoordination;
this.walFactory = factory;
PipelineController controller = new PipelineController();
this.tmpDirName =
conf.get(HConstants.TEMPORARY_FS_DIRECTORY_KEY, HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY);
+ this.tableDescriptors = new FSTableDescriptors(rootFS, rootDir, true, true);
Review comment:
We should reuse the RegionServer FSTableDescriptors instead of creating FSTableDescriptors instance for each WAL split.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-592547871
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 42s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | checkstyle | 1m 16s | master passed |
| +1 :green_heart: | shadedjars | 4m 45s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
| +0 :ok: | spotbugs | 4m 26s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 56s | the patch passed |
| +1 :green_heart: | compile | 1m 0s | the patch passed |
| +1 :green_heart: | javac | 1m 0s | the patch passed |
| -1 :x: | checkstyle | 1m 13s | hbase-server: The patch generated 16 new + 188 unchanged - 3 fixed = 204 total (was 191) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedjars | 4m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 1s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| -1 :x: | javadoc | 0m 37s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | findbugs | 4m 28s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 63m 50s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 123m 31s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.wal.TestWALSplit |
| | hadoop.hbase.wal.TestWALSplitCompressed |
| | hadoop.hbase.wal.TestWALSplitBoundedLogWriterCreation |
| | hadoop.hbase.wal.TestWALSplitToHFile |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1223 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 80c1b6f4bcd9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1223/out/precommit/personality/provided.sh |
| git revision | master / 3c3aae9a2f |
| Default Java | 1.8.0_181 |
| checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/artifact/out/diff-checkstyle-hbase-server.txt |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/artifact/out/whitespace-eol.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/artifact/out/diff-javadoc-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/testReport/ |
| Max. process+thread count | 8971 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r387445203
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
##########
@@ -114,20 +118,24 @@
@VisibleForTesting
WALSplitter(final WALFactory factory, Configuration conf, Path walDir, FileSystem walFS,
- LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination) {
+ Path rootDir, FileSystem rootFS, LastSequenceId idChecker,
+ SplitLogWorkerCoordination splitLogWorkerCoordination) {
this.conf = HBaseConfiguration.create(conf);
String codecClassName =
conf.get(WALCellCodec.WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class.getName());
this.conf.set(HConstants.RPC_CODEC_CONF_KEY, codecClassName);
this.walDir = walDir;
this.walFS = walFS;
+ this.rootDir = rootDir;
+ this.rootFS = rootFS;
this.sequenceIdChecker = idChecker;
this.splitLogWorkerCoordination = splitLogWorkerCoordination;
this.walFactory = factory;
PipelineController controller = new PipelineController();
this.tmpDirName =
conf.get(HConstants.TEMPORARY_FS_DIRECTORY_KEY, HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY);
+ this.tableDescriptors = new FSTableDescriptors(rootFS, rootDir, true, true);
Review comment:
We should reuse the RegionServer FSTableDescriptors instead of creating FSTableDescriptors instance for each WAL split.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594330054
> > I think cache is default on at master side? No?
>
> Yes, during HMaster init we enable it,
>
> https://github.com/apache/hbase/blob/29ea96fad8aea9432836af9e9080d0d35a488ab7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L916
OK. When new FSTableDescrpitor, the cache paramerte is false...... And setCacheOn later...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594320753
> I think cache is default on at master side? No?
Yes, during HMaster init we enable it,
https://github.com/apache/hbase/blob/29ea96fad8aea9432836af9e9080d0d35a488ab7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L916
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-593821890
> I fixed this issue in my local branch, do you mind reviewing that? @infraio @Apache9
Feel free to created a new PR? I didn't finish this one because I want resolve HBASE-23912 firstly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594279830
> Reading table descriptor always from FS wont be good actually, it will cause many RPC to Namenode for same operation from all RS. (Please correct me if wrong)
Read from master still need rpc to master?
> Currently HMaster already load the table decriptors during startup and RS retrieve it. IMO RS should read table descriptor from FS directly only when it fails to retrieve from HM due to incomplete initialization.
Master use FSTableDesctriptors but not use cache now. It need to retrieve from NN too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-593061669
Reading table descriptor always from FS wont be good actually, it will cause many RPC to Namenode for same operation from all RS. (Please correct me if wrong)
Currently HMaster already load the table decriptors during startup and RS retrieve it. IMO RS should read table descriptor from FS directly only when it fails to retrieve from HM due to incomplete initialization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio merged pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio merged pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio opened a new pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio opened a new pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223
…scriptor directly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-594316834
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 28s | master passed |
| +1 :green_heart: | compile | 0m 57s | master passed |
| +1 :green_heart: | checkstyle | 1m 10s | master passed |
| +1 :green_heart: | shadedjars | 4m 49s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
| +0 :ok: | spotbugs | 5m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 9s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 5s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 8s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 49s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 15m 54s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
| +1 :green_heart: | findbugs | 5m 12s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 63m 33s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 123m 8s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.wal.TestWALSplitToHFile |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1223 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 712b7af81728 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1223/out/precommit/personality/provided.sh |
| git revision | master / f411e395d3 |
| Default Java | 1.8.0_181 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/3/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/3/testReport/ |
| Max. process+thread count | 8668 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/3/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r385992441
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/ReadOnlyFSTableDescriptors.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class ReadOnlyFSTableDescriptors extends FSTableDescriptors {
Review comment:
OK, in FSTableDescriptor we need to write the meta table descriptor to fs so it will throw an IOException. Got it. New issue is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r385983902
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##########
@@ -721,20 +720,20 @@ private void initializeFileSystem() throws IOException {
}
private TableDescriptors getFsTableDescriptors() throws IOException {
- return new FSTableDescriptors(this.conf,
- this.dataFs, this.dataRootDir, !canUpdateTableDescriptor(), false, getMetaTableObserver());
- }
-
- protected Function<TableDescriptorBuilder, TableDescriptorBuilder> getMetaTableObserver() {
- return null;
+ if (canUpdateTableDescriptor()) {
+ return new FSTableDescriptors(this.conf, this.dataFs, this.dataRootDir, false,
+ builder -> builder.setRegionReplication(
Review comment:
This changes the semantic? Not sure why but before this patch we only set this for HMaster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-592810938
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 37s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | checkstyle | 1m 14s | master passed |
| +1 :green_heart: | shadedjars | 4m 44s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | master passed |
| +0 :ok: | spotbugs | 4m 31s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 30s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 0s | the patch passed |
| +1 :green_heart: | javac | 1m 0s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 188 unchanged - 3 fixed = 188 total (was 191) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 47s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 4s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| -1 :x: | javadoc | 0m 36s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | findbugs | 4m 27s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 66m 22s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 125m 18s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.wal.TestWALSplitToHFile |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1223 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 49f2a811bae6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1223/out/precommit/personality/provided.sh |
| git revision | master / a420f0482e |
| Default Java | 1.8.0_181 |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/2/artifact/out/diff-javadoc-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/2/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/2/testReport/ |
| Max. process+thread count | 8752 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/2/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-595041198
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 5s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | checkstyle | 1m 21s | master passed |
| +1 :green_heart: | shadedjars | 5m 19s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
| +0 :ok: | spotbugs | 5m 18s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 16s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 6s | the patch passed |
| +1 :green_heart: | compile | 1m 8s | the patch passed |
| +1 :green_heart: | javac | 1m 8s | the patch passed |
| -1 :x: | checkstyle | 1m 18s | hbase-server: The patch generated 6 new + 48 unchanged - 0 fixed = 54 total (was 48) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 5m 17s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 8s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 38s | the patch passed |
| +1 :green_heart: | findbugs | 5m 44s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 25m 35s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 89m 31s | |
| Reason | Tests |
|-------:|:------|
| Unreaped Processes | hbase-server:7 |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1223 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux eb6014e7d3cd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1223/out/precommit/personality/provided.sh |
| git revision | master / 9bd39786df |
| Default Java | 1.8.0_181 |
| checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/artifact/out/diff-checkstyle-hbase-server.txt |
| Unreaped Processes Log | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/artifact/out/patch-unit-hbase-server-reaper.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/testReport/ |
| Max. process+thread count | 8006 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1223/5/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
infraio commented on issue #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#issuecomment-595154742
Ok. The HBase QA passed...... @pankaj72981 Any more comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223#discussion_r385984153
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/ReadOnlyFSTableDescriptors.java
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class ReadOnlyFSTableDescriptors extends FSTableDescriptors {
Review comment:
I think FSTableDescriptor should extends ReadOnlyFSTableDescriptor? And what is the problem for the old implementation where we just use a flag?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] pankaj72981 closed pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
Posted by GitBox <gi...@apache.org>.
pankaj72981 closed pull request #1223: HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table de…
URL: https://github.com/apache/hbase/pull/1223
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services