You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/04/18 00:41:43 UTC
[GitHub] [hbase] arshadmohammad opened a new pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
arshadmohammad opened a new pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546
Create table is slow in large cluster when AccessController is enabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410693331
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
+ return new ArrayList<>();
Review comment:
returning `Collections.emptyList()` would be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] arshadmohammad commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410716897
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
Review comment:
getChildDataAndWatchForNewChildren is called from many places, but it is interrupted from ZKPermissionWatcher#nodeChildrenChanged() by calling childrenChangedFuture.cancel. Here cancel/interrupt indicates that we have more updated data, no need to proceed with the old data. Returning empty is for graceful exit of the processing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615923761
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 34s | master passed |
| +1 :green_heart: | compile | 1m 28s | master passed |
| +1 :green_heart: | shadedjars | 5m 38s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 16s | hbase-zookeeper in master failed. |
| -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 19s | the patch passed |
| +1 :green_heart: | compile | 1m 27s | the patch passed |
| +1 :green_heart: | javac | 1m 27s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 42s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in the patch failed. |
| -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 44s | hbase-zookeeper in the patch passed. |
| -1 :x: | unit | 201m 0s | hbase-server in the patch failed. |
| | | 230m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 975641db6815 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2846ea4700 |
| Default Java | 2020-01-14 |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/testReport/ |
| Max. process+thread count | 3361 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615531766
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 35s | master passed |
| +1 :green_heart: | checkstyle | 1m 23s | master passed |
| +1 :green_heart: | spotbugs | 2m 36s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 17s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 16s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 5m 23s | Patch does not cause any errors with Hadoop 3.1.2. |
| +1 :green_heart: | spotbugs | 2m 46s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 29m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 645fbb60b8e6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e130ffb7a0 |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] arshadmohammad commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410707472
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
##########
@@ -185,25 +185,27 @@ public void run() {
public void nodeChildrenChanged(final String path) {
waitUntilStarted();
if (path.equals(aclZNode)) {
- try {
- final List<ZKUtil.NodeAndData> nodeList =
- ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
- // preempt any existing nodeChildrenChanged event processing
- if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
- boolean cancelled = childrenChangedFuture.cancel(true);
- if (!cancelled) {
- // task may have finished between our check and attempted cancel, this is fine.
- if (! childrenChangedFuture.isDone()) {
- LOG.warn("Could not cancel processing node children changed event, " +
- "please file a JIRA and attach logs if possible.");
- }
+ // preempt any existing nodeChildrenChanged event processing
+ if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
+ boolean cancelled = childrenChangedFuture.cancel(true);
+ if (!cancelled) {
+ // task may have finished between our check and attempted cancel, this is fine.
+ if (!childrenChangedFuture.isDone()) {
+ LOG.warn("Could not cancel processing node children changed event, "
+ + "please file a JIRA and attach logs if possible.");
}
}
- childrenChangedFuture = asyncProcessNodeUpdate(() -> refreshNodes(nodeList));
- } catch (KeeperException ke) {
- LOG.error("Error reading data from zookeeper for path "+path, ke);
- watcher.abort("ZooKeeper error get node children for path "+path, ke);
}
+ childrenChangedFuture = asyncProcessNodeUpdate(() -> {
+ try {
+ final List<ZKUtil.NodeAndData> nodeList =
+ ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
+ refreshNodes(nodeList);
+ } catch (KeeperException ke) {
+ LOG.error("Error reading data from zookeeper for path " + path, ke);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] arshadmohammad commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410716369
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
Review comment:
For branch-1.3.x this is an issue, the reason I have explained in the issue description.
For master and branch-2, this PR change is for improvement as these branches do not use zk based region assignment.
Code change is same, for branch-1.3.x it is for the fix for other branches it is just improvement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615888787
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 36s | master passed |
| +1 :green_heart: | checkstyle | 1m 22s | master passed |
| +1 :green_heart: | spotbugs | 2m 33s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 15s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 5m 25s | Patch does not cause any errors with Hadoop 3.1.2. |
| +1 :green_heart: | spotbugs | 3m 46s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 31m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 3c9157af6265 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2846ea4700 |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410693565
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
##########
@@ -185,25 +185,27 @@ public void run() {
public void nodeChildrenChanged(final String path) {
waitUntilStarted();
if (path.equals(aclZNode)) {
- try {
- final List<ZKUtil.NodeAndData> nodeList =
- ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
- // preempt any existing nodeChildrenChanged event processing
- if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
- boolean cancelled = childrenChangedFuture.cancel(true);
- if (!cancelled) {
- // task may have finished between our check and attempted cancel, this is fine.
- if (! childrenChangedFuture.isDone()) {
- LOG.warn("Could not cancel processing node children changed event, " +
- "please file a JIRA and attach logs if possible.");
- }
+ // preempt any existing nodeChildrenChanged event processing
+ if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
+ boolean cancelled = childrenChangedFuture.cancel(true);
+ if (!cancelled) {
+ // task may have finished between our check and attempted cancel, this is fine.
+ if (!childrenChangedFuture.isDone()) {
+ LOG.warn("Could not cancel processing node children changed event, "
+ + "please file a JIRA and attach logs if possible.");
}
}
- childrenChangedFuture = asyncProcessNodeUpdate(() -> refreshNodes(nodeList));
- } catch (KeeperException ke) {
- LOG.error("Error reading data from zookeeper for path "+path, ke);
- watcher.abort("ZooKeeper error get node children for path "+path, ke);
}
+ childrenChangedFuture = asyncProcessNodeUpdate(() -> {
+ try {
+ final List<ZKUtil.NodeAndData> nodeList =
+ ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
+ refreshNodes(nodeList);
+ } catch (KeeperException ke) {
+ LOG.error("Error reading data from zookeeper for path " + path, ke);
Review comment:
Or better create a string variable and use it for both LOG.error and abort message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615556720
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 52s | master passed |
| +1 :green_heart: | compile | 1m 15s | master passed |
| +1 :green_heart: | shadedjars | 5m 22s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 40s | the patch passed |
| +1 :green_heart: | compile | 1m 13s | the patch passed |
| +1 :green_heart: | javac | 1m 13s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 19s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 40s | hbase-zookeeper in the patch passed. |
| -1 :x: | unit | 201m 30s | hbase-server in the patch failed. |
| | | 228m 26s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 86f838feac41 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e130ffb7a0 |
| Default Java | 1.8.0_232 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/testReport/ |
| Max. process+thread count | 3503 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410710605
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
Review comment:
@arshadmohammad have you seen this issue being the cause for create table being slow or this is an improvement over preempting any existing nodeChildrenChanged event processing (done in ZKPermissionWatcher)?
Basically, when thread is interrupted and we return empty list, what could be the catch? Some incorrect decision could be taken by HMaster?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615908833
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 37s | master passed |
| +1 :green_heart: | compile | 1m 14s | master passed |
| +1 :green_heart: | shadedjars | 5m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | compile | 1m 13s | the patch passed |
| +1 :green_heart: | javac | 1m 13s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 54s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 41s | hbase-zookeeper in the patch passed. |
| -1 :x: | unit | 136m 51s | hbase-server in the patch failed. |
| | | 162m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 542cd2bcd94d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2846ea4700 |
| Default Java | 1.8.0_232 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/testReport/ |
| Max. process+thread count | 5320 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410693444
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
##########
@@ -185,25 +185,27 @@ public void run() {
public void nodeChildrenChanged(final String path) {
waitUntilStarted();
if (path.equals(aclZNode)) {
- try {
- final List<ZKUtil.NodeAndData> nodeList =
- ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
- // preempt any existing nodeChildrenChanged event processing
- if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
- boolean cancelled = childrenChangedFuture.cancel(true);
- if (!cancelled) {
- // task may have finished between our check and attempted cancel, this is fine.
- if (! childrenChangedFuture.isDone()) {
- LOG.warn("Could not cancel processing node children changed event, " +
- "please file a JIRA and attach logs if possible.");
- }
+ // preempt any existing nodeChildrenChanged event processing
+ if (childrenChangedFuture != null && !childrenChangedFuture.isDone()) {
+ boolean cancelled = childrenChangedFuture.cancel(true);
+ if (!cancelled) {
+ // task may have finished between our check and attempted cancel, this is fine.
+ if (!childrenChangedFuture.isDone()) {
+ LOG.warn("Could not cancel processing node children changed event, "
+ + "please file a JIRA and attach logs if possible.");
}
}
- childrenChangedFuture = asyncProcessNodeUpdate(() -> refreshNodes(nodeList));
- } catch (KeeperException ke) {
- LOG.error("Error reading data from zookeeper for path "+path, ke);
- watcher.abort("ZooKeeper error get node children for path "+path, ke);
}
+ childrenChangedFuture = asyncProcessNodeUpdate(() -> {
+ try {
+ final List<ZKUtil.NodeAndData> nodeList =
+ ZKUtil.getChildDataAndWatchForNewChildren(watcher, aclZNode);
+ refreshNodes(nodeList);
+ } catch (KeeperException ke) {
+ LOG.error("Error reading data from zookeeper for path " + path, ke);
Review comment:
Please user `{}` for LOG param `path`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] arshadmohammad commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410707488
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
+ return new ArrayList<>();
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1546: HBASE-24211: Create
table is slow in large cluster when AccessController is enabled.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#issuecomment-615547859
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 3s | master passed |
| +1 :green_heart: | compile | 1m 24s | master passed |
| +1 :green_heart: | shadedjars | 5m 4s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 18s | hbase-zookeeper in master failed. |
| -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 24s | the patch passed |
| +1 :green_heart: | javac | 1m 24s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 16s | hbase-zookeeper in the patch failed. |
| -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 43s | hbase-zookeeper in the patch passed. |
| -1 :x: | unit | 129m 10s | hbase-server in the patch failed. |
| | | 156m 23s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1546 |
| JIRA Issue | HBASE-24211 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3b9b3474f506 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e130ffb7a0 |
| Default Java | 2020-01-14 |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/testReport/ |
| Max. process+thread count | 3887 (vs. ulimit of 12500) |
| modules | C: hbase-zookeeper hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1546/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #1546:
HBASE-24211: Create table is slow in large cluster when AccessController is
enabled.
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1546: HBASE-24211: Create table is slow in large cluster when AccessController is enabled.
URL: https://github.com/apache/hbase/pull/1546#discussion_r410954893
##########
File path: hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
##########
@@ -741,6 +741,10 @@ public static int getNumberOfChildren(ZKWatcher zkw, String znode)
if (nodes != null) {
List<NodeAndData> newNodes = new ArrayList<>();
for (String node : nodes) {
+ if (Thread.interrupted()) {
+ // Partial data should not be processed. Cancel processing by sending empty list.
Review comment:
Thanks @arshadmohammad
It seems `ZKPermissionWatcher#nodeChildrenChanged()` is interrupting `ZKPermissionWatcher#refreshNodes()` and that method is already taking care of `Thread.interrupted()` check. That would not interrupt `ZKUtil#getChildDataAndWatchForNewChildren()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services