You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/07/09 16:30:54 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #3192: HADOOP-17795. Provide fallbacks for callqueue.impl and scheduler.impl
virajjasani opened a new pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on a change in pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
tasanuma commented on a change in pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#discussion_r667842295
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
Review comment:
Is there a problem with simply deleting it?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
Review comment:
Is there a problem with simply deleting the method?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
Review comment:
@jojochuang Thanks for your explanation. I understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#issuecomment-879092072
I have addressed comments @jojochuang @tasanuma. Please take a look when you get some time.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma merged pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
tasanuma merged pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
tasanuma commented on pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#issuecomment-879830257
Merged it. Thanks for your contribution, @virajjasani. Thanks for your review, @jojochuang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#discussion_r667807423
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
static Class<? extends BlockingQueue<Call>> getQueueClass(
String prefix, Configuration conf) {
String name = prefix + "." + CommonConfigurationKeys.IPC_CALLQUEUE_IMPL_KEY;
Class<?> queueClass = conf.getClass(name, LinkedBlockingQueue.class);
return CallQueueManager.convertQueueClass(queueClass, Call.class);
}
+ /**
+ * Return class configured by property 'ipc.<port>.callqueue.impl'. If the
+ * config is not present, default config (without port) is used to derive
+ * class 'ipc.callqueue.impl' and returned if class value is present and
+ * valid. If default config is also not present, default
+ * class {@link LinkedBlockingQueue} is returned.
+ *
+ * @param namespace Namespace "ipc".
+ * @param port Server's listener port.
+ * @param conf Configuration properties.
+ * @return Class returned based on configuration.
+ */
+ static Class<? extends BlockingQueue<Call>> getQueueClass(
+ String namespace, int port, Configuration conf) {
+ String nameWithPort = namespace + "." + port + "."
+ + CommonConfigurationKeys.IPC_CALLQUEUE_IMPL_KEY;
+ String nameWithoutPort = namespace + "."
Review comment:
It would be nice to add this property to our core-default.xml, https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L2507
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -746,6 +773,50 @@ private String getQueueClassPrefix() {
return CallQueueManager.convertSchedulerClass(schedulerClass);
}
+ /**
+ * Return class configured by property 'ipc.<port>.scheduler.impl'. If the
+ * config is not present, and if property 'ipc.<port>.callqueue.impl'
+ * represents FairCallQueue class, return DecayRpcScheduler. If that config
+ * does not have value FairCallQueue, default config (without port) is used
+ * to derive class 'ipc.scheduler.impl'. If default config is also not
+ * present, default class {@link DefaultRpcScheduler} is returned.
+ *
+ * @param namespace Namespace "ipc".
+ * @param port Server's listener port.
+ * @param conf Configuration properties.
+ * @return Class returned based on configuration.
+ */
+ static Class<? extends RpcScheduler> getSchedulerClass(
+ String namespace, int port, Configuration conf) {
+ String schedulerKeyNameWithPort = namespace + "." + port + "."
+ + CommonConfigurationKeys.IPC_SCHEDULER_IMPL_KEY;
+ String schedulerKeyNameWithoutPort = namespace + "."
Review comment:
need this property in core-default.xml
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -746,6 +773,50 @@ private String getQueueClassPrefix() {
return CallQueueManager.convertSchedulerClass(schedulerClass);
}
+ /**
+ * Return class configured by property 'ipc.<port>.scheduler.impl'. If the
Review comment:
property '<namespace>.<port>.scheduler.impl'
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
Review comment:
The Server class is public API. It's used by several other projects, including Tez and Ratis. I haven't checked how it is used but it would be great to avoid breakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3192: HADOOP-17795. Provide fallbacks for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#issuecomment-877751755
@aajisaka @ayushtkn @tasanuma Could you please review? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#issuecomment-878329385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3192: HADOOP-17795. Provide fallbacks for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#issuecomment-877409508
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 4s | | trunk passed |
| +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 10s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 20m 11s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 11s | | the patch passed |
| +1 :green_heart: | compile | 18m 11s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 1s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 176m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3192/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3192 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 4399b5db4901 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6bec36fbb7175519043abbd396e5df185fa9cff9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3192/1/testReport/ |
| Max. process+thread count | 1500 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3192/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3192: HADOOP-17795. Provide fallback properties for callqueue.impl and scheduler.impl
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3192:
URL: https://github.com/apache/hadoop/pull/3192#discussion_r667852647
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -714,13 +714,40 @@ private String getQueueClassPrefix() {
return CommonConfigurationKeys.IPC_NAMESPACE + "." + port;
}
+ @Deprecated
static Class<? extends BlockingQueue<Call>> getQueueClass(
String prefix, Configuration conf) {
String name = prefix + "." + CommonConfigurationKeys.IPC_CALLQUEUE_IMPL_KEY;
Class<?> queueClass = conf.getClass(name, LinkedBlockingQueue.class);
return CallQueueManager.convertQueueClass(queueClass, Call.class);
}
+ /**
+ * Return class configured by property 'ipc.<port>.callqueue.impl'. If the
+ * config is not present, default config (without port) is used to derive
+ * class 'ipc.callqueue.impl' and returned if class value is present and
+ * valid. If default config is also not present, default
+ * class {@link LinkedBlockingQueue} is returned.
+ *
+ * @param namespace Namespace "ipc".
+ * @param port Server's listener port.
+ * @param conf Configuration properties.
+ * @return Class returned based on configuration.
+ */
+ static Class<? extends BlockingQueue<Call>> getQueueClass(
+ String namespace, int port, Configuration conf) {
+ String nameWithPort = namespace + "." + port + "."
+ + CommonConfigurationKeys.IPC_CALLQUEUE_IMPL_KEY;
+ String nameWithoutPort = namespace + "."
Review comment:
Done
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -746,6 +773,50 @@ private String getQueueClassPrefix() {
return CallQueueManager.convertSchedulerClass(schedulerClass);
}
+ /**
+ * Return class configured by property 'ipc.<port>.scheduler.impl'. If the
+ * config is not present, and if property 'ipc.<port>.callqueue.impl'
+ * represents FairCallQueue class, return DecayRpcScheduler. If that config
+ * does not have value FairCallQueue, default config (without port) is used
+ * to derive class 'ipc.scheduler.impl'. If default config is also not
+ * present, default class {@link DefaultRpcScheduler} is returned.
+ *
+ * @param namespace Namespace "ipc".
+ * @param port Server's listener port.
+ * @param conf Configuration properties.
+ * @return Class returned based on configuration.
+ */
+ static Class<? extends RpcScheduler> getSchedulerClass(
+ String namespace, int port, Configuration conf) {
+ String schedulerKeyNameWithPort = namespace + "." + port + "."
+ + CommonConfigurationKeys.IPC_SCHEDULER_IMPL_KEY;
+ String schedulerKeyNameWithoutPort = namespace + "."
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org