You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/24 16:09:21 UTC
[GitHub] [flink] tillrohrmann commented on a change in pull request #16946: [FLINK-23654][runtime] Splitting IO and future threadpools and adding advanced configurations
tillrohrmann commented on a change in pull request #16946:
URL: https://github.com/apache/flink/pull/16946#discussion_r694984379
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/JobMasterServiceLeadershipRunnerFactory.java
##########
@@ -94,7 +94,7 @@ public JobManagerRunner createJobManagerRunner(
final DefaultJobMasterServiceFactory jobMasterServiceFactory =
new DefaultJobMasterServiceFactory(
- jobManagerServices.getScheduledExecutorService(),
+ jobManagerServices.getFutureExecutor(),
Review comment:
Same here. The factory uses this executor to create the `JobMasterService`. This is a blocking operation for which we should use the `ioExecutor`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServices.java
##########
@@ -97,7 +106,8 @@ public void shutdown() throws Exception {
Throwable firstException = null;
try {
- scheduledExecutorService.shutdownNow();
+ futureExecutor.shutdownNow();
+ ioExecutor.shutdownNow();
Review comment:
I would recommend splitting this into two calls that are individually wrapper with `try-catch`. Otherwise we won't stop the `ioExecutor` if `futureExecutor.shutdownNow()` fails for some reason.
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/RestOptions.java
##########
@@ -160,6 +160,20 @@
.withDescription(
"The number of threads for the asynchronous processing of requests.");
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
+ public static final ConfigOption<Integer> JOBMANAGER_FUTURE_THREADS =
+ key("jobmanager.future-executor.numThreads")
+ .defaultValue(0)
+ .withDescription(
+ "The number of jobmanager future-executor threads for shared services.");
+
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
Review comment:
I think we should change this to `EXPERT_CLUSTER` if we put the options in the `ClusterOptions` class.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServicesTest.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.util.TestingFatalErrorHandler;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.util.concurrent.ThreadPoolExecutor;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests auxiliary shared services created by {@link JobManagerSharedServices} and used by the
+ * {@link JobMaster}.
+ */
+public class JobManagerSharedServicesTest extends TestLogger {
+
+ private static final int NUM_FUTURE_THREADS = 128;
+ private static final int NUM_IO_THREADS = 256;
+
+ @Test
+ public void testBasicMapping() throws Exception {
+ Configuration config = new Configuration();
+ config.setInteger(RestOptions.JOBMANAGER_FUTURE_THREADS, NUM_FUTURE_THREADS);
+ config.setInteger(RestOptions.JOBMANAGER_IO_THREADS, NUM_IO_THREADS);
+
+ final JobManagerSharedServices jobManagerSharedServices =
+ JobManagerSharedServices.fromConfiguration(
+ config,
+ new BlobServer(config, new VoidBlobStore()),
+ new TestingFatalErrorHandler());
+ assertEquals(
+ NUM_FUTURE_THREADS,
+ ((ThreadPoolExecutor) jobManagerSharedServices.getFutureExecutor())
+ .getCorePoolSize());
+ assertEquals(
+ NUM_IO_THREADS,
+ ((ThreadPoolExecutor) jobManagerSharedServices.getIoExecutor()).getCorePoolSize());
Review comment:
This test is a bit brittle because it relies on `jobManagerSharedServices.getFutureExecutor()` being a `ThreadPoolExecutor`. If the implementation changes, then this test will break.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java
##########
@@ -343,10 +346,10 @@ private SchedulerNG createScheduler(
slotPoolServiceSchedulerFactory.createScheduler(
log,
jobGraph,
- scheduledExecutorService,
+ ioExecutor,
Review comment:
Let's also update the parameter names in `SlotPoolServiceSchedulerFactory.createScheduler` and its implementations.
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/RestOptions.java
##########
@@ -160,6 +160,20 @@
.withDescription(
"The number of threads for the asynchronous processing of requests.");
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
+ public static final ConfigOption<Integer> JOBMANAGER_FUTURE_THREADS =
+ key("jobmanager.future-executor.numThreads")
+ .defaultValue(0)
+ .withDescription(
+ "The number of jobmanager future-executor threads for shared services.");
+
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
+ public static final ConfigOption<Integer> JOBMANAGER_IO_THREADS =
+ key("jobmanager.io-executor.numThreads")
+ .defaultValue(0)
+ .withDescription(
+ "The number of jobmanager io-executor threads for shared services.");
Review comment:
Maybe we could even reuse the `ClusterOptions.CLUSTER_IO_EXECUTOR_POOL_SIZE` for the `ioExecutor`. Then we could add another `jobmaster.future-pool.size` that we also add to `ClusterOptions`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServices.java
##########
@@ -147,11 +157,23 @@ public static JobManagerSharedServices fromConfiguration(
failOnJvmMetaspaceOomError ? fatalErrorHandler : null,
checkClassLoaderLeak));
+ final int numJobManagerFutureThreads =
+ config.getInteger(RestOptions.JOBMANAGER_FUTURE_THREADS);
Review comment:
If you remove the default value from this option, then you can do something like this:
`config.getInteger(RestOptions.JOBMANAGER_FUTURE_THREADS, Hardware.getNumberCPUCores())`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##########
@@ -519,7 +519,7 @@ JobManagerRunner createJobManagerRunner(JobGraph jobGraph, long initializationTi
return Acknowledge.get();
},
- jobManagerSharedServices.getScheduledExecutorService());
+ jobManagerSharedServices.getFutureExecutor());
Review comment:
I think we should use the IO executor here.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServices.java
##########
@@ -47,7 +48,9 @@
*/
public class JobManagerSharedServices {
- private final ScheduledExecutorService scheduledExecutorService;
+ private final ScheduledExecutorService futureExecutor;
+
+ private final ScheduledExecutorService ioExecutor;
Review comment:
Can the `ioExecutor` be a normal `Executor` or maybe `ExecutorService` if we also need to terminate it in this class?
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/RestOptions.java
##########
@@ -160,6 +160,20 @@
.withDescription(
"The number of threads for the asynchronous processing of requests.");
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
+ public static final ConfigOption<Integer> JOBMANAGER_FUTURE_THREADS =
+ key("jobmanager.future-executor.numThreads")
+ .defaultValue(0)
+ .withDescription(
+ "The number of jobmanager future-executor threads for shared services.");
+
+ @Documentation.Section(Documentation.Sections.EXPERT_REST)
+ public static final ConfigOption<Integer> JOBMANAGER_IO_THREADS =
+ key("jobmanager.io-executor.numThreads")
+ .defaultValue(0)
+ .withDescription(
+ "The number of jobmanager io-executor threads for shared services.");
Review comment:
I think we should put these options into the `JobManagerOptions` class.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobManagerSharedServicesTest.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.util.TestingFatalErrorHandler;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.util.concurrent.ThreadPoolExecutor;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Tests auxiliary shared services created by {@link JobManagerSharedServices} and used by the
+ * {@link JobMaster}.
+ */
+public class JobManagerSharedServicesTest extends TestLogger {
+
+ private static final int NUM_FUTURE_THREADS = 128;
+ private static final int NUM_IO_THREADS = 256;
+
+ @Test
+ public void testBasicMapping() throws Exception {
+ Configuration config = new Configuration();
+ config.setInteger(RestOptions.JOBMANAGER_FUTURE_THREADS, NUM_FUTURE_THREADS);
+ config.setInteger(RestOptions.JOBMANAGER_IO_THREADS, NUM_IO_THREADS);
+
+ final JobManagerSharedServices jobManagerSharedServices =
+ JobManagerSharedServices.fromConfiguration(
+ config,
+ new BlobServer(config, new VoidBlobStore()),
+ new TestingFatalErrorHandler());
+ assertEquals(
+ NUM_FUTURE_THREADS,
+ ((ThreadPoolExecutor) jobManagerSharedServices.getFutureExecutor())
+ .getCorePoolSize());
+ assertEquals(
+ NUM_IO_THREADS,
+ ((ThreadPoolExecutor) jobManagerSharedServices.getIoExecutor()).getCorePoolSize());
Review comment:
One way to test the behaviour could be to submit `Runnables` that count down a `CountDownLatch` and then block on another latch. If you have two latches `a = new CountDownLatch(poolSize)` and `b = new CountDownLatch(poolSize + 1)` and then submit `poolSize + 1` runnables, you should see that `a` fires and `b` not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org