You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "Pochatkin (via GitHub)" <gi...@apache.org> on 2023/11/15 12:15:42 UTC

[PR] IGNITE-20839 Compute jobs queue [ignite-3]

Pochatkin opened a new pull request, #2843:
URL: https://github.com/apache/ignite-3/pull/2843

   https://issues.apache.org/jira/browse/IGNITE-20839
   
   Introduced queue for compute jobs with priority mechanism
   
   Thank you for submitting the pull request.
   
   To streamline the review process of the patch and ensure better code quality
   we ask both an author and a reviewer to verify the following:
   
   ### The Review Checklist
   - [ ] **Formal criteria:** TC status, codestyle, mandatory documentation. Also make sure to complete the following:  
   \- There is a single JIRA ticket related to the pull request.  
   \- The web-link to the pull request is attached to the JIRA ticket.  
   \- The JIRA ticket has the Patch Available state.  
   \- The description of the JIRA ticket explains WHAT was made, WHY and HOW.  
   \- The pull request title is treated as the final commit message. The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
   - [ ] **Design:** new code conforms with the design principles of the components it is added to.
   - [ ] **Patch quality:** patch cannot be split into smaller pieces, its size must be reasonable.
   - [ ] **Code quality:** code is clean and readable, necessary developer documentation is added if needed.
   - [ ] **Tests code quality:** test set covers positive/negative scenarios, happy/edge cases. Tests are effective in terms of execution time and resources.
   
   ### Notes
   - [Apache Ignite Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander merged PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1397073029


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -268,46 +224,25 @@ private Object sendExecuteResponse(@Nullable Object result, @Nullable Throwable
         return null;
     }
 
-    private <R, J extends ComputeJob<R>> Class<J> jobClass(ClassLoader jobClassLoader, String jobClassName) {
-        try {
-            return (Class<J>) Class.forName(jobClassName, true, jobClassLoader);
-        } catch (ClassNotFoundException e) {
-            throw new IgniteInternalException("Cannot load job class by name '" + jobClassName + "'", e);
-        }
-    }
-
-    private CompletableFuture<JobContext> jobClassLoader(List<DeploymentUnit> units) {
-        return jobContextManager.acquireClassLoader(units);
-    }

Review Comment:
   What's the point of inlining this method?



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -35,7 +35,20 @@ public interface ComputeComponent extends IgniteComponent {
      * @param <R>      result type
      * @return future execution result
      */
-    <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args);

Review Comment:
   This javadoc should be copied to the corresponding default method and options parameter description should be added.
   I wonder why checkStyle didn't catch both these issues.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/ComputeExecutorImpl.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.compute.ComputeJob;
+import org.apache.ignite.compute.JobExecutionContext;
+import org.apache.ignite.internal.compute.ExecutionOptions;
+import org.apache.ignite.internal.compute.JobExecutionContextImpl;
+import org.apache.ignite.internal.compute.configuration.ComputeConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+
+/**
+ * Base implementation of {@link ComputeExecutor}.
+ */
+public class ComputeExecutorImpl implements ComputeExecutor {
+    private static final IgniteLogger LOG = Loggers.forClass(ComputeExecutorImpl.class);
+
+    private final Ignite ignite;
+
+    private final ComputeConfiguration configuration;
+
+    private PriorityQueueExecutor executorService;
+
+    public ComputeExecutorImpl(Ignite ignite, ComputeConfiguration configuration) {
+        this.ignite = ignite;
+        this.configuration = configuration;
+    }
+
+    @Override
+    public <R> CompletableFuture<R> executeJob(ExecutionOptions options, ComputeJob<R> job, Object[] args) {
+        JobExecutionContext context = new JobExecutionContextImpl(ignite);
+
+        return executorService.submit(() -> job.execute(context, args), options.priority());
+    }
+
+    @Override
+    public void start() {
+        executorService = new PriorityQueueExecutor(
+                configuration,
+                new NamedThreadFactory(NamedThreadFactory.threadPrefix(ignite.name(), "compute"), LOG)
+        );
+    }
+
+    @Override
+    public void stop() {
+        executorService.shutdown();
+    }
+
+    long stopTimeoutMillis() {
+        return configuration.threadPoolStopTimeoutMillis().value();
+    }

Review Comment:
   This method is not used here, it should be moved to the `PriorityQueueExecutor` and used there.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -75,111 +58,118 @@ public class ComputeComponentImpl implements ComputeComponent {
 
     private final InFlightFutures inFlightFutures = new InFlightFutures();
 
-    private final Ignite ignite;
-
     private final MessagingService messagingService;
 
-    private final ComputeConfiguration configuration;
-
     private final JobContextManager jobContextManager;
 
-    private ExecutorService jobExecutorService;
+    private final ComputeExecutor executor;
 
     /**
      * Creates a new instance.
      */
     public ComputeComponentImpl(
-            Ignite ignite,
             MessagingService messagingService,
-            ComputeConfiguration configuration,
-            JobContextManager jobContextManager) {
-        this.ignite = ignite;
+            JobContextManager jobContextManager,
+            ComputeExecutor executor
+    ) {
         this.messagingService = messagingService;
-        this.configuration = configuration;
         this.jobContextManager = jobContextManager;
+        this.executor = executor;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void start() {
+        executor.start();
+
+        messagingService.addMessageHandler(ComputeMessageTypes.class, (message, senderConsistentId, correlationId) -> {
+            assert correlationId != null;
+
+            if (message instanceof ExecuteRequest) {
+                processExecuteRequest((ExecuteRequest) message, senderConsistentId, correlationId);
+            }
+        });
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void stop() throws Exception {
+        if (!stopGuard.compareAndSet(false, true)) {
+            return;
+        }
+
+        busyLock.block();
+
+        executor.stop();
+
+        inFlightFutures.cancelInFlightFutures();
     }
 
     /** {@inheritDoc} */
     @Override
-    public <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args) {
+    public <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
         if (!busyLock.enterBusy()) {
             return failedFuture(new NodeStoppingException());
         }
 
         try {
-            return mapClassLoaderExceptions(jobClassLoader(units), jobClassName)
-                    .thenCompose(context -> doExecuteLocally(this.<R, ComputeJob<R>>jobClass(context.classLoader(), jobClassName), args)
-                            .whenComplete((r, e) -> context.close())
+            CompletableFuture<JobContext> jobContextCompletableFuture = mapClassLoaderExceptions(
+                    jobContextManager.acquireClassLoader(units), jobClassName);
+            return jobContextCompletableFuture
+                    .thenCompose(context ->
+                            doExecuteLocally(options, ComputeUtils.<R>instantiateJob(context.classLoader(), jobClassName), args)
+                                    .whenComplete((r, e) -> context.close())
                     );
         } finally {
             busyLock.leaveBusy();
         }
     }
 
-    private <R> CompletableFuture<R> doExecuteLocally(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        assert jobExecutorService != null : "Not started yet!";
-
-        CompletableFuture<R> future = startLocalExecution(jobClass, args);
+    private <R> CompletableFuture<R> doExecuteLocally(ExecutionOptions options, ComputeJob<R> jobInstance, Object[] args) {
+        CompletableFuture<R> future = executor.executeJob(options, jobInstance, args);
         inFlightFutures.registerFuture(future);
 
         return future;
     }
 
-    private <R> CompletableFuture<R> startLocalExecution(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        try {
-            return CompletableFuture.supplyAsync(() -> executeJob(jobClass, args), jobExecutorService);
-        } catch (RejectedExecutionException e) {
-            return failedFuture(e);
-        }
-    }
-
-    private <R> R executeJob(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        ComputeJob<R> job = instantiateJob(jobClass);
-        JobExecutionContext context = new JobExecutionContextImpl(ignite);
-        // TODO: IGNITE-16746 - translate NodeStoppingException to a public exception

Review Comment:
   What about translating this exception? I can't find this translation in the new code.



##########
modules/compute/src/test/java/org/apache/ignite/internal/compute/queue/PriorityQueueExecutorTest.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.compute.configuration.ComputeConfiguration;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.testframework.BaseIgniteAbstractTest;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Test suite for {@link PriorityQueueExecutor}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class PriorityQueueExecutorTest extends BaseIgniteAbstractTest {
+    private static final IgniteLogger LOG = Loggers.forClass(PriorityQueueExecutorTest.class);
+
+    @InjectConfiguration
+    private ComputeConfiguration configuration;
+
+    private PriorityQueueExecutor priorityQueueExecutor;
+
+
+    private void initExecutor(int threads) {
+        if (priorityQueueExecutor != null) {
+            priorityQueueExecutor.shutdown();
+        }
+
+        assertThat(configuration.change(computeChange -> computeChange.changeThreadPoolSize(threads)), willCompleteSuccessfully());
+
+        priorityQueueExecutor = new PriorityQueueExecutor(
+                configuration,
+                new NamedThreadFactory(NamedThreadFactory.threadPrefix("testNode", "compute"), LOG)
+        );
+    }
+
+    @Test
+    public void testQueueIsWorking() {
+        initExecutor(1);
+
+        CountDownLatch latch1 = new CountDownLatch(1);
+        CountDownLatch latch2 = new CountDownLatch(1);
+
+        CompletableFuture<Integer> task1 = priorityQueueExecutor.submit(() -> {
+            latch1.await();
+            return 0;
+        });
+
+        CompletableFuture<Integer> task2 = priorityQueueExecutor.submit(() -> {
+            latch2.await();
+            return 1;
+        });
+
+        assertThat(task1.isDone(), is(false));
+        assertThat(task2.isDone(), is(false));
+
+        latch2.countDown();
+
+        assertThat(task1, willTimeoutIn(100, TimeUnit.MILLISECONDS));
+        assertThat(task2, willTimeoutIn(100, TimeUnit.MILLISECONDS));
+
+        latch1.countDown();
+
+        assertThat(task1, willCompleteSuccessfully());
+        assertThat(task2, willCompleteSuccessfully());
+    }
+
+    @Test
+    public void testSubmitWithPriority() {
+        initExecutor(1);
+
+        CountDownLatch latch1 = new CountDownLatch(1);
+        CountDownLatch latch2 = new CountDownLatch(1);
+        CountDownLatch latch3 = new CountDownLatch(1);
+
+        CompletableFuture<Integer> task1 = priorityQueueExecutor.submit(() -> {
+            latch1.await();
+            return 0;
+        });
+
+        CompletableFuture<Integer> task2 = priorityQueueExecutor.submit(() -> {
+            latch2.await();
+            return 1;
+        }, 1);
+
+        CompletableFuture<Integer> task3 = priorityQueueExecutor.submit(() -> {
+            latch3.await();
+            return 1;
+        }, 2);
+
+
+        assertThat(task1.isDone(), is(false));
+        assertThat(task2.isDone(), is(false));
+        assertThat(task3.isDone(), is(false));
+
+        latch1.countDown();
+
+        assertThat(task1, willCompleteSuccessfully());
+        assertThat(task2.isDone(), is(false));
+        assertThat(task3.isDone(), is(false));
+
+        //Current executing task is 3 because of priority.
+        latch2.countDown();
+        assertThat(task2, willTimeoutIn(100, TimeUnit.MILLISECONDS));
+        assertThat(task3, willTimeoutIn(100, TimeUnit.MILLISECONDS));
+
+        latch3.countDown();
+        assertThat(task2, willCompleteSuccessfully());
+        assertThat(task3, willCompleteSuccessfully());
+    }
+
+    @Test
+    public void testSameOrder() {
+        initExecutor(1);
+
+        CountDownLatch latch1 = new CountDownLatch(1);
+        CountDownLatch latch2 = new CountDownLatch(1);
+        CountDownLatch latch3 = new CountDownLatch(1);
+
+        CompletableFuture<Integer> task1 = priorityQueueExecutor.submit(() -> {
+            latch1.await();
+            return 0;
+        }, 1);
+
+        CompletableFuture<Integer> task2 = priorityQueueExecutor.submit(() -> {
+            latch2.await();
+            return 1;
+        }, 1);
+
+        CompletableFuture<Integer> task3 = priorityQueueExecutor.submit(() -> {
+            latch3.await();
+            return 1;
+        }, 1);
+
+        assertThat(task1.isDone(), is(false));
+        assertThat(task2.isDone(), is(false));
+        assertThat(task3.isDone(), is(false));
+
+        latch1.countDown();
+        assertThat(task1, willCompleteSuccessfully());
+        assertThat(task2.isDone(), is(false));
+        assertThat(task3.isDone(), is(false));
+
+        //Current executing task is 3 because it was first.

Review Comment:
   ```suggestion
           //Current executing task is 2 because it was added first.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/QueueEntry.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Class for queue's entries.
+ * This class implement comparable mechanism for {@link it.unimi.dsi.fastutil.PriorityQueue}
+ * which used as queue in {@link java.util.concurrent.ThreadPoolExecutor}.
+ * Each entry has unique seqNum which used for comparable entries with identity priority.

Review Comment:
   ```suggestion
    * Each entry has unique seqNum which is used for comparing entries with identical priority.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -75,111 +58,118 @@ public class ComputeComponentImpl implements ComputeComponent {
 
     private final InFlightFutures inFlightFutures = new InFlightFutures();
 
-    private final Ignite ignite;
-
     private final MessagingService messagingService;
 
-    private final ComputeConfiguration configuration;
-
     private final JobContextManager jobContextManager;
 
-    private ExecutorService jobExecutorService;
+    private final ComputeExecutor executor;
 
     /**
      * Creates a new instance.
      */
     public ComputeComponentImpl(
-            Ignite ignite,
             MessagingService messagingService,
-            ComputeConfiguration configuration,
-            JobContextManager jobContextManager) {
-        this.ignite = ignite;
+            JobContextManager jobContextManager,
+            ComputeExecutor executor
+    ) {
         this.messagingService = messagingService;
-        this.configuration = configuration;
         this.jobContextManager = jobContextManager;
+        this.executor = executor;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void start() {
+        executor.start();
+
+        messagingService.addMessageHandler(ComputeMessageTypes.class, (message, senderConsistentId, correlationId) -> {
+            assert correlationId != null;
+
+            if (message instanceof ExecuteRequest) {
+                processExecuteRequest((ExecuteRequest) message, senderConsistentId, correlationId);
+            }
+        });
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void stop() throws Exception {
+        if (!stopGuard.compareAndSet(false, true)) {
+            return;
+        }
+
+        busyLock.block();
+
+        executor.stop();
+
+        inFlightFutures.cancelInFlightFutures();
     }
 
     /** {@inheritDoc} */
     @Override
-    public <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args) {
+    public <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
         if (!busyLock.enterBusy()) {
             return failedFuture(new NodeStoppingException());
         }
 
         try {
-            return mapClassLoaderExceptions(jobClassLoader(units), jobClassName)
-                    .thenCompose(context -> doExecuteLocally(this.<R, ComputeJob<R>>jobClass(context.classLoader(), jobClassName), args)
-                            .whenComplete((r, e) -> context.close())
+            CompletableFuture<JobContext> jobContextCompletableFuture = mapClassLoaderExceptions(
+                    jobContextManager.acquireClassLoader(units), jobClassName);
+            return jobContextCompletableFuture
+                    .thenCompose(context ->
+                            doExecuteLocally(options, ComputeUtils.<R>instantiateJob(context.classLoader(), jobClassName), args)

Review Comment:
   There's a subtle difference - previously job was instantiated on the executor thread, I wonder whether it could have any implications or not.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -75,111 +58,118 @@ public class ComputeComponentImpl implements ComputeComponent {
 
     private final InFlightFutures inFlightFutures = new InFlightFutures();
 
-    private final Ignite ignite;
-
     private final MessagingService messagingService;
 
-    private final ComputeConfiguration configuration;
-
     private final JobContextManager jobContextManager;
 
-    private ExecutorService jobExecutorService;
+    private final ComputeExecutor executor;
 
     /**
      * Creates a new instance.
      */
     public ComputeComponentImpl(
-            Ignite ignite,
             MessagingService messagingService,
-            ComputeConfiguration configuration,
-            JobContextManager jobContextManager) {
-        this.ignite = ignite;
+            JobContextManager jobContextManager,
+            ComputeExecutor executor
+    ) {
         this.messagingService = messagingService;
-        this.configuration = configuration;
         this.jobContextManager = jobContextManager;
+        this.executor = executor;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void start() {
+        executor.start();
+
+        messagingService.addMessageHandler(ComputeMessageTypes.class, (message, senderConsistentId, correlationId) -> {
+            assert correlationId != null;
+
+            if (message instanceof ExecuteRequest) {
+                processExecuteRequest((ExecuteRequest) message, senderConsistentId, correlationId);
+            }
+        });
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void stop() throws Exception {
+        if (!stopGuard.compareAndSet(false, true)) {
+            return;
+        }
+
+        busyLock.block();
+
+        executor.stop();
+
+        inFlightFutures.cancelInFlightFutures();
     }
 
     /** {@inheritDoc} */
     @Override
-    public <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args) {
+    public <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
         if (!busyLock.enterBusy()) {
             return failedFuture(new NodeStoppingException());
         }
 
         try {
-            return mapClassLoaderExceptions(jobClassLoader(units), jobClassName)
-                    .thenCompose(context -> doExecuteLocally(this.<R, ComputeJob<R>>jobClass(context.classLoader(), jobClassName), args)
-                            .whenComplete((r, e) -> context.close())
+            CompletableFuture<JobContext> jobContextCompletableFuture = mapClassLoaderExceptions(
+                    jobContextManager.acquireClassLoader(units), jobClassName);
+            return jobContextCompletableFuture

Review Comment:
   ```suggestion
               return mapClassLoaderExceptions(jobContextManager.acquireClassLoader(units), jobClassName)
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/ComputeExecutorImpl.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.compute.ComputeJob;
+import org.apache.ignite.compute.JobExecutionContext;
+import org.apache.ignite.internal.compute.ExecutionOptions;
+import org.apache.ignite.internal.compute.JobExecutionContextImpl;
+import org.apache.ignite.internal.compute.configuration.ComputeConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+
+/**
+ * Base implementation of {@link ComputeExecutor}.
+ */
+public class ComputeExecutorImpl implements ComputeExecutor {
+    private static final IgniteLogger LOG = Loggers.forClass(ComputeExecutorImpl.class);
+
+    private final Ignite ignite;
+
+    private final ComputeConfiguration configuration;
+
+    private PriorityQueueExecutor executorService;
+
+    public ComputeExecutorImpl(Ignite ignite, ComputeConfiguration configuration) {
+        this.ignite = ignite;
+        this.configuration = configuration;
+    }
+
+    @Override
+    public <R> CompletableFuture<R> executeJob(ExecutionOptions options, ComputeJob<R> job, Object[] args) {
+        JobExecutionContext context = new JobExecutionContextImpl(ignite);

Review Comment:
   Should we assert that the `executorService` is not null here as in previous code?



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/configuration/ComputeConfigurationSchema.java:
##########
@@ -39,4 +39,8 @@ public class ComputeConfigurationSchema {
     @Range(min = 1)
     @Value(hasDefault = true)
     public final long threadPoolStopTimeoutMillis = 10_000;
+
+    @Range(min = 1)
+    @Value(hasDefault = true)
+    public final int queueMaxSize = Integer.MAX_VALUE;

Review Comment:
   Please add javadoc.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/message/ExecuteRequest.java:
##########
@@ -20,16 +20,20 @@
 import java.util.List;
 import java.util.Set;
 import org.apache.ignite.internal.compute.ComputeMessageTypes;
+import org.apache.ignite.internal.compute.ExecutionOptions;
 import org.apache.ignite.network.NetworkMessage;
 import org.apache.ignite.network.annotations.Marshallable;
 import org.apache.ignite.network.annotations.Transferable;
 import org.jetbrains.annotations.Nullable;
 
 /**
- * Used to implement remote job execution in {@link org.apache.ignite.compute.IgniteCompute#execute(Set, Class, Object...)}.
+ * Used to implement remote job execution in {@link org.apache.ignite.compute.IgniteCompute#execute(Set, List, String, Object...)}.
  */
 @Transferable(value = ComputeMessageTypes.EXECUTE_REQUEST)
 public interface ExecuteRequest extends NetworkMessage {
+    @Marshallable

Review Comment:
   Please add javadoc.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/QueueEntry.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Class for queue's entries.
+ * This class implement comparable mechanism for {@link it.unimi.dsi.fastutil.PriorityQueue}

Review Comment:
   ```suggestion
    * This class implement comparable mechanism for {@link java.util.concurrent.PriorityBlockingQueue}
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/QueueEntry.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Class for queue's entries.
+ * This class implement comparable mechanism for {@link it.unimi.dsi.fastutil.PriorityQueue}
+ * which used as queue in {@link java.util.concurrent.ThreadPoolExecutor}.
+ * Each entry has unique seqNum which used for comparable entries with identity priority.
+ * It means that entries with same priority has FIFO resolving strategy in queue.
+ *
+ * @param <R> Compute job return type.
+ */
+public class QueueEntry<R> implements Runnable, Comparable<QueueEntry<R>> {
+    private static final AtomicLong seq = new AtomicLong(Long.MIN_VALUE);
+
+    private final CompletableFuture<R> future = new CompletableFuture<>();
+
+    private final Callable<R> job;
+
+    private final int priority;
+
+    private final long seqNum;
+
+    /**
+     * Constructor.
+     *
+     * @param job Compute job callable.
+     * @param priority Job priority.
+     */
+    public QueueEntry(Callable<R> job, int priority) {
+        this.job = job;
+        this.priority = priority;
+        seqNum = seq.getAndIncrement();
+    }
+
+    @Override
+    public void run() {
+        try {
+            future.complete(job.call());
+        } catch (Exception e) {
+            future.completeExceptionally(e);
+        }
+    }
+
+    /**
+     * To {@link CompletableFuture} transformer.
+     *
+     * @return Completable future that will be finished when Compute job finished.

Review Comment:
   ```suggestion
        * @return Completable future that will be finished when Compute job is finished.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1398886117


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -30,21 +30,72 @@ public interface ComputeComponent extends IgniteComponent {
     /**
      * Executes a job of the given class on the current node.
      *
-     * @param jobClassName name of the job class
-     * @param args     job args
-     * @param <R>      result type
-     * @return future execution result
+     * @param options Job execution options.
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.

Review Comment:
   ```suggestion
        * @param units Deployment units which will be loaded for execution.
        * @param jobClassName Name of the job class.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -30,21 +30,72 @@ public interface ComputeComponent extends IgniteComponent {
     /**
      * Executes a job of the given class on the current node.
      *
-     * @param jobClassName name of the job class
-     * @param args     job args
-     * @param <R>      result type
-     * @return future execution result
+     * @param options Job execution options.
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
+     */
+    <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    );
+
+    /**
+     * Executes a job of the given class on the current node with default execution options {@link ExecutionOptions.DEFAULT}.
+     *
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
      */
-    <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args);
+    default <R> CompletableFuture<R> executeLocally(
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
+        return executeLocally(ExecutionOptions.DEFAULT, units, jobClassName, args);
+    }
 
     /**
      * Executes a job of the given class on a remote node.
      *
+     * @param options Job execution options.
+     * @param remoteNode name of the job class

Review Comment:
   ```suggestion
        * @param remoteNode name of the job class.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -30,21 +30,72 @@ public interface ComputeComponent extends IgniteComponent {
     /**
      * Executes a job of the given class on the current node.
      *
-     * @param jobClassName name of the job class
-     * @param args     job args
-     * @param <R>      result type
-     * @return future execution result
+     * @param options Job execution options.
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
+     */
+    <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    );
+
+    /**
+     * Executes a job of the given class on the current node with default execution options {@link ExecutionOptions.DEFAULT}.
+     *
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
      */
-    <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args);
+    default <R> CompletableFuture<R> executeLocally(
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
+        return executeLocally(ExecutionOptions.DEFAULT, units, jobClassName, args);
+    }
 
     /**
      * Executes a job of the given class on a remote node.
      *
+     * @param options Job execution options.
+     * @param remoteNode name of the job class
+     * @param units Deployment units which will be loaded for execution.
+     * @param jobClassName name of the job class
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
+     */
+    <R> CompletableFuture<R> executeRemotely(
+            ExecutionOptions options,
+            ClusterNode remoteNode,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    );
+
+    /**
+     * Executes a job of the given class on a remote node with default execution options {@link ExecutionOptions.DEFAULT}.
+     *
      * @param remoteNode name of the job class
+     * @param units Deployment units which will be loaded for execution.
      * @param jobClassName name of the job class

Review Comment:
   ```suggestion
        * @param jobClassName name of the job class.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/LimitedPriorityBlockingQueue.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of {@link PriorityBlockingQueue} with max size limitation.
+ *
+ * @param <E> The type of elements held in this queue.
+ */
+public class LimitedPriorityBlockingQueue<E> extends PriorityBlockingQueue<E> {
+    private final Supplier<Integer> maxSize;
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize) {
+        this.maxSize = maxSize;
+    }
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     * @param initialCapacity Initial queue capacity.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize, int initialCapacity) {
+        super(initialCapacity);
+        this.maxSize = maxSize;
+        checkInsert(initialCapacity);
+    }
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     * @param initialCapacity Initial queue capacity.
+     * @param comparator the comparator that will be used to order this priority queue.
+     *     If {@code null}, the {@linkplain Comparable natural ordering} of the elements will be used.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize, int initialCapacity, Comparator<? super E> comparator) {
+        super(initialCapacity, comparator);
+        this.maxSize = maxSize;
+        checkInsert(initialCapacity);
+    }
+
+    @Override
+    public boolean add(E o) {
+        checkInsert(1);
+        return super.add(o);
+    }
+
+    @Override
+    public boolean offer(E o) {
+        checkInsert(1);
+        return super.offer(o);
+    }
+
+    @Override
+    public boolean offer(E e, long timeout, TimeUnit unit) {
+        checkInsert(1);
+        return super.offer(e, timeout, unit);
+    }
+
+    @Override
+    public void put(E o) {
+        checkInsert(1);
+        super.put(o);
+    }
+
+    @Override
+    public int remainingCapacity() {
+        return maxSize.get() - size();
+    }
+
+    @Override
+    public boolean addAll(Collection<? extends E> c) {
+        checkInsert(c.size());
+        return super.addAll(c);
+    }
+
+    private void checkInsert(int size) {
+        Integer maxSize = this.maxSize.get();
+        int currentSize = size();
+        if (currentSize + size > maxSize) {

Review Comment:
   Better safe, than sorry
   ```suggestion
           if (currentSize > maxSize - size ) {
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -30,21 +30,72 @@ public interface ComputeComponent extends IgniteComponent {
     /**
      * Executes a job of the given class on the current node.
      *
-     * @param jobClassName name of the job class
-     * @param args     job args
-     * @param <R>      result type
-     * @return future execution result
+     * @param options Job execution options.
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
+     */
+    <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    );
+
+    /**
+     * Executes a job of the given class on the current node with default execution options {@link ExecutionOptions.DEFAULT}.
+     *
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.

Review Comment:
   ```suggestion
        * @param units Deployment units which will be loaded for execution.
        * @param jobClassName Name of the job class.
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponent.java:
##########
@@ -30,21 +30,72 @@ public interface ComputeComponent extends IgniteComponent {
     /**
      * Executes a job of the given class on the current node.
      *
-     * @param jobClassName name of the job class
-     * @param args     job args
-     * @param <R>      result type
-     * @return future execution result
+     * @param options Job execution options.
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
+     */
+    <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    );
+
+    /**
+     * Executes a job of the given class on the current node with default execution options {@link ExecutionOptions.DEFAULT}.
+     *
+     * @param jobClassName Name of the job class.
+     * @param units Deployment units which will be loaded for execution.
+     * @param args Job args.
+     * @param <R> Job result type.
+     * @return Future execution result.
      */
-    <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args);
+    default <R> CompletableFuture<R> executeLocally(
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
+        return executeLocally(ExecutionOptions.DEFAULT, units, jobClassName, args);
+    }
 
     /**
      * Executes a job of the given class on a remote node.
      *
+     * @param options Job execution options.
+     * @param remoteNode name of the job class
+     * @param units Deployment units which will be loaded for execution.
+     * @param jobClassName name of the job class

Review Comment:
   ```suggestion
        * @param jobClassName name of the job class.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "nevdmitry (via GitHub)" <gi...@apache.org>.
nevdmitry commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1403288480


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/BoundedPriorityBlockingQueue.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.AbstractQueue;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of {@link PriorityBlockingQueue} with max size limitation.
+ *
+ * @param <E> The type of elements held in this queue.
+ */
+public class BoundedPriorityBlockingQueue<E> extends AbstractQueue<E> implements BlockingQueue<E> {
+    private final PriorityQueue<E> queue;
+
+    private final Supplier<Integer> maxCapacity;
+
+    private final ReentrantLock lock = new ReentrantLock();
+
+    private final Condition notEmpty = lock.newCondition();
+
+    /**
+     * Constructor.
+     *
+     * @param maxCapacity Max queue size supplier.
+     */
+    public BoundedPriorityBlockingQueue(Supplier<Integer> maxCapacity) {
+        queue = new PriorityQueue<>();
+        this.maxCapacity = maxCapacity;
+    }
+
+    @Override
+    public Iterator<E> iterator() {
+        lock.lock();
+        try {
+            return new PriorityQueue<>(queue).iterator();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public int size() {
+        lock.lock();
+        try {
+            return queue.size();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public void put(E e) throws InterruptedException {
+        offer(e);
+    }
+
+    @Override
+    public boolean offer(E e) {
+        lock.lock();
+        try {
+            checkInsert(1);
+            boolean result = queue.offer(e);
+            if (result) {
+                notEmpty.signalAll();
+            }
+            return result;
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public boolean offer(E e, long l, TimeUnit timeUnit) {
+        return offer(e);
+    }
+
+    @Override
+    public E take() throws InterruptedException {
+        lock.lockInterruptibly();
+        try {
+            while (queue.isEmpty()) {
+                notEmpty.await();
+            }
+            E x = poll();
+            assert x != null;
+            return x;
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public E poll() {
+        lock.lock();
+        try {
+            return queue.poll();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public E poll(long l, TimeUnit timeUnit) throws InterruptedException {
+        long nanos = timeUnit.toNanos(l);
+        lock.lockInterruptibly();
+        try {
+            E result = poll();
+            while (result == null && nanos > 0) {
+                nanos = notEmpty.awaitNanos(nanos);
+                result = poll();
+            }
+            return result;
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public int remainingCapacity() {
+        lock.lock();
+        try {
+            return maxCapacity.get() - queue.size();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public int drainTo(Collection<? super E> objects) {
+        Objects.requireNonNull(objects);
+
+        if (objects == this) {
+            throw new IllegalArgumentException();
+        }
+
+        lock.lock();
+        try {
+            int j = 0; // num object added
+            while (size() > 0) {
+                objects.add(poll());
+                j++;
+            }
+            return j;
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public int drainTo(Collection<? super E> objects, int i) {
+        Objects.requireNonNull(objects);
+
+        if (objects == this) {
+            throw new IllegalArgumentException();
+        }
+        lock.lock();
+        try {
+            int j = 0; // num object added
+            while (size() > 0 && j < i) {
+                objects.add(poll());
+                j++;
+            }
+            return j;
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public E peek() {
+        lock.lock();
+        try {
+            return queue.peek();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public boolean contains(Object o) {
+        lock.lock();
+        try {
+            return queue.contains(o);
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public <T> T[] toArray(T[] ts) {
+        lock.lock();
+        try {
+            return queue.toArray(ts);
+        } finally {
+            lock.unlock();
+        }
+    }
+
+
+    @Override
+    public String toString() {
+        lock.lock();
+        try {
+            return queue.toString();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public void clear() {
+        lock.lock();
+        try {
+            queue.clear();
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    public boolean remove(Object o) {
+        lock.lock();
+        try {
+            return queue.remove(o);
+        } finally {
+            lock.unlock();
+        }
+    }
+
+    private void checkInsert(int size) {
+        Integer maxSize = this.maxCapacity.get();
+        int currentSize = size();
+        if (currentSize > maxSize - size) {
+            throw new QueueOverflowException("Compute queue overflow when tried to insert " + size + " element(s) to queue. "
+                    + "Current queue size " + currentSize + ". "

Review Comment:
   Current queue size is



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1399197562


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/LimitedPriorityBlockingQueue.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of {@link PriorityBlockingQueue} with max size limitation.
+ *
+ * @param <E> The type of elements held in this queue.
+ */
+public class LimitedPriorityBlockingQueue<E> extends PriorityBlockingQueue<E> {
+    private final Supplier<Integer> maxSize;
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize) {
+        this.maxSize = maxSize;
+    }
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     * @param initialCapacity Initial queue capacity.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize, int initialCapacity) {
+        super(initialCapacity);
+        this.maxSize = maxSize;
+        checkInsert(initialCapacity);
+    }
+
+    /**
+     * Constructor.
+     *
+     * @param maxSize Max queue size supplier.
+     * @param initialCapacity Initial queue capacity.
+     * @param comparator the comparator that will be used to order this priority queue.
+     *     If {@code null}, the {@linkplain Comparable natural ordering} of the elements will be used.
+     */
+    public LimitedPriorityBlockingQueue(Supplier<Integer> maxSize, int initialCapacity, Comparator<? super E> comparator) {
+        super(initialCapacity, comparator);
+        this.maxSize = maxSize;
+        checkInsert(initialCapacity);
+    }
+
+    @Override
+    public boolean add(E o) {
+        checkInsert(1);
+        return super.add(o);
+    }
+
+    @Override
+    public boolean offer(E o) {
+        checkInsert(1);
+        return super.offer(o);
+    }
+
+    @Override
+    public boolean offer(E e, long timeout, TimeUnit unit) {
+        checkInsert(1);
+        return super.offer(e, timeout, unit);
+    }
+
+    @Override
+    public void put(E o) {
+        checkInsert(1);

Review Comment:
   `checkInsert` and any actions that are taken after that are not synchronized. Shall we check and do an action in a thread-safe manner?



##########
modules/compute/src/test/java/org/apache/ignite/internal/compute/queue/PriorityQueueExecutorTest.java:
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willThrow;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureExceptionMatcher.willTimeoutIn;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.compute.configuration.ComputeConfiguration;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.testframework.BaseIgniteAbstractTest;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.lang.IgniteException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Test suite for {@link PriorityQueueExecutor}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class PriorityQueueExecutorTest extends BaseIgniteAbstractTest {
+    private static final IgniteLogger LOG = Loggers.forClass(PriorityQueueExecutorTest.class);
+
+    @InjectConfiguration
+    private ComputeConfiguration configuration;
+
+    private PriorityQueueExecutor priorityQueueExecutor;
+
+    private void initExecutor(int threads) {
+        initExecutor(threads, Integer.MAX_VALUE);
+    }
+
+    private void initExecutor(int threads, int maxQueueSize) {
+        if (priorityQueueExecutor != null) {
+            priorityQueueExecutor.shutdown();
+        }
+
+        assertThat(
+                configuration.change(computeChange -> computeChange.changeThreadPoolSize(threads).changeQueueMaxSize(maxQueueSize)),
+                willCompleteSuccessfully()
+        );
+
+        priorityQueueExecutor = new PriorityQueueExecutor(
+                configuration,
+                new NamedThreadFactory(NamedThreadFactory.threadPrefix("testNode", "compute"), LOG)
+        );
+    }
+
+    @Test
+    public void testQueueIsWorking() {
+        initExecutor(1);
+
+        CountDownLatch latch1 = new CountDownLatch(1);
+        CountDownLatch latch2 = new CountDownLatch(1);
+
+        CompletableFuture<Integer> task1 = priorityQueueExecutor.submit(() -> {
+            latch1.await();
+            return 0;
+        });
+
+        CompletableFuture<Integer> task2 = priorityQueueExecutor.submit(() -> {
+            latch2.await();
+            return 1;
+        });
+
+        assertThat(task1.isDone(), is(false));
+        assertThat(task2.isDone(), is(false));
+
+        latch2.countDown();
+
+        assertThat(task1, willTimeoutIn(100, TimeUnit.MILLISECONDS));
+        assertThat(task2, willTimeoutIn(100, TimeUnit.MILLISECONDS));

Review Comment:
   Could you add a couple of words about priority and why `task2` is not executed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1403365226


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/queue/BoundedPriorityBlockingQueue.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.compute.queue;
+
+import java.util.AbstractQueue;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of {@link PriorityBlockingQueue} with max size limitation.
+ *
+ * @param <E> The type of elements held in this queue.
+ */
+public class BoundedPriorityBlockingQueue<E> extends AbstractQueue<E> implements BlockingQueue<E> {

Review Comment:
   Could you please write some tests for this queue? I'm interested in overflow cases, empty queue polls with a timeout, etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "Pochatkin (via GitHub)" <gi...@apache.org>.
Pochatkin commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1398788246


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -268,46 +224,25 @@ private Object sendExecuteResponse(@Nullable Object result, @Nullable Throwable
         return null;
     }
 
-    private <R, J extends ComputeJob<R>> Class<J> jobClass(ClassLoader jobClassLoader, String jobClassName) {
-        try {
-            return (Class<J>) Class.forName(jobClassName, true, jobClassLoader);
-        } catch (ClassNotFoundException e) {
-            throw new IgniteInternalException("Cannot load job class by name '" + jobClassName + "'", e);
-        }
-    }
-
-    private CompletableFuture<JobContext> jobClassLoader(List<DeploymentUnit> units) {
-        return jobContextManager.acquireClassLoader(units);
-    }

Review Comment:
   What's the point to have non-static one line method with direct method call?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1398814379


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -268,46 +224,25 @@ private Object sendExecuteResponse(@Nullable Object result, @Nullable Throwable
         return null;
     }
 
-    private <R, J extends ComputeJob<R>> Class<J> jobClass(ClassLoader jobClassLoader, String jobClassName) {
-        try {
-            return (Class<J>) Class.forName(jobClassName, true, jobClassLoader);
-        } catch (ClassNotFoundException e) {
-            throw new IgniteInternalException("Cannot load job class by name '" + jobClassName + "'", e);
-        }
-    }
-
-    private CompletableFuture<JobContext> jobClassLoader(List<DeploymentUnit> units) {
-        return jobContextManager.acquireClassLoader(units);
-    }

Review Comment:
   > What's the point to have non-static one line method with direct method call?
   
   Brevity.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20839 Compute jobs queue [ignite-3]

Posted by "Pochatkin (via GitHub)" <gi...@apache.org>.
Pochatkin commented on code in PR #2843:
URL: https://github.com/apache/ignite-3/pull/2843#discussion_r1398875054


##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/ComputeComponentImpl.java:
##########
@@ -75,111 +58,118 @@ public class ComputeComponentImpl implements ComputeComponent {
 
     private final InFlightFutures inFlightFutures = new InFlightFutures();
 
-    private final Ignite ignite;
-
     private final MessagingService messagingService;
 
-    private final ComputeConfiguration configuration;
-
     private final JobContextManager jobContextManager;
 
-    private ExecutorService jobExecutorService;
+    private final ComputeExecutor executor;
 
     /**
      * Creates a new instance.
      */
     public ComputeComponentImpl(
-            Ignite ignite,
             MessagingService messagingService,
-            ComputeConfiguration configuration,
-            JobContextManager jobContextManager) {
-        this.ignite = ignite;
+            JobContextManager jobContextManager,
+            ComputeExecutor executor
+    ) {
         this.messagingService = messagingService;
-        this.configuration = configuration;
         this.jobContextManager = jobContextManager;
+        this.executor = executor;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void start() {
+        executor.start();
+
+        messagingService.addMessageHandler(ComputeMessageTypes.class, (message, senderConsistentId, correlationId) -> {
+            assert correlationId != null;
+
+            if (message instanceof ExecuteRequest) {
+                processExecuteRequest((ExecuteRequest) message, senderConsistentId, correlationId);
+            }
+        });
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void stop() throws Exception {
+        if (!stopGuard.compareAndSet(false, true)) {
+            return;
+        }
+
+        busyLock.block();
+
+        executor.stop();
+
+        inFlightFutures.cancelInFlightFutures();
     }
 
     /** {@inheritDoc} */
     @Override
-    public <R> CompletableFuture<R> executeLocally(List<DeploymentUnit> units, String jobClassName, Object... args) {
+    public <R> CompletableFuture<R> executeLocally(
+            ExecutionOptions options,
+            List<DeploymentUnit> units,
+            String jobClassName,
+            Object... args
+    ) {
         if (!busyLock.enterBusy()) {
             return failedFuture(new NodeStoppingException());
         }
 
         try {
-            return mapClassLoaderExceptions(jobClassLoader(units), jobClassName)
-                    .thenCompose(context -> doExecuteLocally(this.<R, ComputeJob<R>>jobClass(context.classLoader(), jobClassName), args)
-                            .whenComplete((r, e) -> context.close())
+            CompletableFuture<JobContext> jobContextCompletableFuture = mapClassLoaderExceptions(
+                    jobContextManager.acquireClassLoader(units), jobClassName);
+            return jobContextCompletableFuture
+                    .thenCompose(context ->
+                            doExecuteLocally(options, ComputeUtils.<R>instantiateJob(context.classLoader(), jobClassName), args)
+                                    .whenComplete((r, e) -> context.close())
                     );
         } finally {
             busyLock.leaveBusy();
         }
     }
 
-    private <R> CompletableFuture<R> doExecuteLocally(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        assert jobExecutorService != null : "Not started yet!";
-
-        CompletableFuture<R> future = startLocalExecution(jobClass, args);
+    private <R> CompletableFuture<R> doExecuteLocally(ExecutionOptions options, ComputeJob<R> jobInstance, Object[] args) {
+        CompletableFuture<R> future = executor.executeJob(options, jobInstance, args);
         inFlightFutures.registerFuture(future);
 
         return future;
     }
 
-    private <R> CompletableFuture<R> startLocalExecution(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        try {
-            return CompletableFuture.supplyAsync(() -> executeJob(jobClass, args), jobExecutorService);
-        } catch (RejectedExecutionException e) {
-            return failedFuture(e);
-        }
-    }
-
-    private <R> R executeJob(Class<? extends ComputeJob<R>> jobClass, Object[] args) {
-        ComputeJob<R> job = instantiateJob(jobClass);
-        JobExecutionContext context = new JobExecutionContextImpl(ignite);
-        // TODO: IGNITE-16746 - translate NodeStoppingException to a public exception

Review Comment:
   This exception should be translatted by top level exception mapper, because this is IgniteInternalCheckedException



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org