You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "ptupitsyn (via GitHub)" <gi...@apache.org> on 2025/04/17 14:54:20 UTC
[PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
ptupitsyn opened a new pull request, #5666:
URL: https://github.com/apache/ignite-3/pull/5666
https://issues.apache.org/jira/browse/IGNITE-25114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
Posted by "isapego (via GitHub)" <gi...@apache.org>.
isapego commented on code in PR #5666:
URL: https://github.com/apache/ignite-3/pull/5666#discussion_r2065686293
##########
modules/platforms/dotnet/Apache.Ignite/Internal/Compute/Executor/ComputeJobExecutor.cs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+namespace Apache.Ignite.Internal.Compute.Executor;
+
+using System;
+using System.Collections.Generic;
+using System.Threading.Tasks;
+using Buffers;
+using Proto.MsgPack;
+
+/// <summary>
+/// Compute executor utilities.
+/// </summary>
+internal static class ComputeJobExecutor
+{
+ /// <summary>
+ /// Compute executor id.
+ /// </summary>
+ internal static readonly string? IgniteComputeExecutorId = Environment.GetEnvironmentVariable("IGNITE_COMPUTE_EXECUTOR_ID");
+
+ /// <summary>
+ /// Executes compute job.
+ /// </summary>
+ /// <param name="request">Request.</param>
+ /// <param name="response">Response.</param>
+ /// <returns>Task.</returns>
+ internal static async Task ExecuteJobAsync(
+ PooledBuffer request,
+ PooledArrayBuffer response)
+ {
+ var jobReq = Read(request.GetReader());
+ var jobRes = await ExecuteJobAsync(jobReq).ConfigureAwait(false);
+
+ Write(response.MessageWriter, jobRes);
+
+ static JobExecuteRequest Read(MsgPackReader r)
+ {
+ long jobId = r.ReadInt64();
+ string jobClassName = r.ReadString();
+
+ int cnt = r.ReadInt32();
+ List<string> deploymentUnitPaths = new List<string>(cnt);
+ for (int i = 0; i < cnt; i++)
+ {
+ deploymentUnitPaths.Add(r.ReadString());
+ }
+
+ object arg = ComputePacker.UnpackArgOrResult<object>(ref r, null);
+
+ return new JobExecuteRequest(jobId, deploymentUnitPaths, jobClassName, arg);
+ }
+
+ static void Write(MsgPackWriter w, object? res)
+ {
+ w.Write(0); // Flags: success.
+ ComputePacker.PackArgOrResult(ref w, res, null);
+ }
+ }
+
+ private static ValueTask<object?> ExecuteJobAsync(JobExecuteRequest req)
+ {
+ if (req.JobClassName == "TEST_ONLY_DOTNET_JOB:ECHO")
+ {
+ return ValueTask.FromResult(req.Arg)!;
+ }
Review Comment:
Should we put some kind of `TODO` here? This part's purpose seems to be for testing only.
##########
modules/platforms/dotnet/Apache.Ignite.Internal.ComputeExecutor/Program.cs:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System.Net.Security;
+using Apache.Ignite;
+using Microsoft.Extensions.Logging;
+
+const string envServerAddr = "IGNITE_COMPUTE_EXECUTOR_SERVER_ADDRESS";
Review Comment:
Are there any cases when the address is not `127.0.0.1`, or is it mainly about the port?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #5666:
URL: https://github.com/apache/ignite-3/pull/5666#discussion_r2065713451
##########
modules/platforms/dotnet/Apache.Ignite.Internal.ComputeExecutor/Program.cs:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System.Net.Security;
+using Apache.Ignite;
+using Microsoft.Extensions.Logging;
+
+const string envServerAddr = "IGNITE_COMPUTE_EXECUTOR_SERVER_ADDRESS";
Review Comment:
In the future we might run the executor in a separate container or even on a remote machine.
Overall it does not matter much, we don't have to worry about compatibility here (for now, with local-only executors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #5666:
URL: https://github.com/apache/ignite-3/pull/5666#discussion_r2065708584
##########
modules/platforms/dotnet/Apache.Ignite/Internal/Compute/Executor/ComputeJobExecutor.cs:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+namespace Apache.Ignite.Internal.Compute.Executor;
+
+using System;
+using System.Collections.Generic;
+using System.Threading.Tasks;
+using Buffers;
+using Proto.MsgPack;
+
+/// <summary>
+/// Compute executor utilities.
+/// </summary>
+internal static class ComputeJobExecutor
+{
+ /// <summary>
+ /// Compute executor id.
+ /// </summary>
+ internal static readonly string? IgniteComputeExecutorId = Environment.GetEnvironmentVariable("IGNITE_COMPUTE_EXECUTOR_ID");
+
+ /// <summary>
+ /// Executes compute job.
+ /// </summary>
+ /// <param name="request">Request.</param>
+ /// <param name="response">Response.</param>
+ /// <returns>Task.</returns>
+ internal static async Task ExecuteJobAsync(
+ PooledBuffer request,
+ PooledArrayBuffer response)
+ {
+ var jobReq = Read(request.GetReader());
+ var jobRes = await ExecuteJobAsync(jobReq).ConfigureAwait(false);
+
+ Write(response.MessageWriter, jobRes);
+
+ static JobExecuteRequest Read(MsgPackReader r)
+ {
+ long jobId = r.ReadInt64();
+ string jobClassName = r.ReadString();
+
+ int cnt = r.ReadInt32();
+ List<string> deploymentUnitPaths = new List<string>(cnt);
+ for (int i = 0; i < cnt; i++)
+ {
+ deploymentUnitPaths.Add(r.ReadString());
+ }
+
+ object arg = ComputePacker.UnpackArgOrResult<object>(ref r, null);
+
+ return new JobExecuteRequest(jobId, deploymentUnitPaths, jobClassName, arg);
+ }
+
+ static void Write(MsgPackWriter w, object? res)
+ {
+ w.Write(0); // Flags: success.
+ ComputePacker.PackArgOrResult(ref w, res, null);
+ }
+ }
+
+ private static ValueTask<object?> ExecuteJobAsync(JobExecuteRequest req)
+ {
+ if (req.JobClassName == "TEST_ONLY_DOTNET_JOB:ECHO")
+ {
+ return ValueTask.FromResult(req.Arg)!;
+ }
Review Comment:
Added `TODO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5666:
URL: https://github.com/apache/ignite-3/pull/5666#discussion_r2049134546
##########
modules/client-handler/src/main/java/org/apache/ignite/client/handler/ClientInboundMessageHandler.java:
##########
@@ -393,18 +420,36 @@ private void handshake(ChannelHandlerContext ctx, ClientMessageUnpacker unpacker
int clientCode = unpacker.unpackInt();
BitSet clientFeatures = HandshakeUtils.unpackFeatures(unpacker);
- Map<HandshakeExtension, Object> extensions = HandshakeUtils.unpackExtensions(unpacker);
+ Map<HandshakeExtension, Object> clientHandshakeExtensions = HandshakeUtils.unpackExtensions(unpacker);
+ String computeExecutorId = (String) clientHandshakeExtensions.get(HandshakeExtension.COMPUTE_EXECUTOR_ID);
+
+ if (computeExecutorId != null) {
+ CompletableFuture<PlatformComputeConnection> computeConnFut = computeConnectionFunc.apply(computeExecutorId);
+
+ if (computeConnFut == null) {
+ LOG.warn("Invalid compute executor ID, client connection rejected [connectionId=" + connectionId
+ + ", remoteAddress=" + ctx.channel().remoteAddress() + "]: " + computeExecutorId);
+
+ ctx.close();
+ return;
+ }
+
+ // Bypass authentication for compute executor connections.
+ handshakeSuccess(ctx, packer, UserDetails.UNKNOWN, clientFeatures, clientVer, clientCode);
+
+ // Ready to handle compute requests now.
+ computeConnFut.complete(this::executeJobAsync);
Review Comment:
The method reference 'this::executeJobAsync' has a signature that does not match the expected type (PlatformComputeConnection). Consider revising the type conversion or wrapping the call to produce an appropriate PlatformComputeConnection instance.
```suggestion
computeConnFut.complete(() -> new PlatformComputeConnection(this::executeJobAsync));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25114 Support platform job execution in client protocol [ignite-3]
Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn merged PR #5666:
URL: https://github.com/apache/ignite-3/pull/5666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org