You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "valepakh (via GitHub)" <gi...@apache.org> on 2024/09/23 13:36:35 UTC

[PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

valepakh opened a new pull request, #4438:
URL: https://github.com/apache/ignite-3/pull/4438

   https://issues.apache.org/jira/browse/IGNITE-23091


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1774812797


##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -33,22 +33,26 @@ public class JobDescriptor<T, R> {
 
     private final JobExecutionOptions options;
 
+    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+
     private final @Nullable Marshaller<R, byte[]> resultMarshaller;
 
-    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+    private final Class<?> resultClass;

Review Comment:
   I tried it and did not face any issues, `pojoJobWithDifferentClass` compiles and works: https://github.com/apache/ignite-3/pull/4447/commits/5debde84718c39f9fcff964552cb29d752f7b1a5



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1774599750


##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobPacker.java:
##########
@@ -17,13 +17,30 @@
 
 package org.apache.ignite.internal.client.proto;
 
+import static org.apache.ignite.internal.client.proto.ComputeJobType.MARSHALLED_CUSTOM;
+import static org.apache.ignite.internal.client.proto.ComputeJobType.MARSHALLED_POJO;
+import static org.apache.ignite.internal.client.proto.ComputeJobType.MARSHALLED_TUPLE;
+import static org.apache.ignite.internal.client.proto.ComputeJobType.NATIVE;
+import static org.apache.ignite.internal.client.proto.pojo.PojoConverter.toTuple;
+
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
 import org.apache.ignite.internal.binarytuple.inlineschema.TupleWithSchemaMarshalling;
+import org.apache.ignite.internal.client.proto.pojo.PojoConversionException;
 import org.apache.ignite.marshalling.Marshaller;
+import org.apache.ignite.marshalling.MarshallingException;
+import org.apache.ignite.sql.ColumnType;
 import org.apache.ignite.table.Tuple;
 import org.jetbrains.annotations.Nullable;
 
 /** Packs job arguments and results. */
 public final class ClientComputeJobPacker {
+

Review Comment:
   ```suggestion
   ```



##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -100,12 +104,16 @@ public static <T, R> Builder<T, R> builder(Class<? extends ComputeJob<T, R>> job
         return new Builder<>(jobClass.getName());
     }
 
+    public @Nullable Marshaller<T, byte[]> argumentMarshaller() {
+        return argumentMarshaller;
+    }
+
     public @Nullable Marshaller<R, byte[]> resultMarshaller() {
         return resultMarshaller;
     }
 
-    public @Nullable Marshaller<T, byte[]> argumentMarshaller() {
-        return argumentMarshaller;
+    public @Nullable Class<?> resultClass() {

Review Comment:
   So we can infer job argument type from the overridden method, but we can't do it for the result because of type erasure, and we have to ask the user to provide the class explicitly - is my understanding correct?



##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/pojo/PojoConverter.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.proto.pojo;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.math.BigDecimal;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.Period;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.UUID;
+import org.apache.ignite.table.Tuple;
+import org.jetbrains.annotations.Nullable;
+
+/** Converts POJO to Tuple and back. */
+public class PojoConverter {
+
+    /**
+     * Converts POJO to Tuple. Supports public non-static fields, public non-static getter methods starting with "get" (or "is" for boolean
+     * type) followed by the capital letter.
+     *
+     * @param obj POJO to convert.
+     * @return Tuple with columns corresponding to supported fields of the POJO.
+     * @throws PojoConversionException If conversion failed.
+     */
+    public static Tuple toTuple(Object obj) throws PojoConversionException {
+        Class<?> clazz = obj.getClass();
+
+        // TODO https://issues.apache.org/jira/browse/IGNITE-23092
+        if (clazz.getSuperclass() != Object.class) {
+            throw new PojoConversionException("Can't convert subclasses");
+        }
+
+        Map<String, Getter> getters = new HashMap<>();
+
+        for (Field field : clazz.getDeclaredFields()) {
+            if (isSupportedType(field.getType())) {
+                int modifiers = field.getModifiers();
+                if (Modifier.isPublic(modifiers) && !Modifier.isStatic(modifiers)) {
+                    getters.put(field.getName(), () -> field.get(obj));
+                }
+            }
+        }
+
+        for (Method method : clazz.getDeclaredMethods()) {

Review Comment:
   Not only on the server. I think this code can be shared across client and server.



##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -33,22 +33,26 @@ public class JobDescriptor<T, R> {
 
     private final JobExecutionOptions options;
 
+    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+
     private final @Nullable Marshaller<R, byte[]> resultMarshaller;
 
-    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+    private final Class<?> resultClass;

Review Comment:
   ```suggestion
       private final Class<R> resultClass;
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/executor/ComputeExecutorImpl.java:
##########
@@ -137,6 +159,41 @@ private static <T, R> Callable<CompletableFuture<R>> unmarshalExecMarshal(
         );
     }
 
+    static <T, R> @Nullable Class<?> getArgumentType(Class<? extends ComputeJob<T, R>> jobClass) {
+        for (Method method : jobClass.getDeclaredMethods()) {
+            if (method.getParameterCount() == 2
+                    && method.getParameterTypes()[0] == JobExecutionContext.class
+                    && method.getParameterTypes()[1] != Object.class // skip type erased method
+                    && method.getReturnType() == CompletableFuture.class
+                    && "executeAsync".equals(method.getName())
+            ) {
+                return method.getParameterTypes()[1];
+            }
+        }
+        return null;
+    }
+
+    private static Object unmarshallPojo(Class<?> actualArgumentType, Tuple input) {

Review Comment:
   ```suggestion
       private static Object unmarshalPojo(Class<?> actualArgumentType, Tuple input) {
   ```



##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -33,22 +33,26 @@ public class JobDescriptor<T, R> {
 
     private final JobExecutionOptions options;
 
+    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+
     private final @Nullable Marshaller<R, byte[]> resultMarshaller;
 
-    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+    private final Class<?> resultClass;
 
     private JobDescriptor(
             String jobClassName,
             List<DeploymentUnit> units,
             JobExecutionOptions options,
             @Nullable Marshaller<T, byte[]> argumentMarshaller,
-            @Nullable Marshaller<R, byte[]> resultMarshaller
+            @Nullable Marshaller<R, byte[]> resultMarshaller,
+            @Nullable Class<?> resultClass

Review Comment:
   ```suggestion
               @Nullable Class<R> resultClass
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1774673688


##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -100,12 +104,16 @@ public static <T, R> Builder<T, R> builder(Class<? extends ComputeJob<T, R>> job
         return new Builder<>(jobClass.getName());
     }
 
+    public @Nullable Marshaller<T, byte[]> argumentMarshaller() {
+        return argumentMarshaller;
+    }
+
     public @Nullable Marshaller<R, byte[]> resultMarshaller() {
         return resultMarshaller;
     }
 
-    public @Nullable Marshaller<T, byte[]> argumentMarshaller() {
-        return argumentMarshaller;
+    public @Nullable Class<?> resultClass() {

Review Comment:
   > So we can infer job argument type from the overridden method, but we can't do it for the result because of type erasure, and we have to ask the user to provide the class explicitly - is my understanding correct?
   
   Yes, exactly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1771724939


##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobPacker.java:
##########
@@ -66,26 +81,34 @@ private static <T> void pack(@Nullable T obj, @Nullable Marshaller<T, byte[]> ma
         }
 
         if (obj instanceof Tuple) {
-            byte[] marshalledTuple = TupleWithSchemaMarshalling.marshal((Tuple) obj);
+            packer.packInt(MARSHALLED_TUPLE_ID);
 
-            packer.packInt(ComputeJobType.MARSHALLED_TUPLE_ID);
+            packTuple((Tuple) obj, packer);
+            return;
+        }
 
-            if (marshalledTuple == null) {
-                packer.packNil();
-                return;
-            }
+        if (isNativeType(obj.getClass())) {
+            packer.packInt(NATIVE_ID);
 
-            packer.packBinary(marshalledTuple);
+            packer.packObjectAsBinaryTuple(obj);
             return;
         }
 
-        if (obj == null) {
-            packer.packNil();
-            return;
+        try {
+            Tuple tuple = toTuple(obj);
+            packer.packInt(MARSHALLED_POJO_ID);
+            packer.packString(obj.getClass().getName());
+            packTuple(tuple, packer);
+        } catch (PojoConversionException e) {
+            throw new MarshallingException("Can't pack object", e);
         }
+    }
 
-        packer.packInt(ComputeJobType.NATIVE_ID);
+    private static boolean isNativeType(Class<?> clazz) {
+        return Arrays.stream(ColumnType.values()).map(ColumnType::javaClass).anyMatch(c -> c == clazz);

Review Comment:
   Build a set of classes in `ColumnType` on static load and check here with `contains()`.



##########
modules/client-common/src/test/java/org/apache/ignite/internal/client/proto/pojo/UnmarshallablePojo.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.proto.pojo;
+
+/** POJO with incorrect fields. */
+public class UnmarshallablePojo {

Review Comment:
   Could you please split this pojo into 5? For each invalid case we have to have separate pojo. Combining all invalid cases in single class can hide bugs. 



##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobUnpacker.java:
##########
@@ -57,51 +64,79 @@ public final class ClientComputeJobUnpacker {
         }
 
         int typeId = unpacker.unpackInt();
-        var type = ComputeJobType.Type.fromId(typeId);
 
-        switch (type) {
-            case NATIVE:
+        switch (typeId) {
+            case NATIVE_ID:
                 if (marshaller != null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is provided but the object was packed without marshaller."
                     );
                 }
 
                 return unpacker.unpackObjectFromBinaryTuple();
-            case MARSHALLED_TUPLE:
+            case MARSHALLED_TUPLE_ID:
                 return TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary());
 
-            case MARSHALLED_OBJECT:
+            case MARSHALLED_OBJECT_ID:
                 if (marshaller == null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is not provided but the object was packed with marshaller."
                     );
                 }
                 return tryUnmarshalOrCast(marshaller, unpacker.readBinary());
 
+            case MARSHALLED_POJO_ID:
+                return unpackPojo(unpacker);
+
             default:
                 throw new UnmarshallingException("Unsupported compute job type id: " + typeId);
         }
     }
 
     /** Unpacks compute job argument without marshaller. */
-    public static Object unpackJobArgumentWithoutMarshaller(ClientMessageUnpacker unpacker) {
+    public static @Nullable Object unpackJobArgumentWithoutMarshaller(ClientMessageUnpacker unpacker) {
         if (unpacker.tryUnpackNil()) {
             return null;
         }
 
         int typeId = unpacker.unpackInt();
-        var type = ComputeJobType.Type.fromId(typeId);
 
-        switch (type) {
-            case NATIVE:
+        switch (typeId) {
+            case NATIVE_ID:
                 return unpacker.unpackObjectFromBinaryTuple();
-            case MARSHALLED_TUPLE:
+            case MARSHALLED_TUPLE_ID:
                 return TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary());
-            case MARSHALLED_OBJECT:
+            case MARSHALLED_OBJECT_ID:
                 return unpacker.readBinary();
+            case MARSHALLED_POJO_ID:
+                return unpackPojo(unpacker);
             default:
                 throw new UnmarshallingException("Unsupported compute job type id: " + typeId);
         }
     }
+
+    private static Object unpackPojo(ClientMessageUnpacker unpacker) {
+        String className = unpacker.unpackString();
+
+        try {
+            Class<?> clazz = Class.forName(className);
+            Object obj = clazz.getConstructor().newInstance();

Review Comment:
   We can not create an instance of the class that the client sent. We do not know if we have this class in the classpath, we do not know if the client named the pojo `org.apache.Ignite`, who should put the class name in platform clients? 
   
   The proper approach here is to post-pone the actual unpacking to the time where we can get the actual argument of the job and then create an instance of it. 
   
   The main concern is: we do not want client to send the class name to us. 



##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobUnpacker.java:
##########
@@ -57,51 +64,79 @@ public final class ClientComputeJobUnpacker {
         }
 
         int typeId = unpacker.unpackInt();
-        var type = ComputeJobType.Type.fromId(typeId);
 
-        switch (type) {
-            case NATIVE:
+        switch (typeId) {
+            case NATIVE_ID:
                 if (marshaller != null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is provided but the object was packed without marshaller."
                     );
                 }
 
                 return unpacker.unpackObjectFromBinaryTuple();
-            case MARSHALLED_TUPLE:
+            case MARSHALLED_TUPLE_ID:
                 return TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary());
 
-            case MARSHALLED_OBJECT:
+            case MARSHALLED_OBJECT_ID:

Review Comment:
   The naming is confusing here... What is the difference between `MARSHALLED_OBJECT_ID` and `MARSHALLED_POJO_ID`? I think the better name for `MARSHALLED_OBJECT_ID` is `MARSHALLED_CUSTOM_ID`.



##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/pojo/PojoConverter.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.proto.pojo;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.math.BigDecimal;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.Period;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.UUID;
+import org.apache.ignite.table.Tuple;
+import org.jetbrains.annotations.Nullable;
+
+/** Converts POJO to Tuple and back. */
+public class PojoConverter {
+
+    /**
+     * Converts POJO to Tuple. Supports public non-static fields, public non-static getter methods starting with "get" (or "is" for boolean
+     * type) followed by the capital letter.
+     *
+     * @param obj POJO to convert.
+     * @return Tuple with columns corresponding to supported fields of the POJO.
+     * @throws PojoConversionException If conversion failed.
+     */
+    public static Tuple toTuple(Object obj) throws PojoConversionException {
+        Class<?> clazz = obj.getClass();
+
+        // TODO https://issues.apache.org/jira/browse/IGNITE-23092
+        if (clazz.getSuperclass() != Object.class) {
+            throw new PojoConversionException("Can't convert subclasses");
+        }
+
+        Map<String, Getter> getters = new HashMap<>();
+
+        for (Field field : clazz.getDeclaredFields()) {
+            if (isSupportedType(field.getType())) {
+                int modifiers = field.getModifiers();
+                if (Modifier.isPublic(modifiers) && !Modifier.isStatic(modifiers)) {
+                    getters.put(field.getName(), () -> field.get(obj));
+                }
+            }
+        }
+
+        for (Method method : clazz.getDeclaredMethods()) {

Review Comment:
   After the code is merged, can you create a ticket with the following description:
   
   During the pojo marshalling/unmarshalling we traverse the pojo class on the server side each time. Taking into account the fact that we are going to process a lot of data and change the pojo fields rearly, we can cache pojo descriptors in the cache. That should significantly improve the performance. 



##########
modules/runner/src/integrationTest/java/org/apache/ignite/internal/runner/app/client/ItThinClientPojoComputeMarshallingTest.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.runner.app.client;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import org.apache.ignite.compute.JobDescriptor;
+import org.apache.ignite.compute.JobTarget;
+import org.apache.ignite.internal.runner.app.Jobs.PojoArg;
+import org.apache.ignite.internal.runner.app.Jobs.PojoJob;
+import org.apache.ignite.internal.runner.app.Jobs.PojoResult;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+/**
+ * Test the OOTB support for POJOs in compute api.
+ */
+@SuppressWarnings("resource")
+public class ItThinClientPojoComputeMarshallingTest extends ItAbstractThinClientTest {
+
+    @ParameterizedTest
+    @ValueSource(ints = {0, 1})
+    void pojoJobWithMarshallers(int targetNodeIdx) {
+        // Given target node.
+        var targetNode = node(targetNodeIdx);
+
+        // When run job with custom marshaller for pojo argument and result.
+        PojoResult result = client().compute().execute(
+                JobTarget.node(targetNode),
+                JobDescriptor.builder(PojoJob.class).build(),
+                new PojoArg().setIntValue(2).setStrValue("1")

Review Comment:
   The DoD for the previous comments about class name: this test works and the same test but with other pojos (the client pojo names are changed, say `PojoArg2` and `PojoResult2`, but the structure stays the same) on the client site. The server job definition stays absolutely the same (accepts `PojoArg` and returns `PojoResult`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1776670271


##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/pojo/PojoConverter.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.proto.pojo;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.math.BigDecimal;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.Period;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.UUID;
+import org.apache.ignite.table.Tuple;
+import org.jetbrains.annotations.Nullable;
+
+/** Converts POJO to Tuple and back. */
+public class PojoConverter {
+
+    /**
+     * Converts POJO to Tuple. Supports public non-static fields, public non-static getter methods starting with "get" (or "is" for boolean
+     * type) followed by the capital letter.
+     *
+     * @param obj POJO to convert.
+     * @return Tuple with columns corresponding to supported fields of the POJO.
+     * @throws PojoConversionException If conversion failed.
+     */
+    public static Tuple toTuple(Object obj) throws PojoConversionException {

Review Comment:
   Ideally, we should convert directly to `BinaryTuple`, and avoid intermediate `Tuple`. We can do that as a separate ticket to limit the scope here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1773427075


##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobUnpacker.java:
##########
@@ -31,68 +31,83 @@
 import org.apache.ignite.marshalling.UnmarshallingException;
 import org.jetbrains.annotations.Nullable;
 
-/** Unpacks job arguments and results. */
+/** Unpacks job results. */
 public final class ClientComputeJobUnpacker {
-    /**
-     * Unpacks compute job argument. If the marshaller is provided, it will be used to unmarshal the argument. If the marshaller is not
-     * provided and the argument is a native column type or a tuple, it will be unpacked accordingly.
-     *
-     * @param marshaller Marshaller.
-     * @param unpacker Unpacker.
-     * @return Unpacked argument.
-     */
-    public static @Nullable Object unpackJobArgument(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
-        return unpack(marshaller, unpacker);
-    }
-
     /**
      * Unpacks compute job result. If the marshaller is provided, it will be used to unmarshal the result. If the marshaller is not provided
-     * and the result is a native column type or a tuple, it will be unpacked accordingly.
+     * and the result class is provided and the result is a tuple, it will be unpacked as a pojo of that class. If the marshaller is not
+     * provided and the result is a native column type or a tuple, it will be unpacked accordingly.
      *
-     * @param marshaller Marshaller.
      * @param unpacker Unpacker.
+     * @param marshaller Marshaller.
+     * @param resultClass Result class.
      * @return Unpacked result.
      */
-    public static @Nullable Object unpackJobResult(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
-        return unpack(marshaller, unpacker);
-    }
-
-    /** Underlying byte array expected to be in the following format: | typeId | value |. */
-    private static @Nullable Object unpack(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
+    public static @Nullable Object unpackJobResult(
+            ClientMessageUnpacker unpacker,
+            @Nullable Marshaller<?, byte[]> marshaller,
+            @Nullable Class<?> resultClass
+    ) {
         if (unpacker.tryUnpackNil()) {
             return null;
         }
 
+        // Underlying byte array expected to be in the following format: | typeId | value |.
         int typeId = unpacker.unpackInt();
 
         switch (typeId) {
-            case NATIVE_ID:
+            case NATIVE:
                 if (marshaller != null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is provided but the object was packed without marshaller."
                     );
                 }
 
                 return unpacker.unpackObjectFromBinaryTuple();
-            case MARSHALLED_TUPLE_ID:
+            case MARSHALLED_TUPLE:
                 return TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary());
 
-            case MARSHALLED_OBJECT_ID:
+            case MARSHALLED_CUSTOM:
                 if (marshaller == null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is not provided but the object was packed with marshaller."
                     );
                 }
                 return tryUnmarshalOrCast(marshaller, unpacker.readBinary());
 
-            case MARSHALLED_POJO_ID:
-                return unpackPojo(unpacker);
+            case MARSHALLED_POJO:
+                if (resultClass == null) {
+                    throw new UnmarshallingException(
+                            "Can not unpack object because the pojo class is not provided but the object was packed as pojo."
+                    );
+                }
+                return unpackPojo(unpacker, resultClass);
 
             default:
                 throw new UnmarshallingException("Unsupported compute job type id: " + typeId);
         }
     }
 
+    private static Object unpackPojo(ClientMessageUnpacker unpacker, Class<?> pojoClass) {
+        try {
+            Object obj = pojoClass.getConstructor().newInstance();
+
+            fromTuple(obj, TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary()));
+
+            return obj;
+        } catch (NoSuchMethodException e) {
+            throw new UnmarshallingException("Class " + pojoClass.getName() + " doesn't have public default constructor", e);

Review Comment:
   ```suggestion
               throw new UnmarshallingException("Class " + pojoClass.getName() + " doesn't have public default constructor. Add the default constructor or provide Marshaller for "   + pojoClass.getName() + "  in JobDescriptor.resultMarshaller",  e);
   ```



##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/ClientComputeJobUnpacker.java:
##########
@@ -31,68 +31,83 @@
 import org.apache.ignite.marshalling.UnmarshallingException;
 import org.jetbrains.annotations.Nullable;
 
-/** Unpacks job arguments and results. */
+/** Unpacks job results. */
 public final class ClientComputeJobUnpacker {
-    /**
-     * Unpacks compute job argument. If the marshaller is provided, it will be used to unmarshal the argument. If the marshaller is not
-     * provided and the argument is a native column type or a tuple, it will be unpacked accordingly.
-     *
-     * @param marshaller Marshaller.
-     * @param unpacker Unpacker.
-     * @return Unpacked argument.
-     */
-    public static @Nullable Object unpackJobArgument(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
-        return unpack(marshaller, unpacker);
-    }
-
     /**
      * Unpacks compute job result. If the marshaller is provided, it will be used to unmarshal the result. If the marshaller is not provided
-     * and the result is a native column type or a tuple, it will be unpacked accordingly.
+     * and the result class is provided and the result is a tuple, it will be unpacked as a pojo of that class. If the marshaller is not
+     * provided and the result is a native column type or a tuple, it will be unpacked accordingly.
      *
-     * @param marshaller Marshaller.
      * @param unpacker Unpacker.
+     * @param marshaller Marshaller.
+     * @param resultClass Result class.
      * @return Unpacked result.
      */
-    public static @Nullable Object unpackJobResult(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
-        return unpack(marshaller, unpacker);
-    }
-
-    /** Underlying byte array expected to be in the following format: | typeId | value |. */
-    private static @Nullable Object unpack(@Nullable Marshaller<?, byte[]> marshaller, ClientMessageUnpacker unpacker) {
+    public static @Nullable Object unpackJobResult(
+            ClientMessageUnpacker unpacker,
+            @Nullable Marshaller<?, byte[]> marshaller,
+            @Nullable Class<?> resultClass
+    ) {
         if (unpacker.tryUnpackNil()) {
             return null;
         }
 
+        // Underlying byte array expected to be in the following format: | typeId | value |.
         int typeId = unpacker.unpackInt();
 
         switch (typeId) {
-            case NATIVE_ID:
+            case NATIVE:
                 if (marshaller != null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is provided but the object was packed without marshaller."
                     );
                 }
 
                 return unpacker.unpackObjectFromBinaryTuple();
-            case MARSHALLED_TUPLE_ID:
+            case MARSHALLED_TUPLE:
                 return TupleWithSchemaMarshalling.unmarshal(unpacker.readBinary());
 
-            case MARSHALLED_OBJECT_ID:
+            case MARSHALLED_CUSTOM:
                 if (marshaller == null) {
                     throw new UnmarshallingException(
                             "Can not unpack object because the marshaller is not provided but the object was packed with marshaller."
                     );
                 }
                 return tryUnmarshalOrCast(marshaller, unpacker.readBinary());
 
-            case MARSHALLED_POJO_ID:
-                return unpackPojo(unpacker);
+            case MARSHALLED_POJO:
+                if (resultClass == null) {
+                    throw new UnmarshallingException(
+                            "Can not unpack object because the pojo class is not provided but the object was packed as pojo."

Review Comment:
   ```suggestion
                               "Can not unpack object because the pojo class is not provided but the object was packed as pojo. Provide Job result type in JobDescriptor.resultClass."
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/executor/ComputeExecutorImpl.java:
##########
@@ -137,6 +159,39 @@ private static <T, R> Callable<CompletableFuture<R>> unmarshalExecMarshal(
         );
     }
 
+    static <T, R> @Nullable Class<?> getArgumentType(Class<? extends ComputeJob<T, R>> jobClass) {
+        for (Method method : jobClass.getDeclaredMethods()) {
+            if (method.getParameterCount() == 2
+                    && method.getParameterTypes()[0] == JobExecutionContext.class
+                    && method.getParameterTypes()[1] != Object.class // skip type erased method
+                    && method.getReturnType() == CompletableFuture.class
+            ) {
+                return method.getParameterTypes()[1];
+            }
+        }
+        return null;
+    }
+
+    private static Object unmarshallPojo(Class<?> actualArgumentType, Tuple input) {
+        try {
+            Object obj = actualArgumentType.getConstructor().newInstance();
+
+            fromTuple(obj, input);
+
+            return obj;
+        } catch (NoSuchMethodException e) {
+            throw new UnmarshallingException("Class " + actualArgumentType.getName() + " doesn't have public default constructor", e);

Review Comment:
   ```suggestion
               throw new UnmarshallingException("Class " + actualArgumentType.getName() + " doesn't have public default constructor. Add the constructor or define argument marshaller in the compute job", e);
   ```



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/executor/ComputeExecutorImpl.java:
##########
@@ -137,6 +159,39 @@ private static <T, R> Callable<CompletableFuture<R>> unmarshalExecMarshal(
         );
     }
 
+    static <T, R> @Nullable Class<?> getArgumentType(Class<? extends ComputeJob<T, R>> jobClass) {
+        for (Method method : jobClass.getDeclaredMethods()) {
+            if (method.getParameterCount() == 2

Review Comment:
   I would also add the method name check here.



##########
modules/compute/src/main/java/org/apache/ignite/internal/compute/executor/ComputeExecutorImpl.java:
##########
@@ -104,31 +110,47 @@ public <T, R> JobExecutionInternal<R> executeJob(
 
     private static <T, R> Callable<CompletableFuture<R>> unmarshalExecMarshal(
             T input,
+            Class<? extends ComputeJob<T, R>> jobClass,
             ComputeJob<T, R> jobInstance,
             JobExecutionContext context,
             @Nullable Marshaller<T, byte[]> inputMarshaller
     ) {
-        return () -> jobInstance.executeAsync(context, unmarshallOrNotIfNull(inputMarshaller, input));
+        return () -> jobInstance.executeAsync(context, unmarshallOrNotIfNull(inputMarshaller, input, jobClass));
     }
 
-    private static <T> @Nullable T unmarshallOrNotIfNull(@Nullable Marshaller<T, byte[]> marshaller, Object input) {
-        if (marshaller == null || input == null) {
+    private static <T, R> @Nullable T unmarshallOrNotIfNull(
+            @Nullable Marshaller<T, byte[]> marshaller,
+            Object input,

Review Comment:
   ```suggestion
               @Nullable Object input,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "ptupitsyn (via GitHub)" <gi...@apache.org>.
ptupitsyn commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1776650459


##########
modules/client-common/src/main/java/org/apache/ignite/internal/client/proto/pojo/PojoConverter.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.proto.pojo;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.math.BigDecimal;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.Period;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.UUID;
+import org.apache.ignite.table.Tuple;
+import org.jetbrains.annotations.Nullable;
+
+/** Converts POJO to Tuple and back. */
+public class PojoConverter {
+
+    /**
+     * Converts POJO to Tuple. Supports public non-static fields, public non-static getter methods starting with "get" (or "is" for boolean
+     * type) followed by the capital letter.

Review Comment:
   I don't think we need to support getters at all. Only fields, including non-public. This is how serializers and mappers usually work - including the POJO mapper that we already use in Table API.
   
   In fact, I think we should reuse the existing mapping logic instead of inventing another custom serializer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "PakhomovAlexander (via GitHub)" <gi...@apache.org>.
PakhomovAlexander commented on PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#issuecomment-2371447405

   LGTM, left minor comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on code in PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#discussion_r1774676757


##########
modules/api/src/main/java/org/apache/ignite/compute/JobDescriptor.java:
##########
@@ -33,22 +33,26 @@ public class JobDescriptor<T, R> {
 
     private final JobExecutionOptions options;
 
+    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+
     private final @Nullable Marshaller<R, byte[]> resultMarshaller;
 
-    private final @Nullable Marshaller<T, byte[]> argumentMarshaller;
+    private final Class<?> resultClass;

Review Comment:
   This is how I initially made this, but since we're doing pojos, the exact type could be not the result of the job but some other class, so that the `ItThinClientPojoComputeMarshallingTest.pojoJobWithDifferentClass` won't compile.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-23091 Java Client: Automatic Pojo Serialization [ignite-3]

Posted by "valepakh (via GitHub)" <gi...@apache.org>.
valepakh commented on PR #4438:
URL: https://github.com/apache/ignite-3/pull/4438#issuecomment-2373786587

   I realized that I missed map reduce task marshalling. I reordered the code a little and also noticed that we don't have a stable order in the `reduceAsync` argument. I made it the same as the result of the split.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org