You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/08/06 21:33:13 UTC

[PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

the-other-tim-brown opened a new pull request, #13686:
URL: https://github.com/apache/hudi/pull/13686

   ### Change Logs
   
   Currently, our payload based paths use `HoodieAvroRecord` to transport data between Spark Executors. As we move away from the Payloads though, we can start relying on the other record objects directly. The `HoodieAvroIndexedRecord` can fit our needs for transporting the Avro data but needs some changes to match the existing performance.
   
   This change introduces a new class `SerializableIndexedRecord` which is used to manage the serialization of the data in the `HoodieAvroIndexedRecord`. Unlike payloads, the data is only written out to a byte array when it is required. This allows us to keep performance on par with the existing performance when working with data that only resides within a single machine.
   
   For existing workflows that use `HoodieAvroIndexedRecord` like compaction, we expect to see the same performance. This is validate with a JMH Microbenchmark where I validate that the call the `setSchema` does not cause throughput changes when working with the object.
   
   The serialized size of the object when using Kryo is also about 2/3 the size of the existing record with a fairly basic object with 15 fields with mainly numeric or small strings as values.
   
   ### Impact
   
   Allows us to move away from Payloads without performance degradation on serialization
   
   ### Risk level (write none, low medium or high below)
   
   Low
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3169023771

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 455cb89c64587715353134ca60da19542751b1bb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645) 
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162435708

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592) 
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 3d5ea738e2fa9f831f70e9f73410b9bdafb88d16 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161738801

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583) 
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2262463769


##########
hudi-common/src/main/java/org/apache/hudi/common/engine/RecordContext.java:
##########
@@ -70,6 +70,15 @@ protected RecordContext(HoodieTableConfig tableConfig) {
         .orElseThrow(() -> new IllegalArgumentException("No record keys specified and meta fields are not populated")));
   }
 
+  /**
+   * Constructs an instance of {@link RecordContext} that provides field accessor methods but cannot compute the record key for records.
+   */
+  protected RecordContext() {

Review Comment:
   Can we update the constructor with `TypeConverter` as parameter and initialize `typeConverter` to avoid potential NPE?



##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {

Review Comment:
   `GenericRecord` already extends `IndexedRecord`



##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {

Review Comment:
   Seems the `record` will always be non-null since the check in line 47?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263550462


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   The indexing and precombine will all potentially read back the record to get the ordering value and whether the record is a delete right now. We can try to avoid that as part of the effort to move off of the payloads. 
   
   The spillable maps on the reader and merge handle are going to be using the BufferedRecords going forward so we will not have both avro and byte[] objects in the data serialized out to the disk. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162359819

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263187007


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -122,43 +152,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    setSchema(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    setSchema(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    setSchema(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    setSchema(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    setSchema(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    setSchema(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   public boolean isDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    setSchema(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);

Review Comment:
   +1 its just a wrapper pojo - nothing heavy in constructor. caching is not needed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263227420


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+      } catch (IOException e) {
+        throw new HoodieIOException("Failed to parse record into provided schema", e);
+      }
+    }
+  }
+
+  @Override
+  public void write(Kryo kryo, Output output) {
+    byte[] bytes = encodeRecord();
+    output.writeInt(bytes.length, true);
+    output.writeBytes(bytes);
+  }
+
+  @Override
+  public void read(Kryo kryo, Input input) {
+    int length = input.readInt(true);
+    this.recordBytes = input.readBytes(length);
+  }
+
+  @Override
+  public void put(String key, Object v) {
+    Schema.Field field = record.getSchema().getField(key);
+    record.put(field.pos(), v);
+  }
+
+  @Override
+  public Object get(String key) {
+    Schema.Field field = record.getSchema().getField(key);
+    return record.get(field.pos());
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (this == o) {
+      return true;
+    }
+    if (o == null) {
+      return false;
+    }
+    if (o instanceof SerializableIndexedRecord) {
+      SerializableIndexedRecord that = (SerializableIndexedRecord) o;
+      ValidationUtils.checkArgument(record != null && that.record != null, "Records must be deserialized before equality check");
+      return record.equals(that.record);
+    } else if (o instanceof IndexedRecord) {
+      // If the other object is an IndexedRecord, we can compare it directly
+      IndexedRecord that = (IndexedRecord) o;
+      return record != null && record.equals(that);
+    }
+    return false;
+  }
+
+  IndexedRecord getRecord() {
+    ValidationUtils.checkArgument(record != null, "Record must be deserialized before accessing");
+    return record;
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(record);
+  }
+
+  @Override
+  public String toString() {
+    return record == null ? "SERIALIZED: " + new String(recordBytes) : record.toString();

Review Comment:
   we probably have a util to hex print byte arrays? printing record is a bit sensitive. please check once that we are not accidentally logging this more than before etc. Resolve if you have done it already



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264213673


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -122,43 +152,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    setSchema(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    setSchema(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    setSchema(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    setSchema(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    setSchema(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    setSchema(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   public boolean isDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    setSchema(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);

Review Comment:
   We already reuse the isDelete then should be fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2262530265


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+      } catch (IOException e) {
+        throw new HoodieIOException("Failed to parse record into provided schema", e);
+      }
+    }
+  }
+
+  @Override
+  public void write(Kryo kryo, Output output) {
+    byte[] bytes = encodeRecord();
+    output.writeInt(bytes.length, true);
+    output.writeBytes(bytes);
+  }
+
+  @Override
+  public void read(Kryo kryo, Input input) {
+    int length = input.readInt(true);
+    this.recordBytes = input.readBytes(length);
+  }
+
+  @Override
+  public void put(String key, Object v) {
+    Schema.Field field = record.getSchema().getField(key);
+    record.put(field.pos(), v);
+  }
+
+  @Override
+  public Object get(String key) {
+    Schema.Field field = record.getSchema().getField(key);
+    return record.get(field.pos());
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (this == o) {
+      return true;
+    }
+    if (o == null) {
+      return false;
+    }
+    if (o instanceof SerializableIndexedRecord) {
+      SerializableIndexedRecord that = (SerializableIndexedRecord) o;
+      ValidationUtils.checkArgument(record != null && that.record != null, "Records must be deserialized before equality check");
+      return record.equals(that.record);
+    } else if (o instanceof IndexedRecord) {
+      // If the other object is an IndexedRecord, we can compare it directly
+      IndexedRecord that = (IndexedRecord) o;
+      return record != null && record.equals(that);
+    }
+    return false;
+  }
+
+  IndexedRecord getRecord() {
+    return record;

Review Comment:
   Should we ensure the `record` has been assigned by `decodeRecord()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172237208

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650) 
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172307847

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674) 
   * 76b85853d4382f955337b761ab94a21f76f9770f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3169065254

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 455cb89c64587715353134ca60da19542751b1bb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645) 
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3169326251

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264598016


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -126,43 +156,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    decodeRecord(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    decodeRecord(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    decodeRecord(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    decodeRecord(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    decodeRecord(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    decodeRecord(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   protected boolean checkIsDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    decodeRecord(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);
+    return AVRO_RECORD_CONTEXT.isDeleteRecord(data, deleteContext);

Review Comment:
   We should call `deleteContext.withReaderSchema()` before this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263521258


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+      } catch (IOException e) {
+        throw new HoodieIOException("Failed to parse record into provided schema", e);
+      }
+    }
+  }
+
+  @Override
+  public void write(Kryo kryo, Output output) {
+    byte[] bytes = encodeRecord();
+    output.writeInt(bytes.length, true);
+    output.writeBytes(bytes);
+  }
+
+  @Override
+  public void read(Kryo kryo, Input input) {
+    int length = input.readInt(true);
+    this.recordBytes = input.readBytes(length);
+  }
+
+  @Override
+  public void put(String key, Object v) {
+    Schema.Field field = record.getSchema().getField(key);
+    record.put(field.pos(), v);
+  }
+
+  @Override
+  public Object get(String key) {
+    Schema.Field field = record.getSchema().getField(key);
+    return record.get(field.pos());
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (this == o) {
+      return true;
+    }
+    if (o == null) {
+      return false;
+    }
+    if (o instanceof SerializableIndexedRecord) {
+      SerializableIndexedRecord that = (SerializableIndexedRecord) o;
+      ValidationUtils.checkArgument(record != null && that.record != null, "Records must be deserialized before equality check");
+      return record.equals(that.record);
+    } else if (o instanceof IndexedRecord) {
+      // If the other object is an IndexedRecord, we can compare it directly
+      IndexedRecord that = (IndexedRecord) o;
+      return record != null && record.equals(that);
+    }
+    return false;
+  }
+
+  IndexedRecord getRecord() {
+    ValidationUtils.checkArgument(record != null, "Record must be deserialized before accessing");
+    return record;
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(record);
+  }
+
+  @Override
+  public String toString() {
+    return record == null ? "SERIALIZED: " + new String(recordBytes) : record.toString();

Review Comment:
   Currently we don't override the toString of IndexedRecord we work with but we can do that here or remove the toString so it just logs out the class name and instance number. We've removed logging of records in a few error cases already but that does not mean we will not re-add things in the future.



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -225,8 +269,13 @@ public Comparable<?> doGetOrderingValue(Schema recordSchema, Properties props, S
         field -> (Comparable<?>) HoodieAvroUtils.getNestedFieldVal((GenericRecord) data, field, true, consistentLogicalTimestampEnabled));
   }
 
+  private void setSchema(Schema recordSchema) {
+    optimizedRecord.decodeRecord(recordSchema);

Review Comment:
   Yes, updating the name to decodeRecord. This is left over from an old implementation where it did just set a schema.



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();
+  private SerializableIndexedRecord optimizedRecord;

Review Comment:
   It is lazily translated to the byte[], do we still want to call it `binaryRecord`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172247102

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650) 
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265046333


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -126,43 +156,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    decodeRecord(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    decodeRecord(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    decodeRecord(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    decodeRecord(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    decodeRecord(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    decodeRecord(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   protected boolean checkIsDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    decodeRecord(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);
+    return AVRO_RECORD_CONTEXT.isDeleteRecord(data, deleteContext);

Review Comment:
   Thanks, I didn't realize that was needed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162204131

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589) 
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590) 
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161929701

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588) 
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265046571


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();
+  private SerializableIndexedRecord optimizedRecord;

Review Comment:
   updating to `binaryRecord`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265046770


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   Ok, I've updated this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162206517

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590) 
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161883376

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161834422

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583) 
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162095816

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588) 
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589) 
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172587575

   Azure CI passed: https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=7678&view=results


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13686:
URL: https://github.com/apache/hudi/pull/13686


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162223820

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590) 
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3164655843

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * fe07ef49c8970a3f7c401369ba4f078580c9e91f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263154314


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -122,43 +152,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    setSchema(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    setSchema(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    setSchema(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    setSchema(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    setSchema(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    setSchema(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   public boolean isDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    setSchema(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);

Review Comment:
   I'm not sure where we would cache the context. Right now this will match our current performance with the payload records so I'm not sure it is worth the complexity of a cache. 



##########
hudi-common/src/main/java/org/apache/hudi/common/engine/RecordContext.java:
##########
@@ -70,6 +70,15 @@ protected RecordContext(HoodieTableConfig tableConfig) {
         .orElseThrow(() -> new IllegalArgumentException("No record keys specified and meta fields are not populated")));
   }
 
+  /**
+   * Constructs an instance of {@link RecordContext} that provides field accessor methods but cannot compute the record key for records.
+   */
+  protected RecordContext() {

Review Comment:
   Yes, updating now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168646217

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * a4ca0e78b9e906de2daa304ae06f0576a5fdb133 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172224533

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 75579683ab2ec4b7e1a128780d30cde19d6f67d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650) 
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172360003

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7678",
       "triggerID" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 76b85853d4382f955337b761ab94a21f76f9770f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7678) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2258376390


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -391,6 +391,7 @@ public final void write(Kryo kryo, Output output) {
     //       implementation
     writeRecordPayload(data, kryo, output);
     kryo.writeObjectOrNull(output, ignoreIndexUpdate, Boolean.class);
+    kryo.writeClassAndObject(output, orderingValue);

Review Comment:
   Note to reviewers: Previously we were not serializing the ordering value but it seems like we can do this to avoid always recomputing the value



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161699676

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162038257

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588) 
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161879423

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583) 
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162143596

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589) 
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161708618

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583) 
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263053587


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   This was an optimization to avoid recomputing the byte[] if this instance is re-shuffled in spark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263197402


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;

Review Comment:
   explicity set `recordBytes = null` for readability



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264214095


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;

Review Comment:
   Because we already have a `HoodieAvroRecord`, hmm...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3164076321

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 3d5ea738e2fa9f831f70e9f73410b9bdafb88d16 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595) 
   * fe07ef49c8970a3f7c401369ba4f078580c9e91f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168172576

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * fe07ef49c8970a3f7c401369ba4f078580c9e91f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605) 
   * a4ca0e78b9e906de2daa304ae06f0576a5fdb133 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263191341


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -243,7 +292,7 @@ public ByteArrayOutputStream getAvroBytes(Schema recordSchema, Properties props)
   @Override
   protected final void writeRecordPayload(IndexedRecord payload, Kryo kryo, Output output) {
     // NOTE: We're leveraging Spark's default [[GenericAvroSerializer]] to serialize Avro
-    Serializer<GenericRecord> avroSerializer = kryo.getSerializer(GenericRecord.class);
+    Serializer<SerializableIndexedRecord> avroSerializer = kryo.getSerializer(SerializableIndexedRecord.class);

Review Comment:
   note to register `SerializableIndexedRecord` with Kryo. in the right places. Please resolve if its taken care of



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265081570


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   where  did we nullify the bytes, I didn't see it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161904011

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161705171

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ee6c357ca41e479fcd52c6d8b1b5978fa582b56 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583) 
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2261838437


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   the recordBytes can be nullified here now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2258788505


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #setSchema(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord extends GenericData.Record implements IndexedRecord, GenericRecord, KryoSerializable {

Review Comment:
   looks promising!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2262509304


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {

Review Comment:
   Seems the `record` will always be non-null since the check in line 47?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2262520629


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -122,43 +152,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    setSchema(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    setSchema(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    setSchema(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    setSchema(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    setSchema(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    setSchema(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   public boolean isDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    setSchema(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);

Review Comment:
   Can we cache the `deleteContext`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168672689

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * a4ca0e78b9e906de2daa304ae06f0576a5fdb133 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641) 
   * 455cb89c64587715353134ca60da19542751b1bb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264216852


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();
+  private SerializableIndexedRecord optimizedRecord;

Review Comment:
   +1 to `binaryRecord`, `optimized` is ambigious somehow.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264238441


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();

Review Comment:
   Sill think we should move it to `AvroRecordContext` and mark it as public static. It does not belong here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168990504

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 455cb89c64587715353134ca60da19542751b1bb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161924682

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162394675

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592) 
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 3d5ea738e2fa9f831f70e9f73410b9bdafb88d16 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2261836073


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();

Review Comment:
   is this variable serialzied altogeter with this record? should we move it to just `AvroRecordContext`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263181190


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();
+  private SerializableIndexedRecord optimizedRecord;

Review Comment:
   rename: binaryRecord  (indicating its alreaady in binary form?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263176426


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;

Review Comment:
   k  this is a  HoodieRecord.. even though it says `...IndexedRecord` . problem for another day. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263526263


##########
hudi-common/src/test/java/org/apache/hudi/common/model/TestSerializableIndexRecord.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+class TestSerializableIndexRecord {

Review Comment:
   Added some more tests for this and assert that the same instance is returned after repeated calls



##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;

Review Comment:
   Ack, updated



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -243,7 +292,7 @@ public ByteArrayOutputStream getAvroBytes(Schema recordSchema, Properties props)
   @Override
   protected final void writeRecordPayload(IndexedRecord payload, Kryo kryo, Output output) {
     // NOTE: We're leveraging Spark's default [[GenericAvroSerializer]] to serialize Avro
-    Serializer<GenericRecord> avroSerializer = kryo.getSerializer(GenericRecord.class);
+    Serializer<SerializableIndexedRecord> avroSerializer = kryo.getSerializer(SerializableIndexedRecord.class);

Review Comment:
   Ack, updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263155881


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+      } catch (IOException e) {
+        throw new HoodieIOException("Failed to parse record into provided schema", e);
+      }
+    }
+  }
+
+  @Override
+  public void write(Kryo kryo, Output output) {
+    byte[] bytes = encodeRecord();
+    output.writeInt(bytes.length, true);
+    output.writeBytes(bytes);
+  }
+
+  @Override
+  public void read(Kryo kryo, Input input) {
+    int length = input.readInt(true);
+    this.recordBytes = input.readBytes(length);
+  }
+
+  @Override
+  public void put(String key, Object v) {
+    Schema.Field field = record.getSchema().getField(key);
+    record.put(field.pos(), v);
+  }
+
+  @Override
+  public Object get(String key) {
+    Schema.Field field = record.getSchema().getField(key);
+    return record.get(field.pos());
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (this == o) {
+      return true;
+    }
+    if (o == null) {
+      return false;
+    }
+    if (o instanceof SerializableIndexedRecord) {
+      SerializableIndexedRecord that = (SerializableIndexedRecord) o;
+      ValidationUtils.checkArgument(record != null && that.record != null, "Records must be deserialized before equality check");
+      return record.equals(that.record);
+    } else if (o instanceof IndexedRecord) {
+      // If the other object is an IndexedRecord, we can compare it directly
+      IndexedRecord that = (IndexedRecord) o;
+      return record != null && record.equals(that);
+    }
+    return false;
+  }
+
+  IndexedRecord getRecord() {
+    return record;

Review Comment:
   Makes sense, adding a validation with a helpful error message  instead of simply ending up with an NPE



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263221341


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   yep. But, most stages don't decode the `recordBytes` again though? e.g. indexing, precombine, workload profile - all works off the `HoodieRecord` fields.. 
   
   So, if decoding only happens in write handles for e.g, it may be good to nullify to reduce memory pressure. (not have both avro and bytes in a spillable map) .. Can we do an analysis around this and close this out?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265088535


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   Sorry forgot to commit that part, it is updated now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172325487

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7678",
       "triggerID" : "76b85853d4382f955337b761ab94a21f76f9770f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674) 
   * 76b85853d4382f955337b761ab94a21f76f9770f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7678) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264229194


##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Wrapper class for {@link IndexedRecord} that will serialize the record as bytes without the schema for efficient shuffling performance in Spark.
+ * After deserialization, the schema can be set using {@link #decodeRecord(Schema)} and that will also trigger deserialization of the record.
+ * This allows the record to stay in the serialized form until the data needs to be accessed, which allows deserialization to be avoided if data is not read.
+ */
+class SerializableIndexedRecord implements IndexedRecord, GenericRecord, KryoSerializable {
+  private IndexedRecord record;
+  private byte[] recordBytes;
+
+  static SerializableIndexedRecord createInstance(IndexedRecord record) {
+    return record == null ? null : new SerializableIndexedRecord(record);
+  }
+
+  private SerializableIndexedRecord(IndexedRecord record) {
+    this.record = record;
+  }
+
+  @Override
+  public void put(int i, Object v) {
+    record.put(i, v);
+  }
+
+  @Override
+  public Object get(int i) {
+    return record.get(i);
+  }
+
+  @Override
+  public Schema getSchema() {
+    return record.getSchema();
+  }
+
+  private byte[] encodeRecord() {
+    if (recordBytes == null) {
+      recordBytes = HoodieAvroUtils.avroToBytes(record);
+    }
+    return recordBytes;
+  }
+
+  void decodeRecord(Schema schema) {
+    if (record == null) {
+      try {
+        record = HoodieAvroUtils.bytesToAvro(recordBytes, schema);

Review Comment:
   > will all potentially read back the record to get the ordering value and whether the record is a delete right now.
   
   As long as we have the `IndexedRecord` and `ReaderContext` there, the ordering value and isDelete can be fethed, the bytes is only useful for serialization IMO. Nullify the bytes[] will have 50% memory cost cut when the record is deserialized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162102965

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589) 
   * bfbdbb55b989e84ead2001a615a198c73cafdb48 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3161927017

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 8e240ff741f889887e0dab320a32b62f90d7f1da Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584) 
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588) 
   * 2bfe5539f90b1542fcf06fe0448390b3c889c932 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162377101

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * b467eeafe99872c71d91f7a6ce71208d856ee0e0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592) 
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3162579490

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 3d5ea738e2fa9f831f70e9f73410b9bdafb88d16 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3164286358

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * 3d5ea738e2fa9f831f70e9f73410b9bdafb88d16 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595) 
   * fe07ef49c8970a3f7c401369ba4f078580c9e91f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3172283642

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7650",
       "triggerID" : "75579683ab2ec4b7e1a128780d30cde19d6f67d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b0ca2da1ab4d1d67c015858b9c13b7f670059959",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674",
       "triggerID" : "6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * b0ca2da1ab4d1d67c015858b9c13b7f670059959 UNKNOWN
   * 6b4ce8e1940cf24ff5d2dab0b5bbc7bd7601e48c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7674) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3173232048

   Landed it first to unblock the write path payload deprecation. All the comments are addressed, we can address the left over comments in the payload usage deprecation PR if possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2264589118


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -122,43 +152,54 @@ public Object[] getColumnValues(Schema recordSchema, String[] columns, boolean c
 
   @Override
   public Object getColumnValueAsJava(Schema recordSchema, String column, Properties props) {
+    setSchema(recordSchema);
     return AvroRecordContext.getFieldValueFromIndexedRecord(data, column);
   }
 
   @Override
   public HoodieRecord joinWith(HoodieRecord other, Schema targetSchema) {
+    setSchema(targetSchema);
     GenericRecord record = HoodieAvroUtils.stitchRecords((GenericRecord) data, (GenericRecord) other.getData(), targetSchema);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord prependMetaFields(Schema recordSchema, Schema targetSchema, MetadataValues metadataValues, Properties props) {
+    setSchema(recordSchema);
     GenericRecord newAvroRecord = HoodieAvroUtils.rewriteRecordWithNewSchema(data, targetSchema);
     updateMetadataValuesInternal(newAvroRecord, metadataValues);
     return new HoodieAvroIndexedRecord(key, newAvroRecord, operation, metaData);
   }
 
   @Override
   public HoodieRecord updateMetaField(Schema recordSchema, int ordinal, String value) {
+    setSchema(recordSchema);
     data.put(ordinal, value);
     return new HoodieAvroIndexedRecord(key, data, operation, metaData);
   }
 
   @Override
   public HoodieRecord rewriteRecordWithNewSchema(Schema recordSchema, Properties props, Schema newSchema, Map<String, String> renameCols) {
+    setSchema(recordSchema);
     GenericRecord record = HoodieAvroUtils.rewriteRecordWithNewSchema(data, newSchema, renameCols);
     return new HoodieAvroIndexedRecord(key, record, operation, metaData);
   }
 
   @Override
   public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, String keyFieldName) {
+    setSchema(recordSchema);
     ((GenericRecord) data).put(keyFieldName, StringUtils.EMPTY_STRING);
     return this;
   }
 
   @Override
   public boolean isDelete(Schema recordSchema, Properties props) {
-    return false;
+    if (getData().equals(SENTINEL)) {
+      return false; // Sentinel record is not a delete
+    }
+    setSchema(recordSchema);
+    DeleteContext deleteContext = new DeleteContext(props, recordSchema);

Review Comment:
   ```
   public DeleteContext(Properties props, Schema tableSchema) {
       this.customDeleteMarkerKeyValue = getCustomDeleteMarkerKevValue(props);
       this.hasBuiltInDeleteField = hasBuiltInDeleteField(tableSchema);
       this.hoodieOperationPos = getHoodieOperationPos(tableSchema);
     }
   ```
   
   Seems it's not just a wrapper pojo,  3 fields in DeleteContext will be calculated for every record, and they are same for different records because their `recordSchema` are same.
   
   Since the performance has met expectations, I'm okay with putting the caching on hold to keep things less complex.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2265047326


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();

Review Comment:
   Right, makes sense. I added an instance there that is accessed with `getFieldAccessorInstance` as a way to try to keep the use case clear to callers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263051097


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -47,40 +48,62 @@
  * This only use by reader returning.
  */
 public class HoodieAvroIndexedRecord extends HoodieRecord<IndexedRecord> {
+  private static final long serialVersionUID = 1L;
+  private static final AvroRecordContext AVRO_RECORD_CONTEXT = new AvroRecordContext();

Review Comment:
   No, this is a static variable, not an instance variable. Also the serialization is implemented by hand for this class to ensure only the minimum amount of data is serialized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263212843


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroIndexedRecord.java:
##########
@@ -225,8 +269,13 @@ public Comparable<?> doGetOrderingValue(Schema recordSchema, Properties props, S
         field -> (Comparable<?>) HoodieAvroUtils.getNestedFieldVal((GenericRecord) data, field, true, consistentLogicalTimestampEnabled));
   }
 
+  private void setSchema(Schema recordSchema) {
+    optimizedRecord.decodeRecord(recordSchema);

Review Comment:
   this is a lil misleading. its decoding the record, when you probably expect just setting schema



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13686:
URL: https://github.com/apache/hudi/pull/13686#discussion_r2263230407


##########
hudi-common/src/test/java/org/apache/hudi/common/model/TestSerializableIndexRecord.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+class TestSerializableIndexRecord {

Review Comment:
   we can prob expand with more cases around avoiding unnecessary decoding .. encode/decode etc. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168271586

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * fe07ef49c8970a3f7c401369ba4f078580c9e91f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605) 
   * a4ca0e78b9e906de2daa304ae06f0576a5fdb133 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9641] Improve Serialization Performance of HoodieAvroIndexedRecord [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13686:
URL: https://github.com/apache/hudi/pull/13686#issuecomment-3168777672

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7583",
       "triggerID" : "0ee6c357ca41e479fcd52c6d8b1b5978fa582b56",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3815c8cd66963aa86cef4e07edf8ad80a07ef8cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7584",
       "triggerID" : "8e240ff741f889887e0dab320a32b62f90d7f1da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "99a9609a73082d521bce18206fe44d2050dd38b9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7588",
       "triggerID" : "438fc5a9d7feda31f3b9fbc32fdf6e0a45210a0b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7589",
       "triggerID" : "2bfe5539f90b1542fcf06fe0448390b3c889c932",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7590",
       "triggerID" : "bfbdbb55b989e84ead2001a615a198c73cafdb48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7592",
       "triggerID" : "b467eeafe99872c71d91f7a6ce71208d856ee0e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "78e42a2b4d74dce00fd5830ff5e4fa8696c969c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7595",
       "triggerID" : "3d5ea738e2fa9f831f70e9f73410b9bdafb88d16",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7605",
       "triggerID" : "fe07ef49c8970a3f7c401369ba4f078580c9e91f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641",
       "triggerID" : "a4ca0e78b9e906de2daa304ae06f0576a5fdb133",
       "triggerType" : "PUSH"
     }, {
       "hash" : "455cb89c64587715353134ca60da19542751b1bb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645",
       "triggerID" : "455cb89c64587715353134ca60da19542751b1bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3815c8cd66963aa86cef4e07edf8ad80a07ef8cb UNKNOWN
   * 99a9609a73082d521bce18206fe44d2050dd38b9 UNKNOWN
   * 78e42a2b4d74dce00fd5830ff5e4fa8696c969c4 UNKNOWN
   * a4ca0e78b9e906de2daa304ae06f0576a5fdb133 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7641) 
   * 455cb89c64587715353134ca60da19542751b1bb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7645) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org